FEC in Coherent DWDM:
Why Your Link Budget Might Be Wrong

When I was reviewing a DWDM link budget calculator recently, I found a surprisingly common mistake: the tool was using the OIF 800ZR required OSNR of 27 dB as a baseline without FEC, and then subtracting 3 dB for sFEC coding gain to get an "effective" threshold of 24 dB. The problem is that this is wrong — and on a marginal link, it can make the difference between a design that works in the field and one that doesn't.

In our network (800G PS-16QAM, 135 GBaud, 150 GHz channel spacing, sFEC 20–27%), links start failing when the measured OSNR drops below 25.5 dB. Not 24 dB. Not 22 dB. 25.5 dB. That 1.5–3 dB gap is not noise. It is the difference between a confident PASS and an unpredictable margin that erodes with temperature, aging, and splice degradation.

This article explains why this mistake happens, what FEC actually does to your OSNR budget, and how to apply it correctly.

What FEC Does for a Coherent Link

Forward Error Correction works by adding redundant overhead bits to the transmitted data stream. The receiver uses this redundancy to detect and correct bit errors introduced by the optical channel — primarily by OSNR degradation from accumulated ASE noise and non-linear impairments.

The key metric is the Net Coding Gain (NCG): the difference in dB between the OSNR you need for error-free operation without FEC and the OSNR you need with FEC. Modern soft-decision FEC (sFEC) with 20% overhead achieves NCG of approximately 10–11 dB.

This means:

• Raw DP-16QAM without any FEC requires roughly 37–38 dB OSNR for error-free operation (BER < 10⁻¹⁵).
• With sFEC 20% overhead (NCG ≈ 11 dB), the effective threshold drops to approximately 26–27 dB.
• With probabilistic shaping (PS-16QAM), an additional ~1.5 dB improvement brings it to approximately 24.5–25.5 dB.

This effective threshold — 25.5 dB for 800G PS-16QAM — is what the transponder actually needs at its input to deliver error-free output. The FEC overhead is already doing its job. The 25.5 dB is not "before FEC" — it is the minimum OSNR at which the FEC can still correct errors.

Pre-FEC vs. Post-FEC: The Source of Confusion

The confusion arises from how OSNR thresholds are reported. There are two distinct definitions in use:

The OIF 800ZR Implementation Agreement specifies 27.0 dB as the required OSNR — this is the pre-FEC BER threshold (BER ≤ 2×10⁻²). The FEC then corrects from this error-laden signal to error-free output. But 27 dB is already the effective operating minimum with the FEC running. Below 27 dB, the pre-FEC BER exceeds what the FEC can correct, and the link fails.

sFEC 20–27%: What the Overhead Actually Means

FEC overhead percentage refers to the ratio of redundancy bits to payload bits. A 20% overhead FEC transmits 1.2 bits for every 1 payload bit, consuming 20% of your line rate for error correction.

• Hard-decision FEC (HD-FEC, ~7% overhead): NCG ≈ 8–9 dB. Used in older 10G/40G systems.
• Soft-decision FEC (SD-FEC / sFEC, 15–20% overhead): NCG ≈ 10–11 dB. Standard for 100G+ coherent systems.
• oFEC (Open FEC, ~15% overhead): NCG ≈ 11.6 dB. Defined in OIF 400ZR and 800ZR standards.

Higher overhead does not linearly increase coding gain. Going from 20% to 27% overhead typically provides less than 1 dB additional NCG. The main reason to use higher overhead is to handle more burst errors or to get a flatter BER curve, not to dramatically reduce the OSNR requirement.

How to Apply This in Your Link Budget

The correct methodology is straightforward once you understand the pre-FEC threshold concept:

• Obtain the minimum required OSNR from your vendor's datasheet for the specific format, baud rate, and FEC type in use. This value already accounts for FEC correction.
• Calculate your link's achieved OSNR using the cascaded amplifier noise accumulation formula.
• Compare directly: Margin = Achieved OSNR − Required OSNR
• Do not subtract FEC gain again. It is already included in the required OSNR.

Non-linear impairment penalties (SPM, XPM, FWM on G.653/G.655 fiber) are additive: they increase the effective required OSNR above the baseline. A link on G.655 NZDSF may need an additional 1.5–3.0 dB added to the required OSNR to account for the non-linear degradation.

Why Vendor Values Can Differ from Standards

You may notice that real-world vendor thresholds (like our 25.5 dB for 800G PS-16QAM) differ from what a direct calculation from the OIF 800ZR spec might suggest. This is normal and expected:

• Implementation margin: Vendors add an internal margin to their specifications to ensure reliability over operating temperature and equipment lifetime.
• Baud rate differences: Our system runs at 135 GBaud in a 150 GHz slot. OIF 800ZR is defined at different parameters.
• Constellation optimization: The exact PS shaping gain depends on implementation. 1.5 dB is typical but not universal.

Always use field-observed or vendor-specified thresholds in your link budget, not theoretical calculations alone. If you have seen your links fail at a specific OSNR level, that is your design threshold.

Key Takeaways