On PTX3k/5k with Type-3 FPC(FPC3-SFF-PTX-1X) and 10x100GE PIC, we have the following specifications on the Forward Error Correction (FEC) mode:
Although there is a config knob to configure the FEC mode via CLI, there is no need to change the FEC mode at all.
On 10X100GE PIC, there is a signal integrity issue on port 2 with LR4 optics, so the hardware design is different from the other ports. To overcome the signal integrity issue on port 2 in 100G mode, FEC91 is enabled between PFE ASIC and Retimer chip. FEC91 is always enabled on port 2. This will protect the data payload between PFE ASIC and Retimer chip with FEC when optic type is LR4.
Do not use FEC mode NONE on port 2. No CLI configuration is needed. If port 2 is connected to a 10x100G port, do not configure FEC mode on the peer port, because FEC is terminated on the Retimer. The data that comes out of port 2 has no FEC in the data stream.
[1] LR4 type optics
FPC 0 REV 40 750-057064 ACPJ9270 FPC3-SFF-PTX-1X CPU BUILTIN BUILTIN SMPC PMB PIC 0 REV 05 750-070203 ACPN1419 10x100/10x40/40x10GE Xcvr 2 REV 01 740-058732 1GCQA21002X QSFP-100GBASE-LR4 FPC 8 REV 46 750-057064 BCAE8009 FPC3-SFF-PTX-1X CPU BUILTIN BUILTIN SMPC PMB PIC 0 REV 06 750-070203 BCAD9837 10x100/10x40/40x10GE Xcvr 0 REV 01 740-058732 1DJQA2190FN QSFP-100GBASE-LR4 Xcvr 1 REV 01 740-058732 1BTQA218075 QSFP-100GBASE-LR4 Xcvr 2 REV 01 740-058732 1GCQA1160EU QSFP-100GBASE-LR4
FPC 0 REV 40 750-057064 ACPJ9270 FPC3-SFF-PTX-1X CPU BUILTIN BUILTIN SMPC PMB PIC 0 REV 05 750-070203 ACPN1419 10x100/10x40/40x10GE Xcvr 2 REV 01 740-058732 1GCQA21002X QSFP-100GBASE-LR4 FPC 8 REV 46 750-057064 BCAE8009 FPC3-SFF-PTX-1X CPU BUILTIN BUILTIN SMPC PMB PIC 0 REV 06 750-070203 BCAD9837 10x100/10x40/40x10GE Xcvr 0 REV 01 740-058732 1DJQA2190FN
QSFP-100GBASE-LR4 Xcvr 1 REV 01 740-058732 1BTQA218075 QSFP-100GBASE-LR4 Xcvr 2 REV 01 740-058732 1GCQA1160EU QSFP-100GBASE-LR4
The port 2 to port 2 connection uses FEC91 by default in between PFE ASIC and PIC Retimer.
PTX3000 et-0/0/2 -------- Up -------- et-8/0/2 PTX3000 QSFP-100GBASE-LR4 QSFP-100GBASE-LR4 PFE ASIC PIC Retimer LR4 LR4 PIC Retimer PFE ASIC FEC FEC optics optics FEC FEC Enabled -- FEC91 --> Terminated -- >--- no FEC data --->--- Enabled -- FEC91 --> Terminated Terminated <-- FEC91 --- Enabled--- <-- no FEC data ---< --- Terminated <-- FEC91 --- Enabled
PTX3000 et-0/0/2 -------- Up -------- et-8/0/2 PTX3000 QSFP-100GBASE-LR4 QSFP-100GBASE-LR4
PFE ASIC PIC Retimer LR4 LR4 PIC Retimer PFE ASIC FEC FEC optics optics FEC FEC Enabled -- FEC91 --> Terminated -- >--- no FEC data --->--- Enabled -- FEC91 --> Terminated Terminated <-- FEC91 --- Enabled--- <-- no FEC data ---< --- Terminated <-- FEC91 --- Enabled
User@PTX3000> show configuration interfaces
User@PTX3000> show interfaces et-0/0/2 media Physical interface: et-0/0/2, Enabled, Physical link is Up Interface index: 144, SNMP ifIndex: 587 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 4842 Errored blocks 4986 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 10981643 FEC Uncorrected Errors 160411 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
User@PTX3000> show interfaces et-0/0/2 media
Physical interface: et-0/0/2, Enabled, Physical link is Up Interface index: 144, SNMP ifIndex: 587 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 4842 Errored blocks 4986 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 10981643 FEC Uncorrected Errors 160411 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
User@PTX3000> show interfaces et-8/0/2 media Physical interface: et-8/0/2, Enabled, Physical link is Up Interface index: 133, SNMP ifIndex: 554 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 13 Errored blocks 12 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 208198928 FEC Uncorrected Errors 1038527 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
User@PTX3000> show interfaces et-8/0/2 media
Physical interface: et-8/0/2, Enabled, Physical link is Up Interface index: 133, SNMP ifIndex: 554 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 13 Errored blocks 12 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 208198928 FEC Uncorrected Errors 1038527 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
Port 2 uses FEC91 in between PFE ASIC and Retimer, and port 1 uses NONE by default.
PTX3000 et-0/0/2 -------- Up -------- et-8/0/1 PTX3000 QSFP-100GBASE-LR4 QSFP-100GBASE-LR4 PFE ASIC PIC Retimer LR4 LR4 PIC Retimer PFE ASIC FEC91 FEC91 optics optics FEC NONE FEC NONE Enabled --- FEC91 --> Terminated --- >--- no FEC data ---> --------no FEC data ----------- Terminated <-- FEC91 --- Enabled --- <--- no FEC data ---< --------no FEC data -----------
PTX3000 et-0/0/2 -------- Up -------- et-8/0/1 PTX3000
QSFP-100GBASE-LR4 QSFP-100GBASE-LR4
PFE ASIC PIC Retimer LR4 LR4 PIC Retimer PFE ASIC FEC91 FEC91 optics optics FEC NONE FEC NONE Enabled --- FEC91 --> Terminated --- >--- no FEC data ---> --------no FEC data ----------- Terminated <-- FEC91 --- Enabled --- <--- no FEC data ---< --------no FEC data -----------
User@PTX3000> show interfaces et-0/0/2 media Physical interface: et-0/0/2, Enabled, Physical link is Up Interface index: 144, SNMP ifIndex: 587 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 4842 Errored blocks 4986 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 10981643 FEC Uncorrected Errors 160411 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0 User@PTX3000> show interfaces et-8/0/1 media Physical interface: et-8/0/1, Enabled, Physical link is Up Interface index: 143, SNMP ifIndex: 553 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : NONE Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
User@PTX3000> show interfaces et-8/0/1 media
Physical interface: et-8/0/1, Enabled, Physical link is Up Interface index: 143, SNMP ifIndex: 553 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : NONE Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
Port 2 uses FEC91 in between PFE ASIC and Retimer.
Data that comes out of port 2 has no FEC in the data stream because FEC is terminated on the Retimer on port 2. If on the peer port connected to port 2 is FEC enabled, the link will be DOWN due to FEC mode mismatch, because the peer expects to receive FEC data.
User@PTX3000> show configuration interfaces et-8/0/1 { gigether-options { fec fec91; } } User@PTX3000> show interfaces et-0/0/2 media Physical interface: et-0/0/2, Enabled, Physical link is Down Interface index: 144, SNMP ifIndex: 587 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Down Active alarms : LINK Active defects : LINK , LOCAL-FAULT PCS statistics Seconds Bit errors 20 Errored blocks 90 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 48596652 FEC Uncorrected Errors 487695 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0 User@PTX3000> show interfaces et-8/0/1 media Physical interface: et-8/0/1, Enabled, Physical link is Down Interface index: 143, SNMP ifIndex: 553 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Down Active alarms : LINK Active defects : LINK , LOCAL-FAULT PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
User@PTX3000> show configuration interfaces et-8/0/1 { gigether-options { fec fec91; } }
Physical interface: et-0/0/2, Enabled, Physical link is Down Interface index: 144, SNMP ifIndex: 587 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Down Active alarms : LINK Active defects : LINK , LOCAL-FAULT PCS statistics Seconds Bit errors 20 Errored blocks 90 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 48596652 FEC Uncorrected Errors 487695 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
Physical interface: et-8/0/1, Enabled, Physical link is Down Interface index: 143, SNMP ifIndex: 553 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Down Active alarms : LINK Active defects : LINK , LOCAL-FAULT PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
The non port 2 to non port 2 connection do not use FEC.
User@PTX3000> show interfaces et-8/0/0 extensive Physical interface: et-8/0/0, Enabled, Physical link is Up Interface index: 129, SNMP ifIndex: 504, Generation: 143 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 2 Errored blocks 9 Ethernet FEC Mode : NONE Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
User@PTX3000> show interfaces et-8/0/0 extensive
Physical interface: et-8/0/0, Enabled, Physical link is Up Interface index: 129, SNMP ifIndex: 504, Generation: 143 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 2 Errored blocks 9 Ethernet FEC Mode : NONE Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
User@PTX3000> show interfaces et-8/0/1 extensive Physical interface: et-8/0/1, Enabled, Physical link is Up Interface index: 143, SNMP ifIndex: 553 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : NONE Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
User@PTX3000> show interfaces et-8/0/1 extensive
Physical interface: et-8/0/1, Enabled, Physical link is Up Interface index: 143, SNMP ifIndex: 553 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : NONE Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
[2] SR4 type optics
For SR4 opotics, FEC is enabled by default on all ports, and we turn off the FEC on the Retimer. This will maintain the end to end FEC (far end port need to have FEC enabled). If FEC is configured to NONE on an end, the link will be Down due to FEC mode mismatch.
User@PTX3000> show interfaces et-0/0/2 media Physical interface: et-0/0/2, Enabled, Physical link is Up Interface index: 134, SNMP ifIndex: 587 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 2752 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 45 FEC Uncorrected Errors Rate 0
Physical interface: et-0/0/2, Enabled, Physical link is Up Interface index: 134, SNMP ifIndex: 587 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running
Active alarms : None Active defects : None PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 2752 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 45 FEC Uncorrected Errors Rate 0
User@PTX3000> show interfaces et-0/0/6 media Physical interface: et-0/0/6, Enabled, Physical link is Up Interface index: 138, SNMP ifIndex: 591 Link-level type: Ethernet, MTU: 1514, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled Device flags : Present Running Active alarms : None Active defects : None PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC Mode : FEC91 Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0
The PIC (10x100/10x40/40x10GE) has a signal integrity issue on port 2 in 100Gb mode with LR4 optics. In order to make this port work reliably, the hardware design is different from the other ports.
Junos software has made a change to run RS-FEC between PFE ASIC and Retimer. RS-FEC is always enabled on port 2 to protect the link between ASIC and Retimer.
If we do not enable FEC, then the link becomes unstable. Do not use CLI to change FEC mode on this port. It will cause issues. If we connect port 2 to another 10x100G port, we must not configure FEC on the peer, and it will work correctly; if you set FEC, it will break this link.
If SR4 type optics is installed, the Retimer by default will not have FEC enabled, and FEC end to end will be activated. For non 100G operation, FEC will not be activated.
Normally, PTX3k/5k users do not need to specify which FEC mode they want to use via CLI. The driver software does all the work and sets FEC mode as needed.