This article details the investigation of PEChip IGP errors seen on QFX10008 and one of the possible causes.
IGP Errors are seen incrementing on one of the PEChips:
root@switch> show pfe statistics error
...
Slot 1
PEChip Errors: 6
--------------
PEChip 0
IGP errs received from IQM/DBM : 0
PEChip 1
PEChip 2
IGP errs received from IQM/DBM : 2640 <<<<<<<< this counter is seen incrementing
PEChip 3
PEChip 4
PEChip 5
CLI command "show pfe statistics error" output shows increasing IGP errors on fpc1's PEChip2:
Syslog messages show numerous counts of entries for a specific interface - et-1/0/10 in this particular example.
This interface flaps frequently, has a high count of CRC alignment errors, pe.igp.intr.status trapcode for PEChip 2 is incremented.
Sep 25 16:42:05.892 2024 switch fpc1 wanio: IFD_UP - et-1/0/10 <<<<<<<<<<< interface flap
Sep 25 16:42:12.010 2024 switch fpc1 INTR: throttle 60sec PECHIP[2]:pe.ipw.misc_int.status:iq_disabled(0): (Count:1)err_pkt(0): (Count:1)iqm_perr(0): (Count:1)preclass_l2_cam_miss(0): (Count:1)
Sep 25 16:42:12.019 2024 switch fpc1 INTR: throttle 60sec PECHIP[2]:pe.igp.intr.status:pkt_trapcode(0): (Count:1) <<<<<<<<<<< IGP err trapcode is incremented
Sep 25 16:42:15.558 2024 switch fpc1 ifp et-1/0/10 ifd_mdown: 7346351 ms
Sep 29 12:47:49.663 2024 switch : %PFE-6: fpc1 pepic_if_mtip_chmac_hw_clear_fcs_stats: CRC Align error equal or exceed max value et-1/0/10 count 113249697660928 <<<<<<<<<<< CRC alignment errors
Sep 29 12:47:49.666 2024 switch : %PFE-6: fpc1 wanio: IFD_UP - et-1/0/10
Sep 29 12:47:49.000 2024 switch : %USER-6: fpc1 fpc1 dcpfe: pepic_if_mtip_chmac_hw_clear_fcs_stats: CRC Align error equal or exceed max value et-1/0/10 count 113249697660928
Sep 29 12:47:49.000 2024 switch : %USER-6: fpc1 fpc1 dcpfe: wanio: IFD_UP - et-1/0/10
Sep 29 12:58:32.930 2024 switch : %PFE-6: fpc1 wanio: asic:2 hw-link-scan IFD_DOWN - et-1/0/10 <<<<<<<<<<< interface flap
Sep 29 12:58:32.000 2024 switch : %USER-6: fpc1 fpc1 dcpfe: wanio: asic:2 hw-link-scan IFD_DOWN - et-1/0/10
Sep 29 12:58:32.000 2024 switch : %USER-6: fpc1 fpc1 dcpfe: ifp et-1/0/10 ifd_mdown: 339522044 ms
Looking at the interface statistics (or in the RSI) with CLI command "show interface <id> extensive", we see that et-1/0/10 statistics were cleared recently (4 days prior in this example) and since then, this interface flapped over 2000 times, tens of thousands of PCS errors were seen, as well as very high counts of framing/CRC align errors.
root@switch> show interface et-1/0/10 extensive
Statistics last cleared: 2024-09-26 07:03:15 +04 (4d 03:45 ago)
Traffic statistics:
Input bytes : 638890686216 2827472 bps
Output bytes : 530745665756 28074048 bps
Input packets: 533234295 1491 pps
Output packets: 464640049 2727 pps
IPv6 transit statistics:
Input bytes : 0
Output bytes : 0
Input packets: 0
Output packets: 0
Input errors:
Errors: 112150186033152, Drops: 0, Framing errors: 112150186033152, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 00 <<<<<<<<<<<<<<
Output errors:
Carrier transitions: 2054, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0 <<<<<<<<<<<<<<
…
PCS statistics Seconds
Bit errors 44135 <<<<<<<<<<<
Errored blocks 44126
Ethernet FEC Mode : NONE
Ethernet FEC statistics Errors
MAC statistics: Receive Transmit
Total octets 638890686216 530745665756
Total packets 533234295 464640049
Unicast packets 532896335 464309227
Broadcast packets 581 0
Multicast packets 329408 330822
CRC/Align errors 112150186033152 0 <<<<<<<<<<<<<
These stats are incremented by a physical layer issue and following steps are recommended when the IGP error counter increments:
Check which pechip shows errors.
Check the interfaces listed under that pechip
Check interface extensive output to see if any interface under that pechip shows framing errors.
Once you find the interface, then rule out physical layer issue by swapping the optic and fiber cable on the receiving end router.
If the error still appears, then swap optic on the sender side.
If the error still appears, then rule out ODF issues - check if / move to a direct connection
If the error continues to appear, then rule out PIC issue buy moving the port to a different port in another slot.
If the above steps resolve the issue, then restart the PIC and check. Otherwise, replace it.
2024-10-30 : Changed formatting
2024-10-02 : Article Created