Due to SIB ASIC issue on fabric, reorder/past cell timeouts will be seen on PFE (Packet Forwarding Engine). Reorder timeouts can cause EPW errors. These errors are not auto-cleared on a couple of FPCs.

A hardware issue was identified (PF asic issue) on a SIB where the interrupt "pf.pf_outqd_reg.intr.status" counter was non-zero. 

$ grep -Ri "pf.pf_outqd_reg.intr.status" * | grep -v 00000000 pfchip_interrupt_cprod_outputs/sib1_pf_4_intr.txt:0x03001080

pf.pf_outqd_reg.intr.status 00004000 <<<<<<<< 00004000 instead of 00000000 

This interrupt is fatal and the software does not poll this register in 20.4R3-S1.3.

If major errors "PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR (0x2101aa)" are reported, please engage JTAC for further investigations.

Below errors are observed when this issue occurs:



<timestamp> <hostname> fpc1 fpc1 dcpfe: Cmerror Op Set: PE Chip: PE0[0]: EPW: crc error#012 (URI: /fpc/1/pfe/0/cm/0/PE_Chip/0/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

<timestamp> <hostname> fpc1 fpc1 dcpfe: Cmerror Op Set: PE Chip: PE1[1]: EPW: crc error#012 (URI: /fpc/1/pfe/0/cm/0/PE_Chip/1/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

The Engineering team identified a hardware issue (PF asic issue) on SIB1 (PFCHIP 4) where the interrupt "pf.pf_outqd_reg.intr.status" counter was non-zero. 

Root cause:

On all inserted FPCs of Junos based QFX10K8/QFX10K16 platforms,due to SIB (Switch Interface Board) ASIC (Application-Specific Integrated Circuit) issue on fabric, packets are getting dropped and major errors "PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR (0x2101aa)" are reported. These errors are not auto-cleared on a couple of FPCs.

Trigger 

This issue might be seen if the following conditions are met:

* Junos based QFX10K8/QFX10K16 platforms

* When bad SIB ASIC is present 

Fix:

A fix has been implemented to offline SIB when this issue occurs. This fix is in 20.4R3-S8 and future code versions.