Description

This article explains the details of finding the reason for the random PE Chip: CRC errors set/clear messages.

Symptoms

There will be random PE Chip CRC error set and clear messages on QFX10008 device

 

snap
====

Dec 10 10:41:48 qfx02-re0 fpc2 CCL: 1 CRC errors seen on link PE4-Avg-28nm-link-10-18
Dec 10 10:41:48 re0 fpc2 fpc2 dcpfe: pechip_cmerror_set_error:3188: Level: Major, cmerror_code: 0x2101aa (id=426), recover_err: 0 chip_id: 4, current count: 0, threshold: 16, (counter: 0), fh_msg: 0x0

Dec 10 10:41:48 re0 fpc2 pechip_cmerror_set_error:3188: Level: Major, cmerror_code: 0x2101aa (id=426), recover_err: 0 chip_id: 4, current count: 0, threshold: 16, (counter: 0), fh_msg: 0x0

Dec 10 10:41:48 re0 fpc2 fpc2 dcpfe: Cmerror Op Set: PE Chip: PE4[4]: EPW: crc error#012 (URI: /fpc/2/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

Dec 10 10:41:48 re0 kernel: qfx02-re0 fpc2 fpc2 dcpfe: Cmerror Op Set: PE Chip: PE4[4]: EPW: crc error#012 (URI: /fpc/2/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

Dec 10 10:41:48 re0 fpc2 Cmerror Op Set: PE Chip: PE4[4]: EPW: crc error (URI: /fpc/2/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

Dec 10 10:41:48 re0 kernel: re0 fpc2 Cmerror Op Set: PE Chip: PE4[4]: EPW: crc error (URI: /fpc/2/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

Dec 10 10:43:10 re0 fpc2 pechip_cmerror_clear_error:3321: Clear cmerror_code = 0x2101aa

Dec 10 10:43:10 re0 fpc2 Cmerror Op Clear: PE Chip: PE4[4]: EPW: clear crc error (URI: /fpc/2/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

Dec 10 10:43:10 re0 kernel: qfx02-re0 fpc2 Cmerror Op Clear: PE Chip: PE4[4]: EPW: clear crc error (URI: /fpc/2/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

Dec 10 10:43:10 re0 fpc2 fpc2 dcpfe: pechip_cmerror_clear_error:3321: Clear cmerror_code = 0x2101aa

Dec 10 10:43:10 re0 fpc2 fpc2 dcpfe: Cmerror Op Clear: PE Chip: PE4[4]: EPW: clear crc error#012 (URI: /fpc/2/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

Dec 10 10:43:10 re0 kernel: re0 fpc2 fpc2 dcpfe: Cmerror Op Clear: PE Chip: PE4[4]: EPW: clear crc error#012 (URI: /fpc/2/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_EPW_MISC_INT_EVENTS_CRC_ERR)

 

Solution

-As the errors are seen on multiple FPCs via few SIB to FPC links, traced the links using fabric health check script output collected from the setup, all the links generating/clearing the errors randomly are from a single SIB

snap from one FPC [FPC2]
======================

From Junos logs for FPC2, the errors are on the link "PE4-Avg-28nm-link-10-18"

re0 fpc2 CCL: 1 CRC errors seen on link PE4-Avg-28nm-link-10-18


From Fabric health check script output, the corresponding link on FPC2 "PE4-Avg-28nm-link-10-18" is from SIB1 

13/12/2023 03:07:29 : (sib1, pf1, tx11, sc2) -> (fpc2, pe4, rx0, sc10)
13/12/2023 03:07:29 : Running cmd: cprod -A fpc2 -c "show ccl statistics detail pe4 rx0 sc10"
13/12/2023 03:07:29 : Name FrameCnt AggrCRCErrCnt LastCRC AggrBERCnt

fec_cor fec_uncor cor_bit_cntr cor_t_max

==========================================================================
PE4-Avg-28nm-link-10-18 0x00000004a99b6afb 0x0000000000000001 0x0 0x0000000000000000
0x000000000039d289 0x00000000000000a8 0x0000000000000000 0x0000000000000000
===========================================================================

-Similarly, all the links creating this issue on other FPCs also coming from a single SIB.
-Replacement of the SIB resolved this issue.
-To collect Fabric health check output, please contact JTAC.

Modification History

2023-12-21 : Article Created

2025-7-30 : Replaced hostname, improved formatting

2025-8-12 : Filled in Modification History