Due to transient or permanent HW errors, log “cmqfx10_sib_fm_periodic” may repeatedly appear from multiple planes. In such scenarios, follow the steps mentioned in this article to narrow down the faulty component causing logs floods/errors.
There is no link marked as faulty in the “show ccl errors” output, whereas syslog has bunch of errors like below:
SW Version and HW
labroot@labrouter-re0> show version
Hostname: labrouter-re0
Model: ptx10016
Junos: 19.2X3.2
No CRC Errors on FPC or SPMB CCL links ie on below outputs
FPC*(labrouter-re0 vty)# show ccl errors
FPC*(labrouter-re0 vty)# show ccl statistics
SPMB*(labrouter-re0 vty)# show ccl errors
SPMB*(labrouter-re0 vty)# show ccl statistics
No Link failure in "show chassis fabric topology"
Multiple Planes reporting cmqfx10_sib_fm_periodic logs
Aug 31 08:16:15 2024 labrouter spmb0 cmqfx10_sib_fm_periodic: cmqfx10_asic_scan returned error for plane 29 sib 4 fault_error 0 link_err 16
Aug 31 08:19:12 2024 labrouter spmb0 cmqfx10_sib_fm_periodic: cmqfx10_asic_scan returned error for plane 33 sib 5 fault_error 0 link_err 16
Aug 31 08:20:49 2024 labrouter spmb0 cmqfx10_sib_fm_periodic: cmqfx10_asic_scan returned error for plane 34 sib 5 fault_error 0 link_err 16
Aug 31 08:23:19 2024 labrouter spmb0 cmqfx10_sib_fm_periodic: cmqfx10_asic_scan returned error for plane 5 sib 0 fault_error 0 link_err 16
Aug 31 08:23:19 2024 labrouter spmb0 cmqfx10_sib_fm_periodic: cmqfx10_asic_scan returned error for plane 15 sib 2 fault_error 0 link_err 16
Aug 31 08:23:19 2024 labrouter spmb0 cmqfx10_sib_fm_periodic: cmqfx10_asic_scan returned error for plane 21 sib 3 fault_error 0 link_err 16
...
Multiple FPCs reporting EPW CRC Error Logs
Aug 29 11:22:48 2024 labrouter fpc0 Cmerror Op Set: PE Chip: PE0[0]: EPW: crc error
Aug 29 11:24:09 2024 labrouter fpc0 Cmerror Op Clear: PE Chip: PE0[0]: EPW: clear crc error
Aug 30 04:20:26 2024 labrouter fpc0 Cmerror Op Set: PE Chip: PE0[0]: EPW: crc error
Aug 30 04:21:48 2024 labrouter fpc0 Cmerror Op Clear: PE Chip: PE0[0]: EPW: clear crc error
Aug 30 20:50:17 2024 labrouter fpc2 Cmerror Op Set: PE Chip: PE1[1]: EPW: crc error
Aug 30 20:51:39 2024 labrouter fpc2 Cmerror Op Clear: PE Chip: PE1[1]: EPW: clear crc error
Sep 1 14:55:42 2024 labrouter fpc2 Cmerror Op Set: PE Chip: PE3[3]: EPW: crc error
Sep 1 14:57:04 2024 labrouter fpc2 Cmerror Op Clear: PE Chip: PE3[3]: EPW: clear crc error
Sep 2 05:06:14 2024 labrouter fpc2 Cmerror Op Set: PE Chip: PE5[5]: EPW: crc error
Sep 2 05:07:36 2024 labrouter fpc2 Cmerror Op Clear: PE Chip: PE5[5]: EPW: clear crc error
Sep 2 07:41:08 2024 labrouter fpc1 Cmerror Op Set: PE Chip: PE2[2]: EPW: crc error
Sep 2 07:42:29 2024 labrouter fpc1 Cmerror Op Clear: PE Chip: PE2[2]: EPW: clear crc error
The error logs "cmqfx10_sib_fm_periodic" and "EPW: crc error" are reported by multiple planes (SIBs) and FPCs. These errors are possibly due to a bad SIB or a FPC. However, as mentioned earlier, logs are reported by multiple planes (SIBs) and there isn’t any clue on logs that leads us to the faulty component.
Hence, to narrow down the faulty component, follow below procedures
1] Collect and save "show chassis fabric topology". I saved it on file fabric_topology
2] Collect below output from SPMB for each of the planes reporting error. We need to look for RRAU channel that has incrementing error count. In this example, I am collecting the logs only for Plane 29
labroot@labrouter> request pfe execute target spmb0 command "show pfchip 29 link_errors" | except " 0"
SENT: Ukern command: show pfchip 29 link_errors
<..>
Link error stats for rrau: 11 sub-channel: 4 ():
ccl_err_cnt: 1065806633
frm_err_cnt: 5150065045
undersize_err_cnt: 254667617
crc_err_cnt: 47464435
3] Use below command to identify Plane mapping with SIB
SPMB0(labrouter-re0 vty)# show ccl asics
Name | ASIC#| ID | ID | SerDes| Links| Inst| Chnls| Flags
--------------------+------+-------+-------+-------+------+-----+------+------
SIB4_PF_5 | 29 | 0 | 0 | 40 | 320 | 20 | 320 | 0x9
4] In the above output (Step 2), RRAU index is 11. Use below command to identify the port group associated with RRAU index 11
SPMB0(labrouter-re0 vty)# show pfchip rau_map
Port | RAU-ID Ring-ID | RAU-Index
Group | (Spry Idx)|
----------------------------------------
<…>
17 | 2 3 | 11
5] From the above outputs, we now know that
5.1] Plane 29 maps with SIB4_PF_5 or S04F5
5.2] RRAU Index 11 is mapped with port group 17
5.3] Sub-Channel for RRAU Index 11 is 4
5.4] "show chassis fabric topology" output is saved to file fabric_topology
6] Run below command from a server to grep the FPC SIB mapping. From the below output, we can see TX of FPC08 PFE4 is connecting to SIB4
Username@Servename Downloads % egrep -E -i "\(17\,4\,17\)" fabric_topology| grep S04F5
FPC08FE4(0,07)->S04F5_0(17,4,17) OK S04F5_0(16,4,16)->FPC08FE4(0,07) OK
6.1] SIB / FPC In Link Mapping for reference
In-link : FPC# FE# ASIC# (TX inst#, TX sub-chnl #) ->
SIB# ASIC#_FCORE# (RX port#, RX sub-chn#, RX inst#)
Out-link : SIB# ASIC#_FCORE# (TX port#, TX sub-chn#, TX inst#) ->
FPC# FE# ASIC# (RX inst#, RX sub-chnl #)
7] Prepare "egrep" command for other planes reported log and check if its pointing to a common FPC.
Username@Servename Downloads % egrep -E -i "\(08\,2\,08\)" fabric_topology| grep S01F1
FPC08FE4(0,12)->S01F1_0(08,2,08) OK S01F1_0(07,2,07)->FPC08FE4(0,12) OK
Username@Servename Downloads % egrep -E -i "\(11\,6\,11\)" fabric_topology| grep S01F3
FPC08FE4(0,08)->S01F3_0(11,6,11) OK S01F3_0(11,2,11)->FPC08FE4(0,08) OK
Username@Servename Downloads % egrep -E -i "\(10\,2\,10\)" fabric_topology| grep S01F4
FPC08FE4(0,10)->S01F4_0(10,2,10) OK S01F4_0(11,2,11)->FPC08FE4(0,10) OK
Based on the above outputs, we can confirm that FPC8 is the culprit causing the errors/logs