This KB explains about the trouble-shooting steps for fabric link error on PTX 10K4/8/16 EVO platforms.
CRC errors on fabric links are generated due to HW issues on FPC or SIB. In this example, CRC errors are detected on the fabric plane19 between SIB3 and FPC#2/PFE#3. user@ptx10008-re0> show system alarms 1 alarms currently active Alarm time Class Description 2023-07-11 03:31:57 JST Major SIB 3 FPC Link Errors user@ptx10008-re0> show system errors active detail System Active Errors Detail Information SIB 3 ---------------------------------------------------------------- Error Name : sib_link_to_fpc_fault Identifier : /sib/3/fabspoked-fchip/3/cm/0/fchip/0/sib_link_to_fpc_fault Description : sib_link_to_fpc_fault State : enabled Scope : board Category : internal Level : major Threshold : 1 Error limit : 144 Occur count : 1 Clear count : 0 Last occurred(ms ago) : 2023-07-11 03:31:57 JST (758100 ms ago)
*** messages *** Jul 11 03:00:49 ptx10008-re0 fabspoked-fchip[13985]: %USER-3-CCL: 2354 CRC errors seen on link SIB3_ZF_0-link-rx10-sc0-192 Jul 11 03:00:50 ptx10008-re0 fabspoked-fchip: %USER-3-FM_LINK_ERROR: Message : SIB 3 Pfe[3]/Plane[19] link Id 0x20030080000c08 asic id 0x200300RX endpoint id 0x80000c08 has Link fault Jul 11 03:00:50 ptx10008-re0 fabspoked-fchip[13985]: %USER-3-CCL: 3450 CRC errors seen on link SIB3_ZF_0-link-rx10-sc0-192 Jul 11 03:00:50 ptx10008-re0 fabspoked-fchip[13985]: %USER-5-CCL: Frame lock loss on SIB3_ZF_0-link-rx10-sc0-192 Jul 11 03:00:50 ptx10008-re0 fabspoked-fchip[13985]: %USER-5-CCL: Frame alignment loss on SIB3_ZF_0-link-rx10-sc0-192 Jul 11 03:00:50 ptx10008-re0 fabspoked-fchip[13985]: %USER-3-CCL: ccl_rx_inst_scan_fn: Disconnecting Rx SC SIB3_ZF_0-chan-rx10-sc0 from core Jul 11 03:00:50 ptx10008-re0 resiliencyd[18178]: %USER-5: CMError: /sib/3/fabspoked-fchip/3/cm/0/fchip/0/sib_link_to_fpc_fault (0x400002), scope: board, category: internal, severity: major, module: fab-fchip@3, type: sib_link_to_fpc_fault Jul 11 03:00:50 ptx10008-re0 resiliencyd[18178]: %USER-5: Performing action cmalarm for error /sib/3/fabspoked-fchip/3/cm/0/fchip/0/sib_link_to_fpc_fault (0x400002) in module: fab-fchip@3 with scope: board category: internal level: major Jul 11 03:00:50 ptx10008-re0 resiliencyd[18178]: %USER-5-EMF_EVO_ALARM_SET: Alarm set: SIB color=red, class=CHASSIS, reason=SIB 3 FPC Link Errors Jul 11 03:00:50 ptx10008-re0 resiliencyd[18178]: %USER-5: Performing action trap for error /sib/3/fabspoked-fchip/3/cm/0/fchip/0/sib_link_to_fpc_fault (0x400002) in module: fab-fchip@3 with scope: board category: internal level: major Jul 11 03:00:50 ptx10008-re0 resiliencyd[18178]: %USER-3-RESILD_PD_TRAP_ACTION: Trap raised for fabric_traps_sib_to_fpc_link_fault on /Chassis[0]/Sib[3] Jul 11 03:00:50 ptx10008-re0 resiliencyd[18178]: %USER-3: Cmerror Op Sub Set: fab-fchip@3: Link Set Err (URI: /sib/3/fabspoked-fchip/3/cm/0/fchip/0/sib_link_to_fpc_fault) Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: Logging CCL statistics for BT1_DP1-chan-tx0-sc6 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: SOT:0x0000000966159707 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: Link:0 FrameCnt:0x0000000000017053 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: Logging errors for BT1_DP1-chan-tx0-sc6 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: Logging SerDes statistics for BT1_DP1-chan-tx0-sc6 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: Logging CCL statistics for BT1_DP1-chan-rx0-sc6 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: SOT:0x0000000000000086 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: FrameCnt:0x0000000000015576 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: LastCRCErrCnt:0x00000000 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: AggrCRCErrCnt:0x0000000000000000 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: AggrBERCnt:0x0000000000000000 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: BT1_DP1-gf-fx14p-link-6-1610612742 CRC error history (last 5 polls): Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: 0x0 0x0 0x0 0x0 0x0 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: FEC Uncorrectable FEC Correctable Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: 00000000, 00000057 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: 00000000, 00005497 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: 00000000, 00000000 Jul 11 03:00:50 ptx10008-fpc2 last message repeated 8 times Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: Logging errors for BT1_DP1-chan-rx0-sc6 Jul 11 03:00:50 ptx10008-fpc2 fabspoked-pfe[18323]: %USER-3-CCL: Logging SerDes statistics for BT1_DP1-chan-rx0-sc6
user@ptx10008-re0> show chassis sibs detail Slot 0 information: State Online Uptime 16 minutes, 55 seconds Fabric links Active Errors None Slot 1 information: State Online Uptime 16 minutes, 58 seconds Fabric links Active Errors None Slot 2 information: State Online Uptime 16 minutes, 59 seconds Fabric links Active Errors None Slot 3 information: State Online Uptime 16 minutes, 56 seconds Fabric links Active Errors Link Errors Slot 4 information: State Online Uptime 16 minutes, 48 seconds Fabric links Active Errors None Slot 5 information: State Online Uptime 16 minutes, 49 seconds Fabric links Active Errors None user@ptx10008-re0> show chassis fabric fpcs Fabric management FPC state: FPC #2 PFE #3 SIB0_Asic0_Fcore0 (plane 0) Plane Enabled, Links ok Fabric OAM success SIB0_Asic0_Fcore0 (plane 1) Plane Enabled, Links ok Fabric OAM success SIB0_Asic1_Fcore0 (plane 2) Plane Enabled, Links ok Fabric OAM success SIB0_Asic1_Fcore0 (plane 3) Plane Enabled, Links ok Fabric OAM success SIB0_Asic2_Fcore0 (plane 4) Plane Enabled, Links ok Fabric OAM success SIB0_Asic2_Fcore0 (plane 5) Plane Enabled, Links ok Fabric OAM success SIB1_Asic0_Fcore0 (plane 6) Plane Enabled, Links ok Fabric OAM success SIB1_Asic0_Fcore0 (plane 7) Plane Enabled, Links ok Fabric OAM success SIB1_Asic1_Fcore0 (plane 8) Plane Enabled, Links ok Fabric OAM success SIB1_Asic1_Fcore0 (plane 9) Plane Enabled, Links ok Fabric OAM success SIB1_Asic2_Fcore0 (plane 10) Plane Enabled, Links ok Fabric OAM success SIB1_Asic2_Fcore0 (plane 11) Plane Enabled, Links ok Fabric OAM success SIB2_Asic0_Fcore0 (plane 12) Plane Enabled, Links ok Fabric OAM success SIB2_Asic0_Fcore0 (plane 13) Plane Enabled, Links ok Fabric OAM success SIB2_Asic1_Fcore0 (plane 14) Plane Enabled, Links ok Fabric OAM success SIB2_Asic1_Fcore0 (plane 15) Plane Enabled, Links ok Fabric OAM success SIB2_Asic2_Fcore0 (plane 16) Plane Enabled, Links ok Fabric OAM success SIB2_Asic2_Fcore0 (plane 17) Plane Enabled, Links ok Fabric OAM success SIB3_Asic0_Fcore0 (plane 18) Plane Enabled, Links ok Fabric OAM success SIB3_Asic0_Fcore0 (plane 19) Plane Disabled, Links ok Fabric OAM success SIB3_Asic1_Fcore0 (plane 20) Plane Enabled, Links ok Fabric OAM success SIB3_Asic1_Fcore0 (plane 21) Plane Enabled, Links ok Fabric OAM success user@ptx10008-re0> show chassis fabric sibs Fabric management SIB state: SIB #3 Online Asic #0 Fcore #0 (plane 19) Active FPC #0 PFE #0 : Links ok PFE #1 : Links ok PFE #2 : Links ok PFE #3 : Links ok PFE #4 : Links ok PFE #5 : Links ok PFE #6 : Links ok PFE #7 : Links ok PFE #8 : Links ok FPC #1 PFE #0 : Links ok PFE #1 : Links ok PFE #2 : Links ok PFE #3 : Links ok PFE #4 : Links ok PFE #5 : Links ok PFE #6 : Links ok PFE #7 : Links ok PFE #8 : Links ok FPC #2 PFE #0 : Links ok PFE #1 : Links ok PFE #2 : Links ok PFE #3 : Links Error PFE #4 : Links ok PFE #5 : Links ok PFE #6 : Links ok PFE #7 : Links ok PFE #8 : Links ok user@ptx10008-re0> show chassis fabric topology In-link : FPC# FE# (TX inst#, TX sub-chnl #) -> SIB# ASIC#_FCORE# (RX port#,RX sub-chn#, RX inst#) Out-link : SIB# ASIC#_FCORE#(TX port#, TX sub-chn#, TX inst#) -> FPC# FE# (RX inst#, RX sub-chnl #) SIB 3 FCHIP 0 FCORE 0 : ----------------------- In-links State Out-links State -------------------------------------------------------------------------------- FPC02FE1(1,03)->S03F0_0(07,2,07) UP S03F0_0(07,2,07)->FPC02FE1(1,03) UP FPC02FE7(1,00)->S03F0_0(08,0,08) UP S03F0_0(08,0,08)->FPC02FE7(1,00) UP FPC02FE2(3,05)->S03F0_0(08,2,08) UP S03F0_0(08,2,08)->FPC02FE2(3,05) UP FPC02FE2(3,07)->S03F0_0(09,0,09) UP S03F0_0(09,0,09)->FPC02FE2(3,07) UP FPC02FE1(1,05)->S03F0_0(09,2,09) UP S03F0_0(09,2,09)->FPC02FE1(1,05) UP FPC02FE3(0,06)->S03F0_0(10,0,10) FAULT S03F0_0(10,0,10)->FPC02FE3(0,06) UP FPC02FE5(0,07)->S03F0_0(10,1,10) UP S03F0_0(10,1,10)->FPC02FE5(0,07) UP FPC02FE8(0,05)->S03F0_0(10,2,10) UP S03F0_0(10,2,10)->FPC02FE8(0,05) UP
user@ptx10008-re0> show chassis fabric degraded-fabric-reachability Degraded Fabric reachability Information: FPC #2 PFE #0 SIB3_Plane19 Link errors FPC/PFEs 2/3 PFE #1 SIB3_Plane19 Link errors FPC/PFEs 2/3 PFE #2 SIB3_Plane19 Link errors FPC/PFEs 2/3 PFE #3 SIB3_Plane19 Link errors FPC/PFEs 0/0 0/1 0/2 0/3 0/4 0/5 0/6 0/7 0/8 1/0 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 2/0 2/1 2/2 2/3 2/4 2/5 2/6 2/7 2/8 4/0 4/1 4/2 4/3 4/4 4/5 4/6 4/7 4/8 PFE #4 SIB3_Plane19 Link errors FPC/PFEs 2/3
Here is the log collection list for basic debug.show versionshow log messagesshow chassis hardwareshow chassis alarmsshow system alarms show system errors active detailshow system errors inactive detailshow chassis sibs detailshow chassis fabric summaryshow chassis fabric topologyshow chassis fabric fpcsshow chassis fabric sibsshow chassis fabric errors autohealshow chassis fabric plane-locationshow chassis fabric degraded-fabric-reachabilityshow chassis fabric reachabilityshow chassis fabric reachability detail
show system fabric app-status
The output of the script /usr/sbin/chk_lnks.sh with root access
1. Execute /usr/sbin/chk_lnks.sh on master RE shell with root permission. Collect .tar.gz from /var/tmp - it is date-timestamped.
user@ptx10008-re0> start shell user root Password: [vrf:none] root@ptx10008-re0:~# /usr/sbin/chk_lnks.shoutput will be stored in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30 SIB0: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB1: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB2: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB3: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB4: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB5: AID: 0x0d08, SIB8/other-type NUM-ZF:3 FPC0: AID:0x0d07, Scapa-LC, NUM-BT:5 FPC stats are in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30/stats1 FPC1: AID:0x0d07, Scapa-LC, NUM-BT:5 FPC stats are in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30/stats1 FPC2: AID:0x0d07, Scapa-LC, NUM-BT:5 FPC stats are in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30/stats1 FPC4: AID:0x0d07, Scapa-LC, NUM-BT:5 FPC stats are in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30/stats1 Sleeping 5 minutes before dumping stats again SIB0: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB1: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB2: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB3: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB4: AID: 0x0d08, SIB8/other-type NUM-ZF:3 SIB5: AID: 0x0d08, SIB8/other-type NUM-ZF:3 FPC0: AID:0x0d07, Scapa-LC, NUM-BT:5 FPC stats are in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2 FPC1: AID:0x0d07, Scapa-LC, NUM-BT:5 FPC stats are in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2 FPC2: AID:0x0d07, Scapa-LC, NUM-BT:5 FPC stats are in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2 FPC4: AID:0x0d07, Scapa-LC, NUM-BT:5 FPC stats are in /var/tmp/chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2 all done. [vrf:none] root@ptx10008-re0:/var/tmp# ls -la chk_lnks/ total 12 drwxr-xr-x 3 root root 4096 Oct 11 10:12 . drwxrwxrwt 5 root root 4096 Oct 11 10:12 .. drwxr-xr-x 3 root root 4096 Oct 11 10:12 chk_2023Oct11_10_12_30 [vrf:none] root@ptx10008-re0:/var/tmp# tar cvzf chk_lnks.tar.gz chk_lnks/ chk_lnks/ chk_lnks/chk_2023Oct11_10_12_30/ chk_lnks/chk_2023Oct11_10_12_30/scft_chk.txt chk_lnks/chk_2023Oct11_10_12_30/chk_lnks_chk.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/ chk_lnks/chk_2023Oct11_10_12_30/stats1/sib_ccl_stats.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc4_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/ chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/sib_ccl_stats.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc4_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/sib_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc4.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc2.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc2_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc1_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc0.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc0_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc_stats_all.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/stats2/fpc1.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/sib_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc4.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc2.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc2_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc1_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc0.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc0_errs.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc_stats_all.txt chk_lnks/chk_2023Oct11_10_12_30/stats1/fpc1.txt The chk_lnks.tar.gz saved under /var/tmp will help provide the recommendation for next steps.
2. Dump debug_collector from system. Refer KB37257 [juniper.net] for how to dump debug_collector.
root@ptx10008-re0> request system debug-info
3. Link Auto Heal (LAH) automatically attempts for recovery when link faults are detected and reported.When a link goes to fault, autoheal at link level is automatically attempted by software. Once a link fault is detected and reported, fabric resiliency waits 10 seconds, then attempts recovery. Link autoheal is attempted at a {fpc, fabricPfe, plane} level. Recovery (link autoheal) will be attempted for each link 3 attempts within 24 hour window. After 3 attempts, LAH is not attempted for that link. Link will be held in last attempted state where failure happened.CLI command to display LAH attempts and success/failure status:show chassis fabric errors autohealNote: It will not show status of manually triggered healing attempts.CLI command to clear earlier status of LAH attempts:clear chassis fabric errors autohealNote: If link failed to recover after 3 attempts in 24 hours, error records must be cleared before manual healing can be attempted.CLI command to perform manual link healing:request chassis fabric link-autoheal fpc <x> pfe <x> plane <x>Note: We have S/W PR1751581 - CMerror is not raised post LAH(link auto-heal) fails during training failureCheck LAH status and attempt manual healing.Display LAH attempts and statusroot@ptx10008-re0> show chassis fabric errors autohealSIB3_Asic0_Fcore0 (plane 19) Plane Disabled, Links ok Fabric OAM successClear LAH status before attempting manual healingroot@ptx10008-re0> clear chassis fabric errors autohealAttempt Manual link healingroot@ptx10008-re0> request chassis fabric link-autoheal fpc 2 pfe 3 plane 19root@ptx10008-re0> show chassis fabric degraded-fabric-reachabilityDegraded Fabric reachability Information:FPC #2 PFE #0 SIB3_Plane19 Link errors FPC/PFEs 2/3 PFE #1 SIB3_Plane19 Link errors FPC/PFEs 2/3 PFE #2 SIB3_Plane19 Link errors FPC/PFEs 2/3 PFE #3 SIB3_Plane19 Link errors FPC/PFEs 0/0 0/1 0/2 0/3 0/4 0/5 0/6 0/7 0/8 1/0 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 2/0 2/1 2/2 2/3 2/4 2/5 2/6 2/7 2/8 4/0 4/1 4/2 4/3 4/4 4/5 4/6 4/7 4/8
Note: We have S/W PR1751581 - CMerror is not raised post LAH(link auto-heal) fails during training failure.
If Manual link healing does not help, try the following steps during a maintenance window to avoid traffic impact. Check if the CRC errors/link fault are cleared, and identify the faulty card that needs to be replaced.
1. Attempt FPC restart or offline/online for FPC2. If 1 does not help, attempt SIB offline/online for SIB3. If 2 does not help, reboot system4. If 3 does not help, do manual OIR(removal/insertion) FPC / SIB.
FPC restart(or offline/online) via CLI
request chassis fpc slot <x> restart request chassis fpc slot <x> offlinerequest chassis fpc slot <x> online
SIB offline/online via CLI
request chassis sib slot <x> offlinerequest chassis sib slot <x> online
Router reboot via CLI
request system reboot
2024-03-17 : Article Created
2025-03-07 : "show system fabric app-status" is added in CLI log collection list
2025-03-17 : /usr/sbin/chk_lnks.sh is moved under Solution
2025-05-07 : Minor formatted