<MX104> Faulty MIC can cause continuous afeb resets, bringing down all the FPCs
Due to the architecture of MX104, a single faulty PIC could cause afeb to reset continuously, bringing down all the other FPCs.
Here are a few snippets of outputs, outlining an instance where IXCHIP 0 located in PIC 0 was faulty but not completely down, resulting in IXCHIP hogging [this can also be accompanied by HSL2 hogging, as they are the ones that connect PIC to Fabric], thereby crashing afeb and generating core dumps:
root@router> show system core-dumps
-rw-r--r-- 1 root wheel 55875228 Feb 24 01:47 /var/crash/core-router-0.gz.core.0
-rw-r--r-- 1 root wheel 55314147 Feb 24 01:50 /var/crash/core-router-0.gz.core.1
-rw-r--r-- 1 root wheel 55204126 Feb 24 01:55 /var/crash/core-router-0.gz.core.2
-rw-r--r-- 1 root wheel 55448293 Feb 24 02:00 /var/crash/core-router-0.gz.core.3
-rw-r--r-- 1 root wheel 55352738 Feb 24 07:54 /var/crash/core-router-0.gz.core.4
Logs only indicate that the connection to afeb is getting lost, which in turn resulted in FPCs being offline and do not directly point to any specific PIC issue:
Feb 24 04:11:14 router : %PFE-5: afeb0 Power off ports setting: 0
Feb 24 04:11:15 router : %PFE-3: afeb0 PCI ERROR: 0:0:0:0 Timestamp 107060 msec.
Feb 24 04:11:20 router : %PFE-4: afeb0 WDDI Log Messages Enabled
Feb 24 04:11:20 router : %PFE-4: afeb0 Acquire memory information
Feb 24 04:11:20 router : %PFE-4: afeb0 Notes:
Feb 24 04:11:20 router : %PFE-4: afeb0 Bus number: 0 ==> param, 1 ==> packet, 2 ==> internal, 3 ==> host
Feb 24 04:11:20 router : %PFE-4: afeb0 Bank number <==> physical chip select
Feb 24 04:11:20 router : %PFE-4: afeb0 32-bit Physical base address + Address offset == Address used by host
Feb 24 04:14:12 router : %PFE-3: afeb0 Ixchip(0): pio_handle(0x482b6030); pio_read_u32() failed: 1(generic failure)! ix_egr_buf_mgr ix_egr_buf_mgr_error_cnt-addr=002a6128
Feb 24 04:13:04 router chassisd[2189]: %DAEMON-3-CHASSISD_PIC_CMD_TIMEOUT: fpc_pic_online_ack_timeout: attempt to bring PIC 0 in FPC 0 online timed out
Feb 24 04:13:04 router chassisd[2189]: %DAEMON-3-CHASSISD_PIC_CMD_TIMEOUT: fpc_pic_online_ack_timeout: attempt to bring PIC 2 in FPC 0 online timed out
Feb 24 04:13:04 router chassisd[2189]: %DAEMON-3-CHASSISD_PIC_CMD_TIMEOUT: fpc_pic_online_ack_timeout: attempt to bring PIC 0 in FPC 1 online timed out
Feb 24 04:13:04 router chassisd[2189]: %DAEMON-3-CHASSISD_PIC_CMD_TIMEOUT: fpc_pic_online_ack_timeout: attempt to bring PIC 2 in FPC 1 online timed out
Feb 24 04:13:04 router chassisd[2189]: %DAEMON-3-CHASSISD_PIC_CMD_TIMEOUT: fpc_pic_online_ack_timeout: attempt to bring PIC 0 in FPC 2 online timed out
Feb 24 04:14:19 router chassisd[2189]: %DAEMON-4-CHASSISD_SHUTDOWN_NOTICE: Shutdown reason: AFEB connection lost
Feb 24 04:14:19 router chassisd[2189]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC @ 0/*/*, jnxFruType 3, jnxFruSlot 0, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Feb 24 04:14:19 notify_fru_remove: Invalid IDEEPROM data for MIC 0
Feb 24 04:14:19 notify_fru_remove: Invalid IDEEPROM data for MIC 1
Feb 24 04:14:19 router chassisd[2189]: %DAEMON-5-CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 20, jnxFruL1Index 1, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName , jnxFruType 11, jnxFruSlot 0)
Feb 24 04:14:19 router chassisd[2189]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC @ 1/*/*, jnxFruType 3, jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Feb 24 04:14:19 router chassisd[2189]: %DAEMON-5-CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 20, jnxFruL1Index 2, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName , jnxFruType 11, jnxFruSlot 1)
Feb 24 04:31:47 CHASSISD_SHUTDOWN_NOTICE: Shutdown reason: AFEB connection lost
Feb 24 04:31:47 CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC @ 0/*/*, jnxFruType 3, jnxFruSlot 0, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Feb 24 04:31:47 CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC @ 1/*/*, jnxFruType 3, jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Feb 24 04:31:47 CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 3, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC @ 2/*/*, jnxFruType 3, jnxFruSlot 2, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Feb 24 04:31:47 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 20, jnxFruL1Index 3, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName , jnxFruType 11, jnxFruSlot 2)
AFEB/FPCs only stabilise after PIC 0 goes hard down:
root@router> show chassis fpc pic-status
Slot 0 Online
PIC 0 Present 16x CHE1T1, RJ48- Hardware Error
PIC 2 Online 16x CHE1T1, RJ48
Slot 1 Online
PIC 0 Online 16x CHE1T1, RJ48
Slot 2 Online
PIC 0 Online 4x 10GE(LAN) SFP+
root@router> show chassis fpc
Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt 1min 5min 15min DRAM (MB) Heap Buffer
0 Online 57 10 0 2048 24 13
1 Online 57 10 0 2048 24 13
2 Online 57 10 0 2048 24 13
root@router> start shell pfe network afeb0 --> IXCHIP[0] is missing after PIC 0 is hard down
MX platform (1000Mhz QorIQ P2020 processor, 2048MB memory, 512KB flash)
MX104(router vty)# sh jspec client
ID Name
1 LUCHIP[0]
2 QXCHIP[0]
3 MQCHIP[0]
5 IXCHIP[2]
6 IXCHIP[4]
7 IXCHIP[6]
When PIC 0 finally went hard down:
Feb 24 07:59:03 router : %PFE-3: afeb0 Ixchip(0): pio_handle(0x482b6250); pio_read_u32() failed: 1(generic failure)! hsl2-addr=02100014
Feb 24 07:59:03 router : %PFE-3: afeb0 hsl2_hss6g_tx_bringup read macro_cfg.hss_reset 0x2100014 <- 0xbad0bad failed
Feb 24 07:59:03 router : %PFE-3: afeb0 Ixchip(0): pio_handle(0x482b6250); pio_read_u32() failed: 1(generic failure)! hsl2-addr=02100814
Feb 24 07:59:06 router : %PFE-3: afeb0 hsl2_set_hss6g_tx_link_coef: next_tap_0 (0x2) != set val (0x0)
Feb 24 07:59:06 router chassisd[2189]: %DAEMON-4-CHASSISD_PIC_HWERROR: PIC 0 in FPC 0 (PIC type 2635, version 527) had hardware error
Feb 24 07:59:06 router chassisd[2189]: %DAEMON-5-CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Failed (jnxFruContentsIndex 8, jnxFruL1Index 1, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: 16x CHE1T1, RJ48 @ 0/0/*, jnxFruType 11, jnxFruSlot 0)
Feb 24 07:59:06 router chassisd[2189]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 20, jnxFruL1Index 1, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName MIC: 16x CHE1T1, RJ48 @ 0/0/*, jnxFruType 11, jnxFruSlot 0, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn 435409)
Feb 24 07:59:06 router alarmd[2396]: %DAEMON-4: Alarm set: PIC id=1996488705, color=RED, class=CHASSIS, reason=FPC 0 PIC 0 Failure
Feb 24 07:59:06 router craftd[2192]: %DAEMON-4: Major alarm set, FPC 0 PIC 0 Failure
Feb 24 07:59:08 router : %PFE-3: afeb0 Training of serdes 57 failed with a count of 4938
Feb 24 07:59:08 router : %PFE-3: afeb0 BIST failed for MQCHIP(0)-Avago 65NM-link-88 link 57
Feb 24 07:59:08 router : %PFE-3: afeb0 [hsl2_channel_trained_bringup] BIST failed MQCHIP(0)-chan-rx-13
Feb 24 07:59:08 router : %PFE-3: afeb0 HSL2: cmtfpc_hsl2_bringup_ix: hsl2_channel_trained_bringup failed (1), dst_asic 16 dst_chan 13
Feb 24 07:59:09 router : %PFE-3: afeb0 CMT: PIC 0 failed HSL2 bringup, xm/mqchip asic id:16, ixchip asic id:20
Feb 24 07:59:10 router : %PFE-3: afeb0 trinity_pio: 7 PIO errors occurred
Feb 24 07:59:10 router : %PFE-3: afeb0 trinity_pio: Last error: 7 MIC0 IX Trinity PCI 0x00000810 Write PCIe 0
Feb 24 07:59:10 router : %PFE-3: afeb0 wp3_pio_destroy:1252: Bogus cb at 0x48f6a548, magic = 0xc2000000
Feb 24 07:59:10 router : %PFE-3: afeb0 CMT: Error initializing PIC 0, ID 0x0a4b
MIC 0 needs to be replaced
PICs are built into MIC, and in the above case, MIC 0 needs replacement
As part of recovery [until MIC 0 is replaced], offline MIC 0
However, since PICs are inbuilt and this model doesn't support PIC offline, both PIC 0 & 1 will be offline:
root@router> request chassis pic pic-slot 0 fpc-slot 0 offline
FPC slot 0 does not support this command
root@router> request chassis mic mic-slot 0 fpc-slot 0 offline
fpc 0 mic 0 offline initiated, use "show chassis fpc pic-status 0" to verify
PIC 0 Offline 1x 10GE XFP
PIC 1 Offline 1x 10GE XFP