The customer saw below alarm on the router.
Router> show chassis alarms no-forwarding 4 alarms currently active Alarm time Class Description 2023-10-23 11:25:50 UTC Major FPC 0 Exhaust-B Temp Sensor Access Failed 2023-10-23 11:25:50 UTC Major FPC 0 Exhaust-A Temp Sensor Access Failed 2023-10-23 11:25:50 UTC Major FPC 0 Intake Temp Sensor Access Failed
>> After going through RSI logs, we see as below
Router> show chassis fpc detail Slot 0 information: State Offline Reason Hardware error Total CPU DRAM 0 MB Total RLDRAM 771 MB Total DDR DRAM 18432 MB FIPS Capable False Temperature Failed Max MPC base power consumption 0 Watts Max MIC1 power consumption 0 Watts Max MPC total power consumption 0 Watts High-Performance mode support True PFEs in High-Performance mode None
>> FPC 0 went offline. >> When I went through the chassisd logs I could see power trip failure logs for FPC 0
Oct 23 11:25:41 Power trip failure seen for Fru slot 0 Fru type FRU_FPC
>> FPC 0 went into problem state. FPC 1 started reporting "Fabric request time out" for FPC 0 stream numbers
Oct 23 11:25:41.415 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 8 fab_stream 0, attempting recovery Oct 23 11:25:41.420 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 3 fab_stream 0, attempting recovery Oct 23 11:25:41.434 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 4 fab_stream 0, attempting recovery Oct 23 11:25:41.452 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 5 fab_stream 0, attempting recovery Oct 23 11:25:41.470 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 6 fab_stream 0, attempting recovery Oct 23 11:25:41.484 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 7 fab_stream 0, attempting recovery Oct 23 11:25:41.497 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 8 fab_stream 1, attempting recovery Oct 23 11:25:41.514 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 9 fab_stream 1, attempting recovery Oct 23 11:25:41.528 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 10 fab_stream 1, attempting recovery Oct 23 11:25:41.541 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 11 fab_stream 1, attempting recovery Oct 23 11:25:41.565 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 12 fab_stream 1, attempting recovery Oct 23 11:25:41.590 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 13 fab_stream 1, attempting recovery Oct 23 11:25:41.615 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 14 fab_stream 1, attempting recovery Oct 23 11:25:41.703 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 15 fab_stream 1, attempting recovery Oct 23 11:25:41.783 Router : %PFE-3: fpc1 CMTFPC: Fabric request time out pfe 0 plane 16 fab_stream 1, attempting recovery
>> Later FPC 1 had "unreachable destinations" alarm which was side effect of FPC 0
Oct 23 11:25:44 FH: Bad dest mask for fpc 1 : 0x00000001 Oct 23 11:25:44 ch_tvp_ng_send_alarm_request: fru_type=3, slot=0, reason=26 Oct 23 11:25:44 send: red alarm set, device FPC 1, reason FPC 1 has unreachable destinations Oct 23 11:25:44 FH: fm_hsl2_get_bad_dpc: One bad FPC detected at slot 1 Oct 23 11:25:44 FH: fm_hsl2_mx_detect_fabric_failures: FPC 1 has started blackholing Oct 23 11:25:44 CHASSISD_FM_FABRIC_DEGRADED: DPCs are seeing grant timeouts; System is blackholing Need to attempt fabric healing. Action will be taken after 10 seconds, to address the fabric down condition. Oct 23 11:25:44 FH: fm_mx_detect_fabric_failures: Fabric is down after interfaces have been created. Oct 23 11:25:44 FH: If fabric does not come up within ten seconds, fabric down actions will be initiated Oct 23 11:25:44 fpc_tvp_fpc_info_sendto_transportd: DEBUG---------fpc state 7 != tvp fpc state 5 for i2c_id (0xca9) @ slotd (0) Oct 23 11:25:47 rcv: ch_ipc_dispatch() ETIMEDOUT TCP/IP pipe ipc read for args 0x9f21bc0 pipe 0x987f160, fru FPC 0 errno 60 Oct 23 11:25:47 rcv: ch_ipc_dispatch() ipc pipe read error for args 0x9f21bc0 pipe 0x987f160, fru FPC 0 errno 0 Oct 23 11:25:47 ch_ipc_dispatch: fru FPC offline_reason = Bad Voltage
>> Here FPC 0 was rebooted for Fabric autohealing
Oct 23 11:25:47 ch_connection_shutdown: Destroying the IPC pipe Oct 23 11:25:47 fpc_disconnect_generic: fpc 0 state Offline cargs 0x9f21bc0 clean_shutdown 0, offline_reason=Bad Voltage Oct 23 11:25:47 -- FPC 0, last request 132, state Offline
>> After FPC 0 was rebooting, alarm for FPC 1 got clear which is clear indication that FPC 0 has gone bad.
Oct 23 11:25:49 send: red alarm clear, device FPC 1, reason FPC 1 has unreachable destinations Oct 23 11:25:49 fm_mx_clear_fabric_down_alarms Cleared fabric down alarm for FPC 1
>> FPC 0 reported Temp sensor errors which caused FPC to go hard down
Oct 23 11:25:50 [Alarm SET] Object 143 reason 63 Device FRU FPC 0, reason FPC 0 TSensor Read (FPC 0 Intake Temp Sensor Access Failed), color 1, slot 0 Alarm id 2399141951 Oct 23 11:25:50 send: red alarm set, device FRU FPC 0, reason FPC 0 Intake Temp Sensor Access Failed Oct 23 11:25:50 [Alarm SET] Object 143 reason 66 Device FRU FPC 0, reason FPC 0 TSensor Read (FPC 0 Exhaust-A Temp Sensor Access Failed), color 1, slot 0 Alarm id 2399141954 Oct 23 11:25:50 send: red alarm set, device FRU FPC 0, reason FPC 0 Exhaust-A Temp Sensor Access Failed Oct 23 11:25:50 [Alarm SET] Object 143 reason 69 Device FRU FPC 0, reason FPC 0 TSensor Read (FPC 0 Exhaust-B Temp Sensor Access Failed), color 1, slot 0 Alarm id 2399141957 Oct 23 11:25:50 send: red alarm set, device FRU FPC 0, reason FPC 0 Exhaust-B Temp Sensor Access Failed Oct 23 11:25:54 fru_nmi_timer: NMI timer expired for FPC 0 Oct 23 11:25:54 fru_trigger_nmi: No NMI support for FPC slot 0
>> Below is the time FPC went into offline and did not come up due to hardware error.
Oct 23 11:31:37 FPC 0 offlined for reason Hardware error, not powering up
>> FPC 0 has hard errors, create RMA for FPC 0 in this case.
Once hardware error is seen, create RMA for FPC.