The FPC line card remains offline despite multiple attempts to reboot the line card. The issue is characterized by communication errors between the chassis and the FPC, causing the FPC to halt.
1) FPC remains offline with error: Chassis connection dropped while checking the FPC status.
>show chassis fpc Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%) Slot State (C) Total Interrupt 1min 5min 15min DRAM (MB) Heap Buffer 0 Empty 1 Offline ---Chassis connection dropped--- 2 Online 45 17 1 15 14 14 3168
>show chassis fpc
Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt 1min 5min 15min DRAM (MB) Heap Buffer
0 Empty
1 Offline ---Chassis connection dropped---
2 Online 45 17 1 15 14 14 3168
2) FPC may not come online even after reboot of the line card.
3) Below logs may be observed along with hard error on the FPC.
send: red alarm set, device FPC 1, reason FPC 1 Hard errors CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Failed (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: MPC7E 3D MRATE-12xQSFPP-XGE-XLGE-CGE @ 1/*/*, jnxFruType 3, jnxFruSlot 1) rcv: ch_ipc_dispatch() null ipc read for args 0xa152b40 pipe 0x9d0d780 ch_connection_shutdown: Destroying the IPC pipefru_nmi_timer: NMI timer expired for FPC 1i2cs_fru_graceful_offline_prepare: FPC#1 prepare graceful offlinefru_send_graceful_offline: Invoking rsh -Ji -t 1 fpc1 /sbin/halt fru_send_graceful_offline: shutdown command on FPC 1 failed CHASSISD_FRU_NMI_RESTART: fru_nmi_timer: Restart FPC 1 due to NMI timeout acb_update_local_esw_status: ESW port 1 (connected to FPC1) link up acb_update_local_esw_status: ESW port 1 (connected to FPC1) link down acb_update_local_esw_status: ESW port 1 (connected to FPC1) link up
send: red alarm set, device FPC 1, reason FPC 1 Hard errors CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Failed (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: MPC7E 3D MRATE-12xQSFPP-XGE-XLGE-CGE @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
rcv: ch_ipc_dispatch() null ipc read for args 0xa152b40 pipe 0x9d0d780
ch_connection_shutdown: Destroying the IPC pipefru_nmi_timer: NMI timer expired for FPC 1i2cs_fru_graceful_offline_prepare: FPC#1 prepare graceful offlinefru_send_graceful_offline: Invoking rsh -Ji -t 1 fpc1 /sbin/halt fru_send_graceful_offline: shutdown command on FPC 1 failed
CHASSISD_FRU_NMI_RESTART: fru_nmi_timer: Restart FPC 1 due to NMI timeout
acb_update_local_esw_status: ESW port 1 (connected to FPC1) link up
acb_update_local_esw_status: ESW port 1 (connected to FPC1) link down
Non-mask able interrupt (NMI) is a hardware interrupt that standard interrupt-masking techniques in the system cannot ignore.It typically occurs to signal attention for non-recoverable hardware errors. It basically means chassisd communication with FPC was broken and that caused FPC to halt. This message is typically a symptom of a component having issues and being restarted in an attempt to recover.
If FPC restart is already performed then please reach out to Juniper Technical Support Center for further investigation.