Description

QFX10008 | FPC crash due to "Power trip failure | Bad Voltage" | Traffic black-hole

Symptoms

Power trip failure observed on an FPC due to Bad Voltage but IPC connection is not dropped immediately causing traffic black-hole because interfaces stayed up, even though the FPC was in problem state. 

In the logs below, it is observed that "Power trip failure" was registered at Sep 11 03:17:41 but "Dropped IPC connection for FPC 0" was registered 19 seconds later. 

Logs before the fix: 

Sep 11 03:17:41  Power trip failure seen for Fru slot 0 Fru type FRU_FPC

<>

Sep 11 03:18:00  fpc_disconnect_generic: fpc 0 state Offline cargs 0x99710e0 clean_shutdown 0, offline_reason=Bad Voltage
Sep 11 03:18:00 CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 0

 

Solution

A fix has been implemented to drop the IPC connection immediately when "Power trip failure" is detected.

 

Logs after the fix:

Jun 15 17:29:32  Power trip failure seen for Fru slot 0 Fru type FRU_FPC
Jun 15 17:29:32 CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 0

 

Modification History

2024-03-08 : Article Created