Description

This article explains about the HW Hardware monitoring error on SRX node and how to troubleshoot it.

Symptoms

You will see the below under Hardware Monitoring under 'show chassis cluster status'

Redundancy group: 0 , Failover count: 0
node0 254   primary       no   no    None      
node1 1    secondary      no   no    None      

Redundancy group: 1 , Failover count: 0
node0 200   primary       no   no    None      
node1 0    secondary      no   no    HW    

Solution

If HW Hardware monitoring flag observed in output of "show chassis cluster status" and this HW flag is flapping, then need to check chassisd logs for that node to get more details.

In Chassid logs if you find the below logs,

Sep 5 03:08:13 LCC: ch_info_local_hw_error_blob_update: HWMon rg0_num_errors 0, rg1_num_errors 1
Sep 5 03:08:13 LCC: ch_info_local_hw_error_blob_set: HWMon got RG0 0 flag RG1 1 flag
Sep 5 03:08:13 LCC: ch_srxtvp_ha_failover_on_coredump: srxpfe coredump started
Sep 5 03:08:13 LCC: send_ha_led_update: Set the HA LED to (2)
Sep 5 03:09:43 LCC: rcv: ch_ipc_dispatch() ipc pipe read error for args <> pipe <>, fru FWDD 0 subfru 0 errno 0
Sep 5 03:09:43 LCC: ch_connection_shutdown: Destroying the IPC pipe
Sep 5 03:09:43 LCC: -- FWDD 0, last request 0, state Online
Sep 5 03:09:43 LCC: flowd down detected
Sep 5 03:09:43 LCC: ch_pepsi_flowd_mon: Set flowd_info: 1
Sep 5 03:09:43 CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 0 offline: Error
Sep 5 03:09:43 LCC: fpc_take_offline: Clearing fpc_pfe_init_failed_mask
Sep 5 03:09:43 LCC: fpc_down slot 0 slc_slot 0 reason Error cargs <> pic_start_idx 0 pic_end_idx 7
Sep 5 03:09:43 LCC: fpc_pepsi_disconnect slot is 0
Sep 5 03:09:43 LCC: fpc_offline_now - slot 0, slc_slot 0 reason: Error, error OK transition state 1
Sep 5 03:09:43 CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperDisabled (entPhysicalIndex 24, entStateAdmin 3, entStateAlarm 128)
Sep 5 03:09:43 CHASSISD_SNMP_TRAP0: ENTITY trap generated: entConfigChanged
Sep 5 03:09:43 LCC: notify_fru_power_off: NULL kvpairs for FPC 0
Sep 5 03:09:43 LCC: fru_power_off_generic
Sep 5 03:09:43 LCC: fru_power_off_generic: calling fru_poweroff vector
Sep 5 03:09:43 LCC: Power off FPC 0
Sep 5 03:09:43 CHASSISD_IPC_WRITE_ERR_NO_PIPE: FRU has no connection pipe fru_send_msg FWDD
Sep 5 03:09:43 LCC: send: fwdd, fpc 0 powered off
Sep 5 03:09:43 LCC: fpc_offline_now - slot 0, is_resync_ready cleared
Sep 5 03:09:58 LCC: ch_info_local_hw_error_blob_update: HWMon rg0_num_errors 0, rg1_num_errors 0
Sep 5 03:09:58 LCC: ch_info_local_hw_error_blob_set: HWMon got RG0 0 flag RG1 0 flag
Sep 5 03:09:58 LCC: ch_srxtvp_ha_failover_on_coredump: srxpfe coredump finished


These logs are pointing the issue towards FPC 0.

CHASSISD_IPC_WRITE_ERR_NO_PIPE: FRU has no connection pipe fru_send_msg FWDD :-

This log means that the chassis process (chassisd) could not send a message to the indicated component because the interprocess communication (IPC) pipe to the FRU didn’t exist.


To recover and fix the HW try the below steps: 

1. Restart the FPC 
request chassis fpc slot slot-number (offline | online | restart)

2. Reboot SRX.

Post reboot, if you still see the same behaviour and same logs in Chassid, reach out to JTAC for further checks. 

Modification History

2024-09-09 : Article Created