Description

MX2020 which auto switchover RE after raised major alarm SPMB 0 not online.

root@DNI-PE1-MX2020_RE1> show chassis alarms no-forwarding

2 alarms currently active Alarm time Class Description

2024-03-06 05:14:32 ICT Major SPMB 0 not online

2024-03-06 05:14:02 ICT Minor Backup RE Active

 

Mar 6 05:10:52.412 2024 DNI-PE1-MX2020_RE0 chassisd[9190]: %DAEMON-4-CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for SPMB 0

Mar 6 05:10:52.412 2024 DNI-PE1-MX2020_RE0 alarmd[23770]: %DAEMON-4: Alarm set: SPMB id=33554547, color=RED, class=CHASSIS, reason=SPMB 0 not online

Mar 6 05:10:52.412 2024 DNI-PE1-MX2020_RE0 craftd[9193]: %DAEMON-4: Major alarm set, SPMB 0 not online

 

Symptoms

 

Mar 6 05:10:52 rcv: ch_ipc_dispatch() ipc pipe read error for args 0x11738da0 pipe 0x108c44a0, fru SPMB 0 errno 60

Mar 6 05:10:52 ch_ipc_dispatch: fru SPMB offline_reason = None

Mar 6 05:10:52 ch_connection_shutdown: Destroying the IPC pipe

Mar 6 05:10:52 CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for SPMB 0

Mar 6 05:10:52 -- SPMB 0, last request 0, state Online

Mar 6 05:10:52 send: red alarm set, device SPMB 0, reason SPMB 0 not online

Mar 6 05:10:52 hwdb: entry for spmb 2857 at slot 0 deleted

Mar 6 05:10:52 FM: SPMB connection is lost

Mar 6 05:10:52 SFB soft init mask 0xff

Mar 6 05:10:52 CHASSISD_JAM_RECONNECT_FILE_WRITE: jam_adc_create_reconnect_file: writen 28 bytes to file /usr/share/pfe/spmb_reconnect.txt packed size len 28

Mar 6 05:10:52 ADC soft init mask 0x2a800

Mar 6 05:10:52 spmb_fru_reconnect_window_begin: Opening FRU reconnect window

Mar 6 05:10:52 SPMB 0: spmb_disconnect: Starting restart timer

Mar 6 05:10:52 CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 14, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName SPMB 0, jnxFruType 10, jnxFruSlot 0, jnxFruOfflineReason 2, jnxFruLastPowerOff 1026908651, jnxFruLastPowerOn 2678)

Mar 6 05:11:03 tcb_update_local_esw_status: ESW port 23 (connected to SPMB) link down

Mar 6 05:11:08 tcb_update_local_esw_status: ESW port 23 (connected to SPMB) link up

Mar 6 05:11:13 tcb_update_local_esw_status: ESW port 23 (connected to SPMB) link down

Mar 6 05:11:52 CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 1 to restart SPMB 0 timed out; trying again

Mar 6 05:11:52 SPMB 0: spmb_hw_restart

Mar 6 05:11:52 notify_fru_power_off: NULL kvpairs for SPMB 0

Mar 6 05:12:52 CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 2 to restart SPMB 0 timed out; trying again

Mar 6 05:12:52 SPMB 0: spmb_hw_restart

Mar 6 05:12:52 notify_fru_power_off: NULL kvpairs for SPMB 0

Mar 6 05:13:52 CHASSISD_SPMB_RESTART_TIMEOUT: Attempt 3 to restart SPMB 0 timed out; Switch Mastership

 

Solution

Try  to restart the SPMB 0 :

Use below command "request chassis spmb slot <x> restart"

>request chassis spmb slot 0 restart 

 

If it didn't work, then log in to the shell: 

 

> start shell user root 

# cty -f spmb0

If it is still not reachable , then we need to try reseat the CB0 :

 

> request vmhost halt re0
or
>request system halt <re>
>show chassis routing-engine

> request chassis CB slot 0 offline 
>show chassis environment cb

and then,

 

reseat CB0 and check the behavior.

If after CB reseat,  SPMB is still not coming online, then please contact JTAC for an RMA of the CB. 

Modification History

2024-06-20 : Article Created