Description

Commit does not work after a failure in NSSU upgrade process. Creates multiple core-dumps on the master device and makes member 0 unreachable from the backup switch.

Symptoms

Below conditions are observed during the issue:
 

  • Effected device will be in inaccessible state.

{master:1}[edit]
user@switch>request session member 0 
rlogin: read: Connection reset by peer
rlogin: connection closed
  • Generates multiple core-dumps on the effected switch.

{master:1}[edit]
user@switch# run show system core-dumps 
fpc0:
--------------------------------------------------------------------------
-rw-rw----  1 root  field          0 Jun 4  20:21 /var/tmp/authd.core.4.gz
-rw-rw----  1 root  field     149872 Jun 4  20:14 /var/tmp/bfdd.core.0.gz
-rw-rw----  1 root  field    1261578 Jun 4  20:21 /var/tmp/chassisd.core.0.gz
-rw-rw----  1 root  field     425994 Jun 4  20:21 /var/tmp/fxpc.core.0.gz
-rw-rw----  1 root  field     817911 Jun 4  20:14 /var/tmp/mib2d.core.0.gz
-rw-rw----  1 root  field     706248 Jun 4  20:14 /var/tmp/pfed.core.0.gz
-rw-rw----  1 root  field     139101 Jun 4  20:14 /var/tmp/ppmd.core.0.gz
-rw-rw----  1 root  wheel     208901 Jun 4  20:14 /var/tmp/sdk-vmmd.core.0.gz
-rw-rw----  1 root  field      25792 Jun 4  20:14 /var/tmp/sdk-vmmd.core.1.gz
-rw-rw----  1 root  field      25792 Jun 4  20:14 /var/tmp/sdk-vmmd.core.2.gz
-rw-rw----  1 root  field      25792 Jun 4  20:16 /var/tmp/sdk-vmmd.core.3.gz
-rw-rw----  1 root  field          0 Jun 4  20:21 /var/tmp/sdk-vmmd.core.4.gz
-rw-rw----  1 root  field     278519 Jun 4  20:14 /var/tmp/shm-rtsdbd.core.0.gz
-rw-rw----  1 root  field     804546 Jun 4  20:14 /var/tmp/snmpd.core.0.gz
-rw-rw----  1 root  field     649538 Jun 4  20:14 /var/tmp/vccpd.core.0.gz
  • In virtual-chassis the device status will be shown as active. But cannot be accessible from the cli.

{master:1}
user@switch> show virtual-chassis status 
Preprovisioned Virtual Chassis
Virtual Chassis ID:XXXX.YYYY.XYXY
Virtual Chassis Mode: Enabled
                                                Mstr           Mixed Route Neighbor List
Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
0 (FPC 0)  Prsnt    XXXXXXXXXX ex4600-40f     129   Backup       N  VC   1  vcp-255/0/26
                                                                           1  vcp-255/0/27
1 (FPC 1)  Prsnt    YYYYYYYYYY ex4600-40f     129   Master*      N  VC   0  vcp-255/0/26
                                                                           0  vcp-255/0/27

 

Solution

To restore the services on the switch. Following need to be performed.
 

  • Take physical console of the effected switch and perform a hard reboot to restore services on the switch.
  • Once the device is accessible, clear the storage on the switch.
user@switch>request system storage clean-up member <member-id>
  • After storage clean-up the commit will work and cli session to the switch will be accessible.
  • Configuration on the devices will work as expected.

Modification History

2024-06-14 : Article Created