Router crashed while customer was running prechecks as a part of preparing the device for an upcoming maintenance. After the crash, the devices rebooted and came online in amnesiac mode with an older Junos code. The router booted to 18.3R1-S5.1 instead of customer version 20.3X2.1.
1] No alarms or errors other the device prints a message indicating that its in amnesiac mode
labuser@labrouter> show chassis alarms
Aug 15 01:21:18
No alarms currently active
2] No errors reported in "show vmhost version or show vmhost snapshot" outputs
labuser@labrouter> show vmhost version
Aug 15 01:37:34
Current root details, Device sda, Label: jrootp_S, Partition: sda3
Current boot disk: Secondary
Current root set: p
UEFI Version: CBDE_P_RIO1_00.25.01
Secondary Disk, Snapshot Time: <fresh install>
Version: set p
VMHost Version: 3.1861
VMHost Root: vmhost-x86_64-18.3R1-S1-20181010_1124_builder
VMHost Core: vmhost-core-x86-64-18.3R1-S5.1
kernel: 3.10.100-ovp-rt110-WR6.0.0.31_preempt-rt
Junos Disk: junos-install-acx-x86-64-18.3R1-S5.1
Version: set b
labuser@labrouter> show vmhost snapshot
Aug 15 01:37:49
3] However, one of the DISK is missing in "show vmhost hardware " output
P2145550@IRNHTXOF2QW> show vmhost hardware
Aug 16 00:24:39
Compute cluster: rainier-re-cc
Compute node: rainier-re-cn
Hardware inventory:
Item Capacity Part number Serial number Description
DIMM 0 16384 MB VL41A2G63A-N6SB-JUN 0x52978325 DDR4 2133 MHz
DIMM 1 16384 MB VL41A2G63A-N6SB-JUN 0x52978337 DDR4 2133 MHz
Disk1 100 GB StorFly-VSFBM6CC100G P1T14005244403300124 SLIM SATA SSD
4] Vmhost snapshot creation failing and reporting that the Router is booted with secondary disk
Labroot@labhost> request vmhost snapshot partition
Aug 16 00:36:31
warning: All existing data on the target media will be lost
Proceed ? [yes,no] (no) yes
Aug 16 00:36:37
warning: Proceeding with vmhost snapshot partition
Already running from secondary disk.
Please try recovery option to restore primary!
Snapshot aborted.
5] Routing Engine Console logs or dmesg output reports issues in accessing Primary Disk
ata1: SATA max UDMA/133 abar m2048@0x90f00000 port 0x90f00100 irq 81
ata2: SATA max UDMA/133 abar m2048@0x90f00000 port 0x90f00180 irq 81
ata3: DUMMY
ata4: DUMMY
ata5: DUMMY
ata6: DUMMY
ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata2.00: FORCE: horkage modified (noncq)
ata2.00: ATA-9: StorFly VSFBM6CC100G-JUN1, 0626-000, max UDMA/133
ata1: COMRESET failed (errno=-16)
ata1: link is slow to respond, please be patient (ready=0)
ata1: limiting SATA link speed to 3.0 Gbps
ata1: reset failed, giving up
scsi 1:0:0:0: Direct-Access ATA StorFly VSFBM6CC 0626 PQ: 0 ANSI: 5
sd 1:0:0:0: Attached scsi generic sg0 type 0
sd 1:0:0:0: [sda] 195371568 512-byte logical blocks: (100 GB/93.1 GiB)
sd 1:0:0:0: [sda] Write Protect is off
sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda: sda1 sda2 sda3 sda4 sda5 sda6
sd 1:0:0:0: [sda] Attached SCSI disk
The above Kernal Logs (5) indicates that the device is failing to read the primary disk (ata1, sd 0:0:0:0). As primary disk failed, system renamed secondary disk (ata2, address sd 1:0:0:0) ) as “sda” instead of “sdb”. Hence, no alarms reported stating that the device was booted from secondary disk
In this case, customer did not created vmhost snapshot previously, hence system booted in an older code and in amnesiac mode.Replace Routing Engine to resolve the issue