Description

This article explained cause of reboot failure with errors "Pending installation" and "shutdown aborted" on EX3400 switches after Junos upgrade.

Solution is also listed in this article.

Symptoms

Some customers may encounter reboot failure errors as below on EX3400 switches after Junos upgrade.

{master:0}
root@EX3400-VC> request system reboot all-members
Reboot the system ? [yes,no] (no) yes

error: Pending installation   <<<<<<<<<<<
error: shutdown aborted   <<<<<<<<<<<

Solution

>>> Root cause of this issue is that customers may purge directory /packages/db/ by accident in order to free up storage space on the switch before Junos upgrade.

>>> There is a built-in shell script in Junos used to verify reboot process on EX3400. Purging directory /packages/db/ will fail the reboot check by the script.

Example:

root@jtac-ex3400-24p-r2037:RE:0% sh /usr/libexec/ui/reboot_check; echo $?
Reboot consistency check failed - /packages/sets/active/os-kernel/boot/loader.conf does not exist or is zero size                <<<<<<<<<<<
Pending installation
1

>>> Further check reveals that directory /os-kernel is missed from directory /packages/sets/active/ to pass reboot check.

root@jtac-ex3400-24p-r2037:RE:0% ls -alh /packages/sets/active/
total 40
drwxr-xr-x  4 root  wheel   1.0K Jun 16 09:35 .
drwxrwxrwx  4 root  wheel   512B Jun 16 09:35 ..
-rw-r--r--  1 root  wheel     0B Jun 16 08:58 .firstboot.frob
drwxr-xr-x  2 root  wheel   512B Jun 16 09:35 boot
-rw-r--r--  1 root  wheel    13B Jun 16 08:58 junos-release
drwxr-xr-x  2 root  wheel   512B Jun 16 09:35 optional

>>> If we take a look into a working switch we will see that directory /packages/sets/active/ is linked to directory /packages/db/* as below.

root@s10-41:RE:0% ls -alh /packages/sets/active/
total 48
drwxr-xr-x  4 root  wheel   1.0K May  8 04:45 .
drwxr-xr-x  4 root  wheel   512B May  8 04:51 ..
drwxr-xr-x  2 root  wheel   512B May  8 04:44 boot
lrwxr-xr-x  1 root  wheel    33B May  8 04:42 dsa -> /packages/db/dsa-arm-32-21.4R3.15
-rw-r--r--  1 root  wheel   2.0K May  8 04:42 files.tar
lrwxr-xr-x  1 root  wheel    66B May  8 04:42 fips-mode -> /packages/db/fips-mode-arm-32-20220903.052830_builder_junos_214_r3
lrwxr-xr-x  1 root  wheel    71B May  8 04:42 jail-runtime -> /packages/db/jail-runtime-arm-32-20220817.43c4e23_builder_stable_12_214
<Output truncated>
lrwxr-xr-x  1 root  wheel    75B May  8 04:44 os-boot-junos -> /packages/db/os-boot-junos-ve-arm-32-20220817.43c4e23_builder_stable_12_214
lrwxr-xr-x  1 root  wheel    68B May  8 04:44 os-crypto -> /packages/db/os-crypto-arm-32-20220817.43c4e23_builder_stable_12_214
lrwxr-xr-x  1 root  wheel    72B May  8 04:44 os-kernel -> /packages/db/os-kernel-prd-arm-32-20220817.43c4e23_builder_stable_12_214                          <<<<<<<<<<<<
lrwxr-xr-x  1 root  wheel    69B May  8 04:44 os-libs -> /packages/db/os-libs-12-arm-32-20220817.43c4e23_builder_stable_12_214
lrwxr-xr-x  1 root  wheel    69B May  8 04:45 os-runtime -> /packages/db/os-runtime-arm-32-20220817.43c4e23_builder_stable_12_214
lrwxr-xr-x  1 root  wheel    48B May  8 04:45 packages.xml -> /packages/db/junos-arm-32-21.4R3.15/packages.xml
lrwxr-xr-x  1 root  wheel    64B May  8 04:45 py-base -> /packages/db/py-base-arm-32-20220903.052830_builder_junos_214_r3
lrwxr-xr-x  1 root  wheel    70B May  8 04:45 py-extensions -> /packages/db/py-extensions-arm-32-20220903.052830_builder_junos_214_r3
lrwxr-xr-x  1 root  wheel    63B May  8 04:45 zoneinfo -> /packages/db/os-zoneinfo-20220817.43c4e23_builder_stable_12_214

>>> Jumping on the problematic switch again, we can see directory /packages/db/ is being re-spawned after adding new/target Junos version via CLI command "request system software add /var/tmp/XXX"

root@jtac-ex3400-24p-r2037:RE:0% ls -alh /packages/db/os-kernel-prd-arm-32-20220523.3155a70_builder_stable_12_212/boot/
total 54640
drwxr-xr-x  2 root  wheel   512B Jun  1  2022 .
drwxr-xr-x  3 root  wheel   512B Jun 16 09:38 ..
-rw-r--r--  1 root  wheel   4.7K Jun  1  2022 brcm-hr3.dtb
-rwxr-xr-x  1 root  wheel   9.6M Jun  1  2022 contents.izo   <<<<<<<<<<<<
-rw-r--r--  1 root  wheel   7.0K Jun  1  2022 ex2300-24mp.dtb
-rw-r--r--  1 root  wheel   7.0K Jun  1  2022 ex2300-48mp.dtb
-rw-r--r--  1 root  wheel   7.9K Jun  1  2022 ex2300-c.dtb
-rw-r--r--  1 root  wheel   7.9K Jun  1  2022 ex2300.dtb
-rw-r--r--  1 root  wheel   7.9K Jun  1  2022 ex3400.dtb
-rwxr-xr-x  1 root  wheel    68K Jun  1  2022 if_gmac.ko
-rwxr-xr-x  1 root  wheel   116K Jun  1  2022 iflib.ko
-rwxr-xr-x  1 root  wheel    17M Jun  1  2022 kernel
-rw-r--r--  1 root  wheel   645B Jun  1  2022 loader.conf     <<<<<<<<<<<
-rwxr-xr-x  1 root  wheel   156K Jun  1  2022 miibus.ko

>>> To fix the issue, to pass the reboot check, we have to copy two mandatory files (“loader.conf” and “contents.izo” ) to directory /packages/sets/active/os-kernel/boot/ as below.

root@jtac-ex3400-24p-r2037:RE:0% cp /packages/db/os-kernel-prd-arm-32-20220523.3155a70_builder_stable_12_212/boot/loader.conf /packages/sets/active/os-kernel/boot/loader.conf
root@jtac-ex3400-24p-r2037:RE:0%
root@jtac-ex3400-24p-r2037:RE:0% sh /usr/libexec/ui/reboot_check ; echo $?
Reboot consistency check failed - /packages/sets/active/os-kernel/boot/contents.izo in loader.conf exist or is zero-length
Pending installation
1
root@jtac-ex3400-24p-r2037:RE:0% cp /packages/db/os-kernel-prd-arm-32-20220523.3155a70_builder_stable_12_212/boot/contents.izo /packages/sets/active/os-kernel/boot/contents.izo
root@jtac-ex3400-24p-r2037:RE:0%
root@jtac-ex3400-24p-r2037:RE:0% sh /usr/libexec/ui/reboot_check ; echo $?
Pending installation
Reboot consistency check bypassed - installation will complete upon reboot
0

>>> After that reboot check passed and reboot via CLI succeeded.

{master:0}
labroot@jtac-ex3400-24p-r2037> request system reboot at now
Jun 16 10:55:03
Reboot the system at now? [yes,no] (no) yes
Jun 16 10:55:03
 
                                                                               
*** FINAL System shutdown message from labroot@jtac-ex3400-24p-r2037 ***    
 
System going down IMMEDIATELY                                                 
 
                                                                               
Shutdown NOW!
[pid 16155]

>>> After rebooting, we can see that the switch is now running at the target Junos version as well.

FreeBSD/arm (jtac-ex3400-24p-r2037) (ttyu0)
 
login: root
Password:
Last login: Fri Jun  9 12:10:21 on ttyu0
 
--- JUNOS 21.2R3-S2.9 Kernel 32-bit  JNPR-12.1-20220523.3155a70_buil
root@jtac-ex3400-24p-r2037:RE:0% cli
show {master:0}
root@jtac-ex3400-24p-r2037>                         
 
{master:0}
root@jtac-ex3400-24p-r2037> show version
fpc0:
--------------------------------------------------------------------------
Hostname: jtac-ex3400-24p-r2037
Model: ex3400-24p
Junos: 21.2R3-S2.9           <<<<<<<<<<<< The switch was booted up with the target Junos version.
<Output truncated>
 

Modification History

Publish