This article describes how to recover from a failed software installation of Junos OS or restore the previous working Junos OS configuration if you access the Command Line Interface (CLI).
The device does not boot up to the CLI due to issues during installation after an upgrade. After a failed installation, the QFX5K devices are left in a state that makes them inaccessible remotely and it needs a new software installation by copying the software from the USB to the device or by using the installation from a USB.
There could be various reasons that trigger this issue, such as incompatible configuration for the loaded Junos OS or file-system issues.
If the device boots to CLI, a complete reboot is needed, including the Linux host OS. This is done by using request system reboot hypervisor . Sometimes with a CLI reboot, when pressing "Esc" to get to boot options on the console, the following menu is seen:
request system reboot hypervisor
In such cases, or when CLI is not accessible, we need to power cycle the QFX to get to the right recovery screen in the screenshots mentioned below.
A recovery image is created automatically on the switches. If a previously-running switch is powered on and unable to boot using a Junos OS image, you can boot the switch by using the recovery Junos OS image from one of the saved images.
Also, once the appropriate recovery image is selected, you get the option to format the disk/configuration, which may help recover from any incompatible configurations that prevent the device from booting to a particular Junos OS image. For QFX5100/EX4600
For QFX5200
After selecting Linux recovery on QFX5200:
(This is an example of booting up from one of the recovery images without formatting the disk/configuration to show the options and the boot sequence.)
Recovering Linux ... 4..... root=LABEL=ROOT console=ttyS0,9600n8 quiet i8042: No controller found First Level Bootstrap using initramfs... Mounting boot device LABEL=LINUX-BOOT Unpacking initrd.cpio.gz .... 1033308 blocks Unmount boot device LABEL=LINUX-BOOT Starting init INIT: Mounting VAR to /var Starting udev udevd[361]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory sysvinit-2.88dsf-r500.0.core2_64 running pre rc steps on QFX5_32C_32Q Changing boot device enable from 0x00 to 0x00 0x2C 0x2c 2 logical volume(s) in volume group "vg0_vjunos" now active 2 logical volume(s) in volume group "vg0_vjunos" now active PV VG Fmt Attr PSize PFree /dev/sda6 vg0_vjunos lvm2 a-- 8.86g 0 /dev/sdb vg0_vjunos lvm2 a-- 29.82g 0 VG #PV #LV #SN Attr VSize VFree vg0_vjunos 2 2 0 wz--n- 38.68g 0 LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert lv_junos vg0_vjunos -wi-a--- 30.00g lv_var_third_party vg0_vjunos -wi-a--- 8.68g +------------------------------------------+ | Junos Host applications Recovery | +------------------------------------------+ ================================================= Current image : qfx-5e-18.2X75-D435.20.202104251815 Previous image : qfx-5e-18.2X75-D435.20.202104221228 ================================================= Select an image for recovery or cancel to continue: Select a recovery image: 1) qfx-5e-18.2X75-D435.20.202104251815 2) qfx-5e-18.2X75-D435.20.202104221228 3) cancel Pick an option[1-3]: Pick an option[1-3]: 1 Do you want to format Junos data disk?[y/n] n Do you want to format Junos config disk?[y/n] n Do you want to reimage linux host?[y/n] n ================================================= Recovery options selected: Recovery image : qfx-5e-18.2X75-D435.20.202104251815 Format Junos data disk : no Format Junos config disk : no Reimage linux host : no ================================================= Choose one of the following options: 1) Proceed with the recovery. 2) Go back to the recovery menu. 3) Abort the recovery process. Please confirm[1/2/3]: 1 Setting up Junos host application for upgrade ... Recovering Junos host application image to: qfx-5e-18.2X75-D435.20.202104251815 Recovery staged. Rebostopping rsyslogd ... done Stopping OpenBSD Secure Shell server: sshdno /usr/sbin/sshd found; none killed Stopping atd: OK Stopping domain name service: named. Unmounting cgroups...Done Stopping system message bus: dbus. stopping DNS forwarder and DHCP server: dnsmasq... no /usr/bin/dnsmasq found; none killed Shutting down irqbalance: no irqbalance found; none killed done stopping mountd: done stopping nfsd: done Stopping random number generator daemon. stopping rsyslogd ... done Stopping internet superserver: xinetd. stopping statd: done Stopping portmap daemon... udevd[367]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory Clearing ebtables rulesets: filter nat broute done. ok Stopping crond: FAIL Stopping rpcbind daemon... not running. Stopping S.M.A.R.T. daemon: smartd. Stopping network management services: snmpd snmptrapd libvirtMib_subagent. * Stopping virtualization library daemon: libvirtd [fail] Deconfiguring network interfaces... ifdown: interface lo not configured done. Stopping tcsd: FAIL Sending all processes the TERM signal... Sending all processes the KILL signal... Unmounting remote filesystems... Deactivating swap... Unmounting local filesystems... Rebooting... RE-FPGA-DRV: reboot notifier called with 0x0001 RE-FPGA-DRV: Please standby while rebooting. . .. .. 3h 3h InsydeH2O version : V0018.6U BIOS Build Date : 10/30/2013 System Memory Speed : 1333 MHz Processor Type : Intel(R) Xeon(R) CPU @ 1.80GHz CPUID : 306A9 Press Esc for boot options Now booting : SSD0 : TS32GMTS400 GNU GRUB version 2.00 +--------------------------------------------------------------------------+ |Juniper Linux | |Juniper Linux Debug | |Juniper-Linux-Upgrade | |Juniper-Linux-Recovery | | | | | | | | | | | | | | | | | +--------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a command-line. The highlighted entry will be executed automatically in 0s. Booting `Juniper-Linux-Upgrade' Upgrading Linux ... i8042: No controller found First Level Bootstrap using initramfs... Mounting boot device LABEL=LINUX-BOOT Unpacking initrd.cpio.gz .... 1033308 blocks Unmount boot device LABEL=LINUX-BOOT Starting init INIT: version 2Mouting VAR to /var Starting udev udevd[329]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory sysvinit-2.88dsf-r500.0.core2_64 running pre rc steps on QFX5_32C_32Q Changing boot device enable from 0x00 to 0x00 0x2C 0x2c Moving [HOST_COMPAT_VERSION app_install_record.md5 lxc_pkg_install.sh pkg-app-junos.tgz pkg_install.sh platform_mappings qfx-5e-18.2X75-D435.20.tgz version.txt] to /var/sw/packages/applications 2 logical volume(s) in volume group "vg0_vjunos" now active udevd[329]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory PV VG Fmt Attr PSize PFree /dev/sda6 vg0_vjunos lvm2 a-- 8.86g 0 /dev/sdb vg0_vjunos lvm2 a-- 29.82g 0 VG #PV #LV #SN Attr VSize VFree vg0_vjunos 2 2 0 wz--n- 38.68g 0 LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert lv_junos vg0_vjunos -wi-a--- 30.00g lv_var_third_party vg0_vjunos -wi-a--- 8.68g Setting up boot params ... ------------------------------------------------ Running Junos application installer ------------------------------------------------ ------------------------------- Installing /var/sw/packages/applications/qfx-5e-18.2X75-D435.20.tgz ------------------------------- Removing package: qfx-5e-control-plane-1.0-0@x86_64 Removing package: qfx-5e-host-control-1.0-0@x86_64 Removing package: qfx-5e-host-platform-1.0-0@x86_64 pkg_install_rpms: qfx-5e-base-1.0-0-18.2X75-D435.20.x86_64.rpm pkg_install_rpms: installing LXC supported RPMs ======================================== Installing qfx-5e-host-control-1.0-0-18.2X75-D435.20.x86_64.rpm ... ======================================== Loading cache... Updating cache... ######################################## [100%] Committing transaction... Preparing... ######################################## [ 0%] 1:Installing qfx-5e-host-c.. ######################################## [100%] Output from qfx-5e-host-control-1.0-0@x86_64: Stopping vehostd: Starting vehostd: vehostd startup touch: cannot touch '/var/lock/subsys/vehostd': No such file or directory Installing packages (1): qfx-5e-host-control-1.0-0@x86_64 348.3kB of package files are needed. 1.2MB will be used. ======================================== Installing qfx-5e-host-platform-1.0-0-18.2X75-D435.20.x86_64.rpm ... ======================================== Loading cache... Updating cache... ######################################## [100%] Committing transaction... Preparing... ######################################## [ 0%] Output from qfx-5e-host-platform-1.0-0@x86_64: dcpfe0 is not running 1:Installing qfx-5e-host-p.. ######################################## [100%] Installing packages (1): qfx-5e-host-platform-1.0-0@x86_64 5.8MB of package files are needed. 22.6MB will be used. Saving cache... ======================================== Installing qfx-5e-control-plane-1.0-0-18.2X75-D435.20.x86_64.rpm ... ======================================== Loading cache... Updating cache... ######################################## [100%] Committing transaction... Preparing... ######################################## [ 0%] 1:Installing qfx-5e-contro.. ######################################## [100%] Output from qfx-5e-control-plane-1.0-0@x86_64: ------------------------------- Installing JUNOS image: jinstall-jcp-i386-18.2X75-D435.20.img.gz ------------------------------- Extracting jinstall-jcp-i386-18.2X75-D435.20.img.gz to /recovery/junos/jinstall-jcp-i386-18.2X75-D435.20-202105060903.img Prepare host for virtfs... Integrity check passed for hash-control-plane.md5. Installing packages (1): qfx-5e-control-plane-1.0-0@x86_64 943.3MB of package files are needed. 954.0MB will be used. Saving cache... ======================================== Application installed. Setting up boot params ... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory / --------------------------------------------- Installation for Junos Rebooting the system to complete the installation --------------------------------------------- INIT: Switchingstopping rsyslogd ... done Stopping OpenBSD Secure Shell server: sshdno /usr/sbin/sshd found; none killed Stopping atd: OK Stopping domain name service: named. Unmounting cgroups...Done Stopping system message bus: dbus. stopping DNS forwarder and DHCP server: dnsmasq... no /usr/bin/dnsmasq found; none killed Shutting down irqbalance: no irqbalance found; none killed done stopping mountd: done stopping nfsd: done Stopping random number generator daemon. stopping rsyslogd ... done Stopping internet superserver: xinetd. stopping statd: done Stopping portmap daemon... udevd[329]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory Clearing ebtables rulesets: filter nat broute done. ok Stopping crond: FAIL Stopping rpcbind daemon... not running. Stopping S.M.A.R.T. daemon: smartd. Stopping network management services: snmpd snmptrapd libvirtMib_subagent. * Stopping virtualization library daemon: libvirtd [fail] Deconfiguring network interfaces... ifdown: interface lo not configured done. Stopping tcsd: FAIL Sending all processes the TERM signal... Sending all processes the KILL signal... Unmounting remote filesystems... Deactivating swap... Unmounting local filesystems... Rebooting... RE-FPGA-DRV: reboot notifier called with 0x0001 RE-FPGA-DRV: Please standby while rebooting. . .. .. InsydeH2O version : V0018.6U BIOS Build Date : 10/30/2013 System Memory Speed : 1333 MHz Processor Type : Intel(R) Xeon(R) CPU @ 1.80GHz CPUID : 306A9 Press Esc for boot options Now booting : SSD0 : TS32GMTS400 GNU GRUB version 2.00 +--------------------------------------------------------------------------+ |Juniper Linux | |Juniper Linux Debug | |Juniper-Linux-Recovery | | | | | | | | | | | | | | | | | | | +--------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a command-line. Loading Linux ... i8042: No controller found First Level Bootstrap using initramfs... Mounting boot device LABEL=LINUX-BOOT Unpacking initrd.cpio.gz .... 1033308 blocks Unmount boot device LABEL=LINUX-BOOT Starting init INIT: Mouting VAR to /var Starting udev udevd[329]: can not find '/etc/udev/rules.d/81-i40e-udevhelper.rules': No such file or directory sysvinit-2.88dsf-r500.0.core2_64 running pre rc steps on QFX5_32C_32Q Changing boot device enable from 0x00 to 0x00 0x2C 0x2c 2 logical volume(s) in volume group "vg0_vjunos" now active Starting udev <<<<<OMMITTED PART OF BOOT SEQUENCE FOR BREVITY>>>>> Removing any system startup links for nsca ... update-rc.d: /etc/init.d/hostapd exists during rc.d purge (continuing) Removing any system startup links for hostapd ... update-rc.d: /etc/init.d/wdmd exists during rc.d purge (continuing) Removing any system startup links for wdmd ... /etc/rc0.d/K20wdmd /etc/rc1.d/K20wdmd /etc/rc2.d/S20wdmd /etc/rc3.d/S20wdmd /etc/rc4.d/S20wdmd /etc/rc5.d/S20wdmd Wind River Linux 7.0.0.9 localhost console INIT: Consoles: serial port BIOS drive C: is disk0 BIOS drive D: is disk1 BIOS drive E: is disk2 BIOS drive F: is disk3 BIOS 639kB/3144576kB available memory FreeBSD/x86 bootstrap loader, Revision 1.1 (Thu Apr 23 13:03:02 PDT 2020 [email protected]) - Booting from Junos volume ... | /packages/sets/pending/boot/os-kernel/kernel text=0x45d3d8 data=0x84130+0x30d970 syms=[0x8+0x971d0+0x8+0x8359e] <<<<<OMMITTED PART OF BOOT SEQUENCE FOR BREVITY>>>>> Creating initial configuration: ... mgd: Running FIPS Self-tests veriexec: no fingerprint for file='/sbin/kats/cannot-exec' fsid=203 fileid=51404 gen=1 uid=0 pid=5459 mgd: FIPS Self-tests Passed [1] 5580 mgd: commit complete @ 1620292523 [2021-05-06 14:45:23 IST] mgd done Lock Manager RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com Copyright (c) 1992-2006 Birdstep Technology, Inc. All Rights Reserved. Unix Domain sockets Lock manager Lock manager 'lockmgr' started successfully. Database Initialization Utility RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com Copyright (c) 1992-2006 Birdstep Technology, Inc. All Rights Reserved. Profile database initialized No core dumps found. Prefetching /usr/sbin/rpd ... Prefetching /usr/libexec64/rpd ... Prefetching /usr/sbin/lacpd ... Prefetching /usr/sbin/chassisd ... Starting jlaunchhelperd. 192.168.1.16 root 192.168.1.17 root Thu May 6 14:45:33 IST 2021 platform_fixups: 6157 6255 Thu May 6 14:45:33 IST 2021 platform_fixups: Give sighup to (pid:6157) parse new file ... @ 1620292533 [2021-05-06 14:45:33 IST] mountlater start @ 1620292538 [2021-05-06 14:45:38 IST] mountlater done Starting cron. Thu May 6 14:45:39 IST 2021 FreeBSD/amd64 (jtac-qfx5200-32c-32q-r2001) (ttyu0) login: labroot Password: Last login: Thu May 6 15:05:56 from 10.219.48.52 --- JUNOS 18.2X75-D435.20 Kernel 64-bit JNPR-11.0-20200423.d6acdc5_buil {master:0} labroot@jtac-qfx5200-32c-32q-r2001> {master:0} labroot@jtac-qfx5200-32c-32q-r2001> {master:0} labroot@jtac-qfx5200-32c-32q-r2001> show version localre: -------------------------------------------------------------------------- Hostname: jtac-qfx5200-32c-32q-r2001 Model: qfx5200-32c-32q Junos: 18.2X75-D435.20 JUNOS OS Kernel 64-bit [20200423.d6acdc5_builder_stable_11] JUNOS OS libs [20200423.d6acdc5_builder_stable_11] JUNOS OS runtime [20200423.d6acdc5_builder_stable_11] JUNOS OS time zone information [20200423.d6acdc5_builder_stable_11] JUNOS OS libs compat32 [20200423.d6acdc5_builder_stable_11] JUNOS OS 32-bit compatibility [20200423.d6acdc5_builder_stable_11] JUNOS py extensions [20210215.051819_builder_junos_182_x75_d435] JUNOS py base [20210215.051819_builder_junos_182_x75_d435] JUNOS OS vmguest [20200423.d6acdc5_builder_stable_11] JUNOS OS crypto [20200423.d6acdc5_builder_stable_11] JUNOS network stack and utilities [20210215.051819_builder_junos_182_x75_d435] JUNOS libs [20210215.051819_builder_junos_182_x75_d435] JUNOS libs compat32 [20210215.051819_builder_junos_182_x75_d435] JUNOS runtime [20210215.051819_builder_junos_182_x75_d435] JUNOS na telemetry [18.2X75-D435.20] JUNOS Web Management Platform Package [20210215.051819_builder_junos_182_x75_d435] JUNOS qfx runtime [20210215.051819_builder_junos_182_x75_d435] JUNOS common platform support [20210215.051819_builder_junos_182_x75_d435] JUNOS qfx platform support [20210215.051819_builder_junos_182_x75_d435] JUNOS Openconfig [18.2X75-D435.20] JUNOS dcp network modules [20210215.051819_builder_junos_182_x75_d435] JUNOS modules [20210215.051819_builder_junos_182_x75_d435] JUNOS qfx modules [20210215.051819_builder_junos_182_x75_d435] JUNOS qfx Data Plane Crypto Support [20210215.051819_builder_junos_182_x75_d435] JUNOS daemons [20210215.051819_builder_junos_182_x75_d435] JUNOS qfx daemons [20210215.051819_builder_junos_182_x75_d435] JUNOS Services URL Filter package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services TLB Service PIC package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Telemetry [20210215.051819_builder_junos_182_x75_d435] JUNOS Services SSL [20210215.051819_builder_junos_182_x75_d435] JUNOS Services SOFTWIRE [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Stateful Firewall [20210215.051819_builder_junos_182_x75_d435] JUNOS Services RPM [20210215.051819_builder_junos_182_x75_d435] JUNOS Services PCEF package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services NAT [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Mobile Subscriber Service Container package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services MobileNext Software package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Logging Report Framework package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services LL-PDF Container package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Jflow Container package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Deep Packet Inspection package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services IPSec [20210215.051819_builder_junos_182_x75_d435] JUNOS Services IDS [20210215.051819_builder_junos_182_x75_d435] JUNOS IDP Services [20210215.051819_builder_junos_182_x75_d435] JUNOS Services HTTP Content Management package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Flowd MS-MPC Software package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Crypto [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Captive Portal and Content Delivery Container package [20210215.051819_builder_junos_182_x75_d435] JUNOS Services COS [20210215.051819_builder_junos_182_x75_d435] JUNOS AppId Services [20210215.051819_builder_junos_182_x75_d435] JUNOS Services Application Level Gateways [20210215.051819_builder_junos_182_x75_d435] JUNOS Services AACL Container package [20210215.051819_builder_junos_182_x75_d435] JUNOS SDN Software Suite [20210215.051819_builder_junos_182_x75_d435] JUNOS Extension Toolkit [20210215.051819_builder_junos_182_x75_d435] JUNOS Packet Forwarding Engine Support (DC-PFE) [20210215.051819_builder_junos_182_x75_d435] JUNOS Packet Forwarding Engine Support (M/T Common) [20210215.051819_builder_junos_182_x75_d435] JUNOS J-Insight [20210215.051819_builder_junos_182_x75_d435] JUNOS jfirmware [20210215.051819_builder_junos_182_x75_d435] JUNOS Online Documentation [20210215.051819_builder_junos_182_x75_d435] JUNOS jail runtime [20200423.d6acdc5_builder_stable_11] JUNOS FIPS mode utilities [20210215.051819_builder_junos_182_x75_d435] JUNOS Host Software [3.14.52-rt50-WR7.0.0.9_ovp:3.0.9] JUNOS Host qfx-5e host-control package [18.2X75-D435.20] JUNOS Host qfx-5e host-platform package [18.2X75-D435.20] JUNOS Host qfx-5e control-plane package [18.2X75-D435.20] JUNOS Host qfx-5e platform package [18.2X75-D435.20] JUNOS Host qfx-5e base package [18.2X75-D435.20] JUNOS Host qfx-5e data-plane package [18.2X75-D435.20]
2021-06-10: Added screenshot for the menu seen with CLI reboot and clarified the need to power cycle if you cannot get to the recovery screen or when CLI isn't accessible for recovery. 2021-06-01: Trimmed down the long boot sequence and highlighted some key portions in the boot sequence to watch out for