Description

This article describes the issue of a JUNOS device being stuck in the boot process and failing to load Junos.

Symptoms

A JUNOS device is stuck in loading JUNOS after an upgrade.

JUNOS devices may get stuck in the boot process or fail to boot the OS. In rare cases, after a sudden power loss or ungraceful power shut down, JUNOS devices can experience file system corruption, which prevents the switch from recovering to a functional state. It is recommended that customers minimize their log configurations to prevent excessive read/writes to the file system, which reduces the stress on storage media and reduces the potential occurrence of this issue. Moreover, if abrupt power failures are transient for a very short period of time, the availability of an UPS can also prevent the JUNOS devices from experiencing a sudden power loss.

You do not have to worry about damaging hardware in these situations, as the hardware cannot tell the difference between a graceful shutdown and pulling the power cord. The potential for damage is with the file system structure. It is possible for data to be corrupted, when the computer's power is interrupted with the operating system running. The data could be in the nodes, which could result in files being lost or file contents being corrupted.

Although rare, this issue more likely occurs on platforms that use a UNIX/BSD-based operating system, such as Junos, to access the flash-based storage media. This issue has been noted in O'Reilly Media’s JUNOS Enterprise Switching book:

“Although rare, file system damage can occur with an abrupt power off, which may cause problems on the next boot. Use the request system halt or request system reboot command to gracefully shut down or reboot the OS. Once the OS is halted, it is safe to remove power.”

Solution

If you see the Debug Prompt ( db> ), refer to KB20635 - While booting up, switch stuck in db> mode [juniper.net] .

If you receive a Can't load kernel error message, power cycle the MX device, if the console bootup process does not return to a prompt and stops, when the following error messages are displayed:

can't load '/kernel'
can't load '/kernel.old'
If it continues to stop at the same error messages after a reboot and it does not progress to another prompt or process, then perform a Format Install . If the above methods do not work, a Format Install is the last available recovery option.

Warning : Performing a Format Install formats the entire file system and storage unit. So, the EX switch will lose all the configuration and logs. Subsequently, a Root Cause Analysis or recovery of any information from the EX switch will not be possible. Do not perform this install, if a Root Cause Analysis is required. Instead contact your technical support representative.

To perform the Format Install, refer to  KB20643 - [EX Switch] Rewrite the entire file system by issuing "install --format" command from "Loader" mode (Format Install) [juniper.net] .

Modification History

2020-10-23: Modified the article to reflect multiple models using JUNOS as this is not related to only MX or EX.