Description

Due to some reason, Junos upgrade failure might occur in the middle of the ISSU procedure. The warning: ISSU in progress error message is displayed and commit also fails. The device cannot be rebooted via the request system reboot command.

Symptoms

Customers experience an issue, when attempting a ISSU upgrade on an EX4200 virtual chassis. The system generates the warning: ISSU in progress error message: 

{master:0}[edit]
root# commit 
warning: ISSU in progress, commit disallowed

{master:0}[edit]
root# commit force 
warning: ISSU in progress, commit disallowed

{master:0}[edit]
root# exit 
Exiting configuration mode

{master:0}
root> request system reboot 
warning: ISSU in progress

{master:0}
root>
The issue persists, even after tearing down the EX4200 device from the VC and a normal format installation.

Solution

This issue is due to a flag been set, when performing the ISSU upgrade. The flag bit will not get cleared; so ISSU experienced an unexpected failure. The customer can check the flag via the root shell by using the following command; a non-zero value indicates that the ISSU flag has been set: 

root@:RE:0% sysctl hw.re.issu_state 
hw.re.issu_state: 64
The corresponding paremeter from NVRAM also displays the same value: 
root@:RE:0% nvram getenv hw.ore.issustate   
hw.ore.issustate=64

To resolve this issue, you have to clear the flag via the following shell command: 

root@:RE:0& nvram unsetenv hw.ore.issustate
Or 
root@:RE:0% nvram setenv hw.ore.issustate 0
Note : In certain cases, you may also need to 'unset hw.lc.issuboot' as well. 
root@:RE:0& nvram unsetenv hw.lc.issuboot
This will take affect only after a reboot. To recover from the issue immediately, the following commands have to be executed: 
root@:RE:0% sysctl -w hw.re.issu_state=0  
hw.re.issu_state: 64 -> 0
After this is done, the system can operate normally. The following excerpt is an example of the output of the actions: 
{master:0}[edit]
root# commit
warning: ISSU in progress, commit disallowed

{master:0}[edit]
root# exit
Exiting configuration mode

{master:0}
root> request system reboot
warning: ISSU in progress

{master:0}
root> start shell user root
root@:RE:0%sysctl hw.re.issu_state
hw.re.issu_state: 64
root@:RE:0% sysctl -w hw.re.issu_state=0
hw.re.issu_state: 64 -> 0
root@:RE:0% nvram unsetenv hw.ore.issustate
root@:RE:0% nvram getenv hw.ore.issustate
nvram: variable hw.ore.issustate not found
root@:RE:0% cli
{master:0}
root> configure
Entering configuration mode

{master:0}[edit]
root# commit
configuration check succeeds
commit complete

{master:0}[edit]
root#