Description

This KB consists of general pre-checks that needs to be performed before a software upgrade to reduce the failure incidents during the process and to help troubleshoot or provide RCA in case of failure. During upgrade processes, is important to have all the information and checks ready to provide an accurate RCA and troubleshooting session. 

Solution

On Juniper Networks EX/QFX platforms the following checks can be performed to improve the success during upgrades activities:
 
1. During the image transfer and upgrade process, the platforms could give errors due to storage as follows: 

Error: not enough space to unpack /var/transfer/junos-arm-32-21.2R3-S3.5.tgz
ERROR: Either use 'force' or consider deleting snapshots using 'request system snapshot delete ' 


To avoid these type of issues, check the storage capacity as follows:
root@juniper> show system storage
fpc0:
--------------------------------------------------------------------------
Filesystem      Size  Used  Avail  Capacity  Mounted on
/dev/gpt/junos  1.3G  1.1G  139M   89%       /.mount <-- Not enough space
tmpfs           651M  48K   651M   0%        /.mount/tmp
tmpfs           324M  436K  324M   0%        /.mount/mfs

Then we can clear storage as follows:

root@juniper> request system storage cleanup

Note: For EX2300 and EX3400 please refer to the following KB:

[EX] Not enough storage while upgrading Junos EX2300 and EX3400

2. Another check that is important for any upgrade, is to have console access. This will help access the device if any issue arise and also helps recover the device more quickly. 

3. Based on the point above, is also import to record the session terminal so we can have the information during the upgrade process. For example for Putty is as follows:

image.png

Note: This may vary depending the terminal you are using. 
4. Other check that is important to have is a backup of the configuration using the following commands:
root@switch> show configuration | display set | no-more

And then it can be copied to a notepad or any text editor. 

5. To provide accurate RCA in case the upgrade fails, it's important to collect the RSI and logs before the procedure
root@switch> request support information | save /var/tmp/RSI.txt
root@switch> file archive compress source /var/log destination /var/tmp/logs 

This will create the files on the /var/tmp/ directory and can be collected via WinScp or any FTP method. 

6. Another device that is useful to have during upgrades is an USB flash drive. This will help in case the images needs to be pushed to the device and also if the upgrade fails, the device can be recovered via format install using the flash drive. 

7. Another point that is important to check is the integrity of the software that we are adding to the device and for that we can check it with the md5 checksum. Please refer to the following article for a detail explanation:

[Junos] How to check MD5 checksum on a Junos installation package


8. Sometimes while transferring the image via WinScp or any other file server, there could be an error message when the transfer is being done by a non-root user.  Non-root users/super-user may find that the file copy utility always stages the transfer of jinstall packages for Juniper Networks devices, or any other file to the user's home directory even when the destination path is specified as /var/tmp and for that is always better to use the root user.  Please refer to the following article for a detail explanation:

"file copy" destination defaults to /var/home for non-root user resulting in image transfer failure

It's important to mention that these are general checks and requirements for any platform. If the upgrade fails and device needs to be recovered but an RCA is required, it's important to contact JTAC to collect all the information to provide the cause of the failure. 

As per post checks, this really depends on each customer network and the systems that they are running, so it’s better for each customer to check their systems internally.

For the switch, we can run the following commands:

show version
show chassis routing-engine
show chassis alarms
show system alarms
show systen core-dumps
show chassis hardware 
show system processes extensive | except 0.00%

 
In case of any issue, please contact JTAC.
 

Modification History

Visible to customer  added

Related Information

[EX] Not enough storage while upgrading Junos EX2300 and EX3400

How to collect /var/log files from all members of an EX Switch Virtual Chassis (VC) using the tar command

[Redirected] [EX] How to collect /var/log files from all EX Switch VC members using file archive compress method

Upgrading/Downgrading/Reinstalling Junos

[EX] Performing and resolving any common issues during Format Install on legacy EX platforms