This article specifically applies to a virtual chassis environment. When upgrading an EX switch, you may encounter errors for insufficient space. Here, you will find the steps for how to successfully upgrade EX switches running in virtual chassis environment.
To upgrade Junos on a standalone chassis environment, understanding storage space allocation, and how to verify the amount of space you have before upgrading, refer to the following article: KB31198 - Not enough storage while upgrading Junos EX2300 and EX3400 [juniper.net].
During a software upgrade, if there is not enough storage space, the following error will be observed:
{master:0} root@juniper> show virtual-chassis Preprovisioned Virtual Chassis Virtual Chassis ID: d0e1.846e.3144 Virtual Chassis Mode: Enabled Mstr Mixed Route Neighbor List Member ID Status Serial No Model prio Role Mode Mode ID Interface 0 (FPC 0) Prsnt 1234567890xx ex3400-24p 129 Master* N VC 2 vcp-255/1/0 1 vcp-255/1/1 1 (FPC 1) Prsnt 1234567890xx ex3400-48p 129 Backup N VC 0 vcp-255/1/0 3 vcp-255/1/1 2 (FPC 2) Prsnt 1234567890xx ex3400-24t 0 Linecard N VC 3 vcp-255/1/0 0 vcp-255/1/1 3 (FPC 3) Prsnt 1234567890xx ex3400-24t 0 Linecard N VC 1 vcp-255/1/0 2 vcp-255/1/1 {master:0} root@juniper> request system software add /var/tmp/junos-arm-32-18.1R3-S5.3.tgz unlink reboot force member 3 ERROR: insufficient space for /packages/db/junos-arm-32-18.1R3-S5.3/contents/junos-runtime.tgz ERROR: insufficient space ERROR: Failed to add junos-arm-32-18.1R3-S5.3.tgz veriexec: /usr/libexec/debug.sh: Authentication error veriexec: /usr/libexec/hooks.sh: Authentication error veriexec: /usr/libexec/ui/validate_hooks.sh: Authentication error /usr/libexec/ui/validate-config.sh: DebugOn: not found /usr/libexec/ui/validate-config.sh: DebugOff: not found {master:0} root@juniper> show system storage member 3 fpc3: -------------------------------------------------------------------------- Filesystem Size Used Avail Capacity Mounted on /dev/gpt/junos 1.3G 515M 741M 41% /.mount tmpfs 1.3G 12K 1.3G 0% /.mount/tmp tmpfs 324M 260K 324M 0% /.mount/mfs {master:0} root@juniper>
EX2300 and EX3400 switches come with 2GB of storage space. The Junos install package size (compressed) is typically around 300MB. In this example, the package size of "junos-arm-32-18.4R1.8.tgz" is around 330MB. When a new image needs to be installed on the system, the Junos for EX2300/3400 installation process typically requires 330MB x 3 = ~990MB of free space to be available.
Prior to initiating the upgrade, verify that all members have sufficient storage space (roughly 3 x image size) to avoid failures. If a member does not have enough available space then the image installation would fail. You can verify the available storage space using the following command.
root@juniper> show system storage fpc0: -------------------------------------------------------------------------- Filesystem Size Used Avail Capacity Mounted on /dev/gpt/junos 1.4G 532M 790M 40% /.mount tmpfs 651M 48K 651M 0% /.mount/tmp tmpfs 324M 436K 324M 0% /.mount/mfs
In this output, there is only 790M available in the Junos volume. This will not be sufficient to perform an upgrade, which in this example (18.4R1) will need at least 990M free space.
In a Virtual Chassis environment, first try option 1. if that does not work, do option 2:
Execute storage clean up.
root@juniper> request system storage cleanup all-members
Delete the snapshot on all members:
root@juniper> request system snapshot delete * all-members
/packages/db/
root@juniper> start shell user root root@juniper% cd /packages/db/ root@juniper>% ls -l total 224 drwxr-xr-x 3 root wheel 512 Feb 12 20:34 jdocs-ex-arm-32-20190315.110124_builder_junos_181_r3_s4 drwxr-xr-x 3 root wheel 512 Feb 12 20:34 jpfe-EX34XX-arm-32-20190315.110124_builder_junos_181_r3_s4 drwxr-xr-x 3 root wheel 512 Feb 12 20:34 jsd-arm-32-18.1R3-S4.2-jet-1 drwxr-xr-x 4 root wheel 512 Feb 12 20:34 jsdn-arm-32-18.1R3-S4.2 drwxrwxrwx 5 root wheel 512 Feb 12 19:27 junos-arm-32-15.1X53-D59.4---> old file drwxr-xr-x 4 root wheel 512 Feb 12 20:34 junos-arm-32-18.1R3-S4.2 drwxr-xr-x 3 root wheel 512 Feb 12 20:34 junos-daemons-arm-32-20190315.110124_builder_junos_181_r3_s4 drwxr-xr-x 4 root wheel 512 Feb 12 20:35 junos-dp-crypto-support-ex-arm-32-20190315.110124_builder_junos_181_r3_s4 drwxr-xr-x 3 root wheel 512 Feb 12 19:33 junos-dp-crypto-support-mtx-arm-32-20180914.204810_builder_junos_151_x53_d59--> old file
root@juniper% rm -R [filename]
root@juniper% rm -R junos-arm-32-15.1X53-D59.4 root@juniper% rm -R junos-dp-crypto-support-mtx-arm-32-20180914.204810_builder_junos_151_x53_d59
root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz no-validate reboot
If option 1 does not work, use one of the following methods for a complete and successful software upgrade on the EX2300/EX3400.
Run the " request system storage cleanup all-members" command (won’t clean under /root).
Run the “request system snapshot delete * all-members” command (delete any existing snapshots).
Use the no-copy and force options while performing a software update. For example:
root@juniper> show virtual-chassis Preprovisioned Virtual Chassis Virtual Chassis ID: d0e1.846e.3144 Virtual Chassis Mode: Enabled Mstr Mixed Route Neighbor List Member ID Status Serial No Model prio Role Mode Mode ID Interface 0 (FPC 0) Prsnt 1234567890xx ex3400-24p 129 Master* N VC 2 vcp-255/1/0 1 vcp-255/1/1 1 (FPC 1) Prsnt 1234567890xx ex3400-48p 129 Backup N VC 0 vcp-255/1/0 3 vcp-255/1/1 2 (FPC 2) Inactive 1234567890xx ex3400-24t 0 Linecard N VC 3 vcp-255/1/0 0 vcp-255/1/1 3 (FPC 3) Inactive 1234567890xx ex3400-24t 0 Linecard N VC 1 vcp-255/1/0 2 vcp-255/1/1 {master:0} root@juniper > request system software add /var/tmp/junos-arm-32-18.2R2-S4.3.tgz unlink reboot force member 1
The force option frees up storage space by removing the non-recovery snapshots (if any).
If step 3 fails, run the software update with the force option again (it will work the second time in some situations.)
{master:0} root@juniper> show version | match "fpc|Junos:" fpc0: -------------------------------------------------------------------------- Junos: 18.1R3-S5.3 fpc1: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3 fpc2: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3 fpc3: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3 {master:0} root@juniper> show virtual-chassis status Preprovisioned Virtual Chassis Virtual Chassis ID: d0e1.846e.3144 Virtual Chassis Mode: Enabled Mstr Mixed Route Neighbor List Member ID Status Serial No Model prio Role Mode Mode ID Interface 0 (FPC 0) Prsnt 1234567890xx ex3400-24p 129 Master* N VC 2 vcp-255/1/0 1 vcp-255/1/1 1 (FPC 1) Inactive 1234567890xx ex3400-48p 129 Linecard N VC 0 vcp-255/1/0 3 vcp-255/1/1 2 (FPC 2) Inactive 1234567890xx ex3400-24t 0 Linecard N VC 3 vcp-255/1/0 0 vcp-255/1/1 3 (FPC 3) Inactive 1234567890xx ex3400-24t 0 Linecard N VC 1 vcp-255/1/0 2 vcp-255/1/1
Complete the upgrade for Master RE as above and verify status as bellow:
{master:1} root@juniper> show virtual-chassis Preprovisioned Virtual Chassis Virtual Chassis ID: d0e1.846e.3144 Virtual Chassis Mode: Enabled Mstr Mixed Route Neighbor List Member ID Status Serial No Model prio Role Mode Mode ID Interface 0 (FPC 0) Prsnt 1234567890xx ex3400-24p 129 Backup N VC 2 vcp-255/1/0 1 vcp-255/1/1 1 (FPC 1) Prsnt 1234567890xx ex3400-48p 129 Master* N VC 0 vcp-255/1/0 3 vcp-255/1/1 2 (FPC 2) Prsnt 1234567890xx ex3400-24t 0 Linecard N VC 3 vcp-255/1/0 0 vcp-255/1/1 3 (FPC 3) Prsnt 1234567890xx ex3400-24t 0 Linecard N VC 1 vcp-255/1/0 2 vcp-255/1/1 {master:1} root@juniper> show version | match "fpc|Junos:" fpc0: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3 fpc1: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3 fpc2: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3 fpc3: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3
root@juniper> request system snapshot delete * all-members.
Copy the New SW package file to FPC in line Card mode in /tmp folder which is tmpfs
{master:0} root@juniper> show system storage member 3 fpc3: -------------------------------------------------------------------------- Filesystem Size Used Avail Capacity Mounted on /dev/gpt/junos 1.3G 515M 741M 41% /.mount tmpfs 1.3G 12K 1.3G 0% /.mount/tmp tmpfs 324M 260K 324M 0% /.mount/mfs
Tmpfs is a file system which keeps all files in virtual memory. Everything in tmpfs is temporary in the sense that no files will be created on your hard drive. root@juniper> file copy /var/tmp/junos-arm-32-18.2R2-S4.3.tgz fpc3:/tmp/.
Login to FPC session with "request session member <id> "
{master:0} root@juniper> request session member 3 Last login: Tue Jun 18 08:59:46 from master --- JUNOS 18.1R3-S5.3 Kernel 32-bit JNPR-11.0-20190316.df99236_buil root@juniper:LC:3% cli {linecard:3} root@juniper>
Use the unlink and force options while performing a software update. For example:
root@juniper> request system software add /tmp/junos-arm-32-18.1R3-S5.3.tgz unlink reboot force member 3
If step 5 fails, run the software update with the force option again (it will work the second time in some situations.)
Line will reboot and comes online
{master:0} root@juniper> show virtual-chassis status Preprovisioned Virtual Chassis Virtual Chassis ID: d0e1.846e.3144 Virtual Chassis Mode: Enabled Mstr Mixed Route Neighbor List Member ID Status Serial No Model prio Role Mode Mode ID Interface 0 (FPC 0) Prsnt 1234567890xx ex3400-24p 129 Master* N VC 2 vcp-255/1/0 1 vcp-255/1/1 1 (FPC 1) Prsnt 1234567890xx ex3400-48p 129 Backup N VC 0 vcp-255/1/0 3 vcp-255/1/1 2 (FPC 2) Prsnt 1234567890xx ex3400-24t 0 Linecard N VC 3 vcp-255/1/0 0 vcp-255/1/1 3 (FPC 3) Inactive 1234567890xx ex3400-24t 0 Linecard N VC 1 vcp-255/1/0 2 vcp-255/1/1 {master:0} root@juniper> show version | match "fpc|Junos:" fpc0: -------------------------------------------------------------------------- Junos: 18.1R3-S5.3 fpc1: -------------------------------------------------------------------------- Junos: 18.1R3-S5.3 fpc2: -------------------------------------------------------------------------- Junos: 18.1R3-S5.3 fpc3: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3 {master:0}
Complete the upgrade of all the EX in line card mode and then moved on to the RE mode switches
root@juniper> show virtual-chassis Preprovisioned Virtual Chassis Virtual Chassis ID: d0e1.846e.3144 Virtual Chassis Mode: Enabled Mstr Mixed Route Neighbor List Member ID Status Serial No Model prio Role Mode Mode ID Interface 0 (FPC 0) Prsnt 1234567890xx ex3400-24p 129 Master* N VC 2 vcp-255/1/0 1 vcp-255/1/1 1 (FPC 1) Prsnt 1234567890xx ex3400-48p 129 Backup N VC 0 vcp-255/1/0 3 vcp-255/1/1 2 (FPC 2) Inactive 1234567890xx ex3400-24t 0 Linecard N VC 3 vcp-255/1/0 0 vcp-255/1/1 3 (FPC 3) Inactive 1234567890xx ex3400-24t 0 Linecard N VC 1 vcp-255/1/0 2 vcp-255/1/1 {master:0} root@juniper> show version | match "fpc|Junos:" fpc0: -------------------------------------------------------------------------- Junos: 18.1R3-S5.3 fpc1: -------------------------------------------------------------------------- Junos: 18.1R3-S5.3 fpc2: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3 fpc3: -------------------------------------------------------------------------- Junos: 18.2R2-S4.3
Note: It's highly recommended to take the necessary recovery snapshots after the end of upgrade activity.
2024-06-05: Removed the archive from the headline and updated the article.2020-02-13: Minor, non-technical edits.2020-01-30: Added option 1 in the solution.2020-03-04: KB31198 [juniper.net] has been modified to take of all kind of upgrade which includes VC and standalone environment scenarios as well