Description

  • Juniper Networks EX2300 and EX3400 Series switches are equipped with a 2GB flash disk, the disk size is sufficient to be able to store release images, system files, and logs. However, when storage space becomes more heavily utilized over time, there may not be enough disk space available to complete a software update.
  • This article is written to provide guidance on how to manage storage space and potentially overcome any obstacles tied to space that are encountered in a software update.
  • If for any reason the information in this article does not address the issues tied to disk space that are encountered during a software update, contact Support for further assistance.

Symptoms

If an issue with available storage space is encountered during a software update, an error will be returned by the system indicating lack of space. The specific error returned will differ slightly depending on how the switch is deployed, either as a standalone device or as part of a Virtual Chassis.

For a system operating as a standalone device, the following error will be seen:

root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz no-copy no-validate
reboot
Removing /var/log/dcd.0.gz
Removing /var/log/wtmp.0.gz
Removing /var/log/install.0.gz
Removing /var/log/chassisd.1.gz
Removing /var/log/chassisd.3.gz
Removing /var/log/dhcp_logfile.0.gz
Removing /var/log/chassisd.0.gz
Removing /var/log/chassisd.2.gz
Removing /var/log/chassisd.8.gz
Removing /var/log/chassisd.7.gz
Removing /var/log/chassisd.6.gz
Removing /var/log/chassisd.5.gz
Removing /var/log/chassisd.4.gz
Removing /var/preserve/jdhcp_client_data
Removing /var/preserve/jdhcp_client_data_bkp
Error: not enough space to unpack /var/tmp/junos-arm-32-18.4R1.8.tgz
ERROR: Either use 'force' or consider deleting snapshots using 'request system snapshot
delete <snap>'

root@juniper> show system storage
fpc0:
--------------------------------------------------------------------------
Filesystem      Size  Used  Avail  Capacity  Mounted on
/dev/gpt/junos  1.3G  1.1G  139M   89%       /.mount <-- Not enough space
tmpfs           651M  48K   651M   0%        /.mount/tmp
tmpfs           324M  436K  324M   0%        /.mount/mfs

As seen above, only 139MB is available for use. This is an insufficient amount of space.

For a system operating in a Virtual Chassis, the following error would be returned when the software update is initiated:

root@juniper> request system software add /var/tmp/junos-arm-32-18.1R3-S5.3.tgz unlink reboot force 
ERROR: insufficient space for /packages/db/junos-arm-32-18.1R3-S5.3/contents/junos-runtime.tgz
ERROR: insufficient space
ERROR: Failed to add junos-arm-32-18.1R3-S5.3.tgz

And, as in the previous example, the available storage seen from show system storage member <x> is insufficient to handle the update:

root@juniper> show system storage member 3   
fpc3:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/gpt/junos          1.3G       1.1G       139M       89%  /.mount <-- Not enough space
tmpfs                   716M        48K       716M        0%  /.mount/tmp
tmpfs                   324M       396K       324M        0%  /.mount/mfs

 

Solution

  • The 2GB of total disk space in EX2300 and EX3400 switches is broken down into two partitions. The first partition, which is used by Junos OS, is allocated 1.3GB to 1.5 GB of space. This is the volume seen under /dev/gpt/junos/ above. The second partition that uses the remainder of the disk is tied to the Operation, Administration and Management (OAM) volume. This is used for the storage of recovery snapshots. Recovery snapshots contain full copies of the software packages and configuration that the system is running to be used in the event of a failure.
  • The Junos OS install package sizes are typically ~300MB, which is the case for the image used in this example (junos-arm-32-18.4R1.8.tgz). So, the 139MB that is available in the system above would be insufficient. However, even if the system had that much space open, the update would still fail. The failure would be tied to how the software installation process works.
  • The installation process takes the following actions. First, the new software image is copied to the Junos OS volume on the switch. Once the image is copied, the system will check to see if there is enough space to unpack the compressed image. To do so, the switch will unzip the image and the packages found within the .tgz file. Before unzipping, it looks for free space, defined as 3X the package size for the operation to complete. This means that if the image size is 300MB, the update will fail unless there is more than 900MB of available space.
  • As discussed above, the size of Junos OS volume in EX2300 and EX3400 switches may not be sufficient, given the need for 3X the package size to complete the update process. A check of show system storage in a standalone switch or show system storage member <X> via the CLI prior to beginning the update process will give an indication if the update will be successful.
  • If there is insufficient disk space, Juniper has developed a solution that uses two independent software packages to handle the update. An “os-package” and a “package-hooks-ex” package to help manage disk space and complete the software update. These packages are available at the Juniper Software Download page.

EX2300-EX3400-os-package

EX2300-EX3400-package-hooks-ex

Juniper recommends using these packages to reclaim disk space if there is insufficient space to perform the software update.
 

Steps:

1. Run the request system storage cleanup command. This will remove any unnecessary files but keep everything under /root intact:

root@juniper> request system storage cleanup

2. Run the request system snapshot delete snap* command. This will delete any existing recovery snapshots that are stored on the system.

root@juniper> request system snapshot delete snap*


Note: If you are unable to delete the  recovery.ufs  snapshot, please follow the steps under KB32170 - Recovery and Non-recovery snapshots on Junos devices [juniper.net]. 


3.  Copy the packages  os-package and package-hooks-ex to /var/tmp .

4. Install the two packages by using the request system software add command:

root@juniper> request system software add /var/tmp/os-package.tgz
root@juniper> request system software add /var/tmp/package-hooks-ex.tgz
      Note: Please do not reboot after installation of the "os-package" and "package-hokks-ex", reboot only after the installation of the Junos image "junos-arm-32-18.4R1.8.tgz" is complete.


5. Install the Junos OS release software by adding options no-copy, unlink, and force under the request system software add command:

root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force unlink no-copy
 

Note: The unlink option is available starting Junos OS Release 18.1R2, please exclude if running earlier versions. Unlink option removes the .tgz file immediately after the unpack so that the image size calculation takes into account the freed-up space tied to the image deletion. This effectively reduces the amount of space needed by the process to 2X instead of 3X. Note that there are some occasions where the add will fail. Try it a second time before moving on.

In Virtual Chassis scenarios where the upgrade fails on the Backup RE or Linecards with log " ERROR: No packages added", option no-copy may need to be excluded.

6. If there is not enough storage after the above, use the following commands as a last resort and repeat step 5:

root@juniper> start shell command "pkg setop rm previous"
root@juniper> start shell command "pkg delete old"
 

Note: The commands above will only affect the primary Routing Engine, for specific devices part of the Virtual Chassis, login to each of them with the command "request session member <member ID>" and then run the two commands above. The commands "pkg setop rm previous" and "pkg delete old" can be run directly from shell or Operational Mode as shown above.


7. Reboot the system to complete the Junos OS release SW installation process.

After the upgrade is complete, remove the two packages and reboot the system. The show version command will provide information about install packages as shown in the below example.

  1. os-package

  2. package hooks

root@juniper> show version                             
fpc0:
--------------------------------------------------------------------------
Hostname: d11-12
Model: ex3400-48t
Junos: 18.2R3-S2.7
JUNOS OS Kernel 32-bit  [20191022.14c2ad5_builder_stable_11]
JUNOS OS libs [20191022.14c2ad5_builder_stable_11]
JUNOS OS runtime [20191022.14c2ad5_builder_stable_11]
JUNOS OS time zone information [20191022.14c2ad5_builder_stable_11]
JUNOS py extensions [20191029.212835_builder_junos_182_r3_s2]
JUNOS py base [20191029.212835_builder_junos_182_r3_s2]
JUNOS network stack and utilities [20191029.212835_builder_junos_182_r3_s2]
JUNOS libs [20191029.212835_builder_junos_182_r3_s2]
JUNOS runtime [20191029.212835_builder_junos_182_r3_s2]
JUNOS package hooks ex [20191022.160523_sjg_dev_common]
JUNOS OS package [20191113.002318_sjg_stable_11]
JUNOS OS crypto [20191022.14c2ad5_builder_stable_11]
JUNOS Web Management Platform Package [20191029.212835_builder_junos_182_r3_s2]

If they are not deleted and the system is rebooted, you will receive a notice after the boot process is completed saying: "At least one package installed on this device has limited support." Delete the two packages by using the request system software delete command:

root@juniper> request system software delete os-package   
root@juniper> request system software delete package-hooks-platform

Now reboot the system to complete the uninstallation process for the packages:

root@juniper> request system reboot all-members

If copying the image to /var/tmp doesn't work, image can be copied to /tmp as an alternative.

 

8. If problem remains set device back to default settings using "request system zeroize". Please save the configuration prior this step so it could be added back to the switch once the zeroize completes.

9. Alternatively, you may force the device to boot from a Bootable USB containing the desired OS by following KB31265 [juniper.net]  

Notes:

  • NSSU will not work with these options.
  • It is highly recommended to take the necessary recovery snapshots after the upgrade activity is complete.

Modification History

2024-06-27: Added "Note: Please do not reboot after installation of the "os-package" and "package-hokks-ex", reboot only after the installation of the Junos image "junos-arm-32-18.4R1.8.tgz" is complete."
2023-08-28: Added Training link to "Introduction to the Junos Operating system"
2023-03-10: Added two additional commands to clear space up. 
2021-03-11: Removed former step 3 as per PR1572323 conversation:  os-package  and  package-hooks-ex should do the same without the need to be root
2020-10-02: Non-technical edits.
2020-08-18: Added "Note: If you are unable to delete the recovery.ufs snapshot, follow the steps under KB32170 [juniper.net]."
2020-02-10: Added the two packages and their download link for seamless software upgrades
2019-05-14: Removed old commands and added new data
2019-05-03: Removed sentences that might cause problems to device
2018-11-02: Minor edit. Non-technical
2018-10-29: Updated with 'force unlink'
2018-02-01: Added 'Please raise a JTAC case if the problem persists'
2017-08-07: Added example for checking snapshots in the Solution section.
2017-05-19: Added information about 3x image size requirement for the upgrade and not 1.5 times. Also added Junos 15.1X53-D56 option to use "force" command while upgrading.
2017-03-21: Removed references to OCCAM.

 

Related Information

Training Courses: