Description

Unable to upgrade ACX device due to lack of storage space. This article provides the steps to use re-partitioning and create enough space for an upgrade.

Symptoms

Unable to proceed with upgrade on ACX. The following message is seen:

lab@root> .../jinstall-host-acx5k-17.3R3.10-signed.tgz
Fetching package...
 
/var/tmp/incoming-package.tgz                          561 MB  361 kBps
Verified jinstall - vjunos -17.3R3.10.tgz signed by JuniperSwitchingProduction_2015
tar: vjunos -17.3R3.10.tgz: Wrote only 0 of 10240 bytes
tar: Skipping to next header
tar: Error exit delayed from previous errors
ERROR: jinstall - vjunos -17.3R3.10-signed fails post-install

The workaround provided in PR1088017 suggests to upgrade the device by using a USB.  However, there are times or situations where the device is not physically near and cannot be accessed.

Solution

On ACX5000 routers, the following operations may fail due to a partition of the file-system running out of space:

  • Upgrade
  • Commit of a large configuration file. It requires /var/tmp to be 1.9 Gig.

lab@root> show system storage 
fpc0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             501M       389M        72M       84%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                 86M        86M         0B      100%  /packages/mnt/jbase
/dev/md1                463M       463M         0B      100%  /packages/mnt/jkernel-acx5k-15.1X54-D51.7
/dev/md2                114M       114M         0B      100%  /packages/mnt/jpfe-acx5k-15.1X54-D51.7
/dev/md3                 11M        11M         0B      100%  /packages/mnt/jdocs-dc-re-15.1X54-D51.7
/dev/md4                261M       261M         0B      100%  /packages/mnt/jroute-acx5k-15.1X54-D51.7
/dev/md5                 32M        32M         0B      100%  /packages/mnt/jcrypto-dc-re-15.1X54-D51.7
/dev/md6                 50M        50M         0B      100%  /packages/mnt/jswitch-acx5k-15.1X54-D51.7
/dev/md7                 12M        12M         0B      100%  /packages/mnt/py-base-i386-15.1X54-D51.7
/dev/md8                605M       8.0K       556M        0%  /tmp
/dev/ad2s1e             504M       122K       463M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f             966M        35M       854M        4%  /var
/dev/ad1s1d             1.5G        20K       1.3G        0%  /var/tmp
/dev/ad1s1a             242M       1.7M       221M        1%  /var/log
/dev/ad1s1e             484M       8.0K       445M        0%  /altconfig
/dev/ad3s1             1006M       4.0K       926M        0%  /user
/var/jails/rest-api       966M        35M       854M      4%  /packages/mnt/jroute-acx5k-15.1X54-D51.7/web-api/var

Please follow the steps below: 

  1. lab@root>start shell user root  
Password:
lab@root​%
lab@root​% cd /etc
lab@root​% mount -rw /
lab@root​% cp rc.pre-early rc.pre-early.original
lab@root​% chmod 777 rc.pre-early

  2. Make changes given in diff below: 

    lab@root% diff -u rc.pre-early rc.pre-early.original
--- rc.pre-early   Wed Mar  6 10:57:56 2019
+++ rc.pre-early.original Wed Mar  6 10:12:59 2019
@@ -224,9 +224,9 @@
     swapsz=15%     # ~600M 
     # On ACX models make /var as 1 GB
-    if [ "${PRODUCT_MODEL.EN_US}" = "acx5048" -o "${PRODUCT_MODEL.EN_US}" = "acx5096" ]; then
-       varsz=24%   # ~1GB
-    fi
+#    if [ "${PRODUCT_MODEL.EN_US}" = "acx5048" -o "${PRODUCT_MODEL.EN_US}" = "acx5096" ]; then
+#      varsz=24%   # ~1GB
+#    fi
 
        # Size based on 4GB data disk starting offset 0
     echo "#" > ${LABEL.EN_US}
@@ -508,6 +508,7 @@
                format_vm_data_disk=1
            fi
 
+           format_vm_data_disk=1
             if [ ${FORMAT_VM_DATA_DISK.EN_US} -eq 1 ]; then
                 format_vm_data_disk ${DRIVE.EN_US} ${SLICE.EN_US} ${USER_SLICE.EN_US}
                 if [ $? != 0 ]; then
Exit

  3. Request system reboot: 

    {master:0}
lab@root> request system reboot 
Reboot the system ? [yes,no] (no) yes 

Shutdown at Wed Mar  6 11:06:05 2019.
[pid 26428]

{master:0}
>                                                                                
*** System shutdown message from remote@blvr-i-5048 ***                  

System going down in 1 minute                                                  

*** System shutdown message from remote@blvr-i-5048 ***                   

System going down in 30 seconds

  4. While booting, observe that re-partition gets executed. Try 'show system storage' to see /var/tmp has 1.9G space 

    {master:0}
lab@root> show system storage
fpc0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             501M       389M        72M       84%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                 86M        86M         0B      100%  /packages/mnt/jbase
/dev/md1                463M       463M         0B      100%  /packages/mnt/jkernel-acx5k-15.1X54-D51.7
/dev/md2                114M       114M         0B      100%  /packages/mnt/jpfe-acx5k-15.1X54-D51.7
/dev/md3                 11M        11M         0B      100%  /packages/mnt/jdocs-dc-re-15.1X54-D51.7
/dev/md4                261M       261M         0B      100%  /packages/mnt/jroute-acx5k-15.1X54-D51.7
/dev/md5                 32M        32M         0B      100%  /packages/mnt/jcrypto-dc-re-15.1X54-D51.7
/dev/md6                 50M        50M         0B      100%  /packages/mnt/jswitch-acx5k-15.1X54-D51.7
/dev/md7                 12M        12M         0B      100%  /packages/mnt/py-base-i386-15.1X54-D51.7
/dev/md8                605M       460K       556M        0%  /tmp
/dev/ad2s1e             504M       122K       463M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f             484M        23M       422M        5%  /var
/dev/ad1s1d             1.9G        20K       1.8G        0%  /var/tmp
/dev/ad1s1a             242M       464K       222M        0%  /var/log
/dev/ad1s1e             484M       8.0K       445M        0%  /altconfig
/dev/ad3s1             1006M       4.0K       926M        0%  /user
/var/jails/rest-api       484M        23M       422M      5%  /packages/mnt/jroute-acx5k-15.1X54-D51.7/web-api/var

    Var is smaller and var tmp is larger. Now, you can upgrade Junos.

Related Information