EX3400 error while upgrading master fpc "ERROR: insufficient space for /packages/db/junos-arm-32-23.4R2-S4.11"
Problem Description:
=================
Customer is getting error only on master fpc0 while upgrading the 2 member VC switch from 23.2R1-S2.5 to junos-arm-32-23.4R2-S4.11.tgz
ERROR: insufficient space for /packages/db/junos-arm-32-23.4R2-S4.11/contents/junos-runtime.tgz
ERROR: insufficient space
ERROR: Failed to add junos-arm-32-23.4R2-S4.11.tgz
root@ex3400-vc> show system storage
fpc0:
--------------------------------------------------------------------------
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/junos 1.3G 832M 424M 66% /.mount
tmpfs 526M 40K 526M 0% /.mount/tmp
tmpfs 323M 680K 322M 0% /.mount/mfs
/dev/gpt/oam 484M 76M 369M 17% /.mount/oam
fpc1:
/dev/gpt/junos 1.3G 522M 734M 42% /.mount
tmpfs 829M 16K 829M 0% /.mount/tmp
tmpfs 323M 600K 323M 0% /.mount/mfs
{master:0}
root@ex3400-vc>
Tried the following steps which did not help to resolve this :
request system storage cleanup
request system snapshot delete snap*
start shell command "pkg setop rm previous"
start shell command "pkg delete old”
request system software delete web-ex
Suggested to follow this KB to add "os-package" and "package-hokks-ex":
https://supportportal.juniper.net/s/article/EX-Not-enough-storage-while-upgrading-Junos-EX2300-and-EX3400?language=en_US
Still customer is seeing problem after trying these steps explained in the above KB.
Find that /var/rundb is utilising more space.
Customer VC has been recovered from the problem state following a power maintenance on the switches.
Need to collect the output of the command run as root from the shell from the customer switch to debug further:
mkdir /somedir && mount -t nullfs / /somedir && du -xah /somedir && umount /somedir && rmdir /somedir