Jan 1 01:00:51 JTAC-sw001 /kernel: %KERN-3: pid 49323 (sshd), uid 0 inumber 11271 on /var: filesystem full Jan 1 01:00:53 JTAC-sw001 /kernel: %KERN-3: pid 49323 (sshd), uid 0 inumber 11271 on /var: filesystem full Jan 1 01:54:36 JTAC-sw001 /kernel: %KERN-3: pid 49399 (sshd), uid 0 inumber 11271 on /var: filesystem full Jan 1 01:54:39 JTAC-sw001 /kernel: %KERN-3: pid 49399 (sshd), uid 0 inumber 11271 on /var: filesystem full Jan 1 02:00:51 JTAC-sw001 /kernel: %KERN-3: pid 49423 (sshd), uid 0 inumber 11271 on /var: filesystem full
Along with the above log messages , the "/mfs capacity 106%" were also seen on device
Dec 12 08:56:09 JTAC-sw0019 smid: %USER-3-SMID_MFS_CAPACITY_ERROR: /mfs capacity 106% Dec 12 08:56:09 JTAC-sw0019 smid: %USER-3-SMID_MFS_CAPACITY_ERROR: /mfs capacity 106%
The device will stop logging messages and will display "/var: filesystem full" errors message in log messages
Upon checking the "show system storage detail" command we see the /var in negative value, which shows the /var partition is utilised beyond 100%
root@JTAC-sw001> show system storage fpc0: -------------------------------------------------------------------------- Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 316M 292M -1008.0K 100% / devfs 1.0K 1.0K 0B 100% /dev /dev/md0 416M 416M 0B 100% /packages/mnt/junos /dev/md1 6.7M 2.1M 4.1M 33% /packages/mfs-fips-mode-powerpc /dev/md2 5.5M 5.5M 0B 100% /packages/mnt/fips-mode-powerpc-21.4R3-S4.18 /dev/md3 8.3M 3.7M 4.0M 48% /packages/mfs-jcrypto-ex /dev/md4 10M 10M 0B 100% /packages/mnt/jcrypto-ex-21.4R3-S4.18 /dev/md5 9.2M 4.6M 3.9M 54% /packages/mfs-jdocs-ex /dev/md6 13M 13M 0B 100% /packages/mnt/jdocs-ex-21.4R3-S4.18 /dev/md7 7.7M 3.1M 4.0M 44% /packages/mfs-jphone-home /dev/md8 9.3M 9.3M 0B 100% /packages/mnt/jphone-home-21.4R3-S4.18 /dev/md9 6.3M 1.6M 4.2M 28% /packages/mfs-jsd /dev/md10 5.6M 5.6M 0B 100% /packages/mnt/jsd-powerpc-21.4R3-S4.18-jet-1 /dev/md11 11M 6.7M 3.7M 64% /packages/mfs-jsdn-powerpc /dev/md12 6.6M 6.6M 0B 100% /packages/mnt/jsdn-powerpc-21.4R3-S4.18 /dev/md13 112M 99M 3.9M 96% /packages/mfs-junos-ex-4300 /dev/md14 257M 257M 0B 100% /packages/mnt/junos-ex-4300-21.4R3-S4.18 /dev/md15 14M 9.6M 3.4M 74% /packages/mfs-jweb-ex /dev/md16 28M 28M 0B 100% /packages/mnt/jweb-ex-21.4R3-S4.18 /dev/md17 11M 6.4M 3.7M 63% /packages/mfs-py-base-powerpc /dev/md18 18M 18M 0B 100% /packages/mnt/py-base-powerpc-21.4R3-S4.18 /dev/md19 9.0M 4.4M 3.9M 53% /packages/mfs-py-extensions-powerpc /dev/md20 14M 14M 0B 100% /packages/mnt/py-extensions-powerpc-21.4R3-S4.18 /dev/md21 9.2M 4.6M 3.9M 54% /packages/mfs-restapi-ex /dev/md22 12M 12M 0B 100% /packages/mnt/restapi-ex-21.4R3-S4.18 /dev/da0s3e 170M 166M -10.1M 106% /var. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< /dev/md23 252M 10.0K 232M 0% /tmp /dev/da0s3d 887M 99M 717M 12% /var/tmp /dev/da0s4d 116M 114K 106M 0% /config /dev/md24 236M 104M 113M 48% /var/rundb procfs 4.0K 4.0K 0B 100% /proc /var/jail/etc 170M 166M -10.1M 106% /packages/mnt/jweb-ex-21.4R3-S4.18/jail/var/etc /var/jail/run 170M 166M -10.1M 106% /packages/mnt/jweb-ex-21.4R3-S4.18/jail/var/run /var/jail/tmp 170M 166M -10.1M 106% /packages/mnt/jweb-ex-21.4R3-S4.18/jail/var/tmp /var/tmp 887M 99M 717M 12% /packages/mnt/jweb-ex-21.4R3-S4.18/jail/var/tmp/uploads /var/jail/sess 170M 166M -10.1M 106% /packages/mnt/jweb-ex-21.4R3-S4.18/jail/var/sess /var/jail/log 170M 166M -10.1M 106% /packages/mnt/jweb-ex-21.4R3-S4.18/jail/var/log devfs 1.0K 1.0K 0B 100% /packages/mnt/jweb-ex-21.4R3-S4.18/jail/dev /var/jail/jweb-app 170M 166M -10.1M 106% /packages/mnt/jweb-ex-21.4R3-S4.18/jail/var/jweb-app /var/jails/rest-api 170M 166M -10.1M 106% /packages/mnt/restapi-ex-21.4R3-S4.18/web-api/var devfs 1.0K 1.0K 0B 100% /packages/mnt/restapi-ex-21.4R3-S4.18/web-api/dev /dev/md25 6.7M 2.1M 4.1M 33% /packages/mfs-fips-mode-powerpc /dev/md26 8.3M 3.7M 4.0M 48% /packages/mfs-jcrypto-ex /dev/md27 9.2M 4.6M 3.9M 54% /packages/mfs-jdocs-ex /dev/md28 7.7M 3.1M 4.0M 44% /packages/mfs-jphone-home /dev/md29 6.3M 1.6M 4.2M 28% /packages/mfs-jsd /dev/md30 11M 6.7M 3.7M 64% /packages/mfs-jsdn-powerpc /dev/md31 112M 99M 3.9M 96% /packages/mfs-junos-ex-4300 /dev/md32 14M 9.6M 3.4M 74% /packages/mfs-jweb-ex /dev/md33 11M 6.4M 3.7M 63% /packages/mfs-py-base-powerpc /dev/md34 9.0M 4.4M 3.9M 53% /packages/mfs-py-extensions-powerpc /dev/md35 9.2M 4.6M 3.9M 54% /packages/mfs-restapi-ex
We need to collect following data from the customer device in problematic state , Shell Commands : ***************** Check for folder utlizations: % date %cd /var %du -cks * | sort -rn % du -ahx /var % df -h /var % df -i % df -i /var % ps -aux % ps -ax -o pid,command,lstart % fstat -f /var % fstat -f % du -ahx / % df -h / % mount % ifsmon -c % ifsmon -z % history CLI commands : ***************** show version show system uptime show system alarms show system storage show system boot-messages show system storage partitions | no-more show system snapshot media internal | no-more show configuration | display set | no-more show chassis routing-engine show chassis fpc show system memory show system processes extensive Additional Data : ***************** % complete /var/log ( from all the problematic device) % Fresh RSI from all the non-working devices. >>From the above outputs collected , we should be able to check which folder and files are consuming more space and contributing to the issue. >> Delete those files to free memory if they are not required >>If we are unable to determine the files that could be consuming memory , then follow any of the below step. >> Perform format install on the device . >>If Format install is not possible or feasible option then RMA the device .
2024-04-01 : Article Created
2025-09-09 : Updated the shell commands