Description

This article provides steps on how to troubleshoot what is taking up the disk space in a Juniper Networks EX Series switch.

Symptoms

When a switch is out of space, several issues may arise, such as the inability to log into the switch or to register new logs. Running the command  ‘show chassis alarms’ on an EX switch, returns the following message:

Minor Host 0 /var partition usage is high.


What could be consuming the disk space?

Example:
When checking the device utilization, the following output is returned:

{master:0}pwd

lab@EX> show system storage
/dev/da0s3e 123M 82M 31M 72% /var

This output does not reveal what could be consuming the disk space.

Running the command, ’file list /var/’ does not provide enough information:

.snap/
BSD.var.dist
account/
at /
backups/
bin/
crash/
cron/
db /
empty/
etc/
etcroot/
heimdal/
home/
jail/
log/
logical-systems/
mail/
mfs/
msgs/
named/
preserve/
root/
run/
rundb/
rwho/
spool/
sw /
tmp/
transfer/
validate/
yp /

Solution

Important Notes: 

  • Unless you are familiar with the Junos OS file system, or you have been advised to do so by a JTAC engineer, do not attempt to delete files outside the /var/home directory, as it can cause your switch to fail.
  • Unless strictly necessary, do not log into the console as ‘root’, because the switch will allow you to delete critical system files.
  • Remember that UNIX does not recognize file extensions, so it is common to have a file with no extension.
  • KB22966 - How to resolve the '/var: filesystem full' issue which occurs as a result of the WTMP file not being archived [juniper.net] . This article explains a similar condition related to the WTMP file taking up space.

The following example uses UNIX shell commands to display ways to find out what is taking up space in an EX switch and remove them:

Note:  Please note that this command is used to find the file that occupies more space in the / partition: “ % find / -size +100000 “ (shell command) 
  1. Enter the command ' start shell user root'

    {master:0}
    lab@EX> start shell user root
    %

    By default, Junos will go to the user's home directory. Confirm this by typing pwd .

    % pwd
    /var/home/lab

  2. Go to the upper level of the /var/ partition by using the following commands:

    % cd /var/
    % pwd
    /var
     
  3. Check folder utilization:

    % du -cks * | sort -rn
    du: cron/tabs: Permission denied
    du: db/entropy: Permission denied
    du: db/certs/common: Permission denied
    du: db/certs/system-key-pair: Permission denied
    du: db/certs/system-cert: Permission denied
    du: db/dhcp_snoop: Permission denied
    du: heimdal: Permission denied
    du: root/.ssh: Permission denied
    du: run/ppp: Permission denied
    du: spool/opielocks: Permission denied
    407216 total
    293838 tmp
    53964 home
    30430 rundb
    20698 root
    6888 log
    378 db
    332 mfs
    204 run
    204 etc
    120 jail
    100 etcroot
    12 spool
    6 at
    4 transfer
    4 sw
    4 crash
    4 BSD.var.dist
    2 yp
    2 validate
    2 rwho
    2 preserve
    2 named
    2 msgs
    2 mail
    2 logical-systems
    2 empty
    2 cron
    2 bin
    2 backups
    2 account

    In this case, it appears that most of the disk space is being consumed by the tmp folder (a partition of its own) and the home file. 
  4. There is a limitation with the du tool. We cannot tell if it is an actual file or a directory. Use the following command to confirm:

    % file home
    home: directory

    Alternatively, the following command can be used:

    % ls -l | grep home
    d rwxr-xr-x 37 root wheel 1024 Sep 5 18:22 home

    The first column shows a lowercase d, which in UNIX indicates it is a directory.
  5. Enter this directory and run the procedure again to find out what is using up the space inside /var/home:

    % cd home/
    % pwd
    /var/home
    % du -cks * | sort -rn
    53962 total
    53802 lab
    46 daltamirano
    16 remote
    14 wheaslip
    12 jrojas
    4 wmoreira

    drwxr-xr-x 3 lab 20 512 Dec 17 09:30 lab

    This reveals that most of the usage comes from the lab directory.
  6. Repeat the procedure once more to see files under the lab directory:

    % cd lab/
    % du -cks * | sort -rn
    53796 total
    28000 foo
    25760 bar
    14 DHCP-CONFIGURATION-MEMO
    8 et inte
    8 JORGE-TEST
    4 h
    2 DHCPclient.PCAP
    0 LC-DHCP-LAB
    0 DHCP.PCAP

    % ls -l | grep foo
    -rw-r--r-- 1 lab field 28655162 Dec 17 09:13 foo
    % ls -l | grep bar
    -rw-r--r-- 1 lab field 26361809 Dec 17 09:13 bar

    The space is being consumed by two files: foo and bar.
  7. Run the files to find out what they are:

    % file foo
    foo: gzip compressed data, from UNIX, max compression
    % file bar
    bar: gzip compressed data, from UNIX, max compression

    This reveals they are tarballs. In most scenarios, /var utilization increases either because tarball upgrade packets were copied to the wrong directory or a packet capture was running for too long.

    % file DHCPclient.PCAP
    DHCPclient.PCAP: tcpdump capture file (little-endian) - version 2.4, capture length 96)
  8. Remove the files:

    % rm foo
    % rm bar
    % ls
    DHCP-CONFIGURATION-MEMO DHCPclient.PCAP LC-DHCP-LAB h
    DHCP.PCAP JORGE-TEST et inte

    Liberating the space in the process:

    /dev/da0s3e 123M 29M 84M 26% /var

    After about 20 minutes, the alarm will clear by itself.
  9. Another helpful command is ' request system storage cleanup dry-run’ 
        
    user@host> request system storage cleanup dry-run
    Currently rotating log files, please wait.
    This operation can take up to a minute.

    List of files to delete:

    Size Date Name
    11.4K Mar 8 15:00 /var/log/messages.1.gz
    7245B Feb 5 15:00 /var/log/messages.3.gz
    11.8K Feb 22 13:00 /var/log/messages.2.gz
    3926B Mar 16 13:57 /var/log/messages.0.gz
    3962B Feb 22 12:47 /var/log/sampled.1.gz
    4146B Mar 8 12:20 /var/log/sampled.0.gz
    4708B Dec 21 11:39 /var/log/sampled.2.gz
    7068B Jan 16 18:00 /var/log/messages.4.gz
    13.7K Dec 27 22:00 /var/log/messages.5.gz
    890B Feb 22 17:22 /var/tmp/sampled.pkts
    65.8M Oct 26 09:10 /var/sw/pkg/jinstall-7.4R1.7-export-signed.tgz
    63.1M Oct 26 09:13 /var/sw/pkg/jbundle-7.4R1.7.tgz
        
  10. You can also check if there is an old version file that could be consuming space with the following steps.
  • Verify the current Junos OS version "show version"

lab@EX> show version
Model: ex2300-48t
Junos: 20.4R3-S1.3

  • Once you know the current version, access the shell and check if there is an old version file in /packages/db directory.

{master:0}
lab@EX> start shell user root
lab@EX:RE:0% cd /var
lab@EX:RE:0% cd /packages/db
lab@EX:RE:0% pwd   --------------> shows the path in which you are located

/packages/db

lab@EX:RE:0% ls -l
total 288
drwxr-xr-x  4 root  wheel  512 Feb 22  2022 fips-mode-arm-32-20211120.090111_builder_junos_204_r3_s1
drwxr-xr-x  2 root  wheel  512 Feb 22  2022 jail-runtime-arm-32-20210909.a3fd70d_builder_stable_11-204ab
drwxr-xr-x  3 root  wheel  512 Feb 22  2022 jpfe-EX34XX-arm-32-20211120.090111_builder_junos_204_r3_s1
drwxr-xr-x  4 root  wheel  512 Feb 22  2022 jphone-home-arm-32-20211120.090111_builder_junos_204_r3_s1
drwxr-xr-x  3 root  wheel  512 Feb 22  2022 jsd-arm-32-20.4R3-S1.3-jet-1
drwxr-xr-x  4 root  wheel  512 Feb 22  2022 jsdn-arm-32-20.4R3-S1.3
drwxrwxrwx  5 root  wheel  512 May 12  2010 junos-arm-32-15.1X53-D590.1 --------> 15.1 file that's consuming space
drwxr-xr-x  4 root  wheel  512 Feb 22  2022 junos-arm-32-20.4R3-S1.3

lab@EX:RE:0% rm -R junos-arm-32-15.1X53-D590.1/ --------------> Remove old file 
lab@EX:RE:0%

In the above example, we see that the switch is currently running version 20.4R3-S1.3, and we still have a file for version 15 that's consuming space, therefore, it was removed to free up space. Also, you could issue the following commands within the shell:
 

lab@EX> start shell user root
lab@EX:RE:0% pkg setop rm previous
lab@EX:RE:0% pkg delete old
lab@EX:RE:0% exit

Or you could issue the following commands:

lab@EX> request system snapshot delete * all-members
lab@EX> request system snapshot delete previous all-members

NOTE:

  • Use "all-members" knob only for VC
  • The aforementioned commands do not apply to EX4200, EX4300.
  • Reboot the unit as it helps delete old pending files.  

In case there is an OAM snapshot, you need to mount the OAM partition in order to delete it with the following commands in shell mode:

mkdir /var/tmp/oam                                <<< Create the new directory where the partition will be mounted
mount /dev/gpt/oam /var/tmp/oam     <<< Mount the partition
cd /var/tmp/oam/snapshot/                 <<< Change the path where the snapshot is
ls                                                               <<< Check the snapshots
rm recovery.ufs.uzip                          <<< Remove the first snapshot file
rm VERSION                                               <<< Remove the second snapshot file

Modification History

2025-05-14: Added steps to delete the OAM snapshot

2020-03-19: Moved important notes to the top of the solution field.
2019-03-05: Minor edit. Non-technical

Related Information