This article provides information on how to resolve the '/var: filesystem full' issue, which occurs as a result of the WTMP file not being archived due to its large size.

• Customers may encounter the /var file system full issue, due to the large size of the WTMP file (which consists of login records for UNIX based systems).
• If the WTMP file is manually deleted to free up storage space, the system may not able to log user login information, so no output of the last command from the shell will be logged.

WTMP consists of login records in Junos OS and is located under /var/log . By default, In the current Junos OS version, the /var/log/wtmp file is rotated once a month. If it grows large enough in this period, it may fill up the /var partition.

For example:

/kernel: %KERN-3: pid 561 (eventd), uid 0 inumber 12316 on /var: filesystem full 

On checking the files system, the /var partition was found to be over 100%:

 root@EX4200> show system storage no-forwarding

 Filesystem              Size       Used      Avail  Capacity   Mounted on
 /dev/da0s2a             184M       103M        67M       61%  /
 devfs                   1.0K       1.0K         0B      100%  /dev
 devfs                   1.0K       1.0K         0B      100%  /dev/
 /dev/md0                 37M        37M         0B      100%  /packages/mnt/jbase
 /dev/md1                 16M        16M         0B      100%  /packages/mnt/jcrypto-ex-9.5R3.7
 /dev/md2                5.4M       5.4M         0B      100%  /packages/mnt/jdocs-ex-9.5R3.7
 /dev/md3                115M       115M         0B      100%  /packages/mnt/jkernel-ex-9.5R3.7
 /dev/md4                 16M        16M         0B      100%  /packages/mnt/jpfe-ex42x-9.5R3.7
 /dev/md5                 38M        38M         0B      100%  /packages/mnt/jroute-ex-9.5R3.7
 /dev/md6                 21M        21M         0B      100%  /packages/mnt/jswitch-ex-9.5R3.7
 /dev/md7                 18M        18M         0B      100%  /packages/mnt/jweb-ex-9.5R3.7
 /dev/md8                126M      10.0K       116M        0%  /tmp
 /dev/da0s2f             123M       122M      -9.4M      108%  /var <---
 /dev/da0s3d             314M       144K       289M        0%  /var/tmp
 /dev/da0s3e              55M       190K        51M        0%  /config
 /dev/md9                118M        11M        97M       10%  /var/rundb
 procfs                  4.0K       4.0K         0B      100%  /proc
 /var/jail/etc           123M       122M      -9.4M      108%  /packages/mnt/jweb-ex-9.5R3.7/jail/var/etc
 /var/jail/run           123M       122M      -9.4M      108%  /packages/mnt/jweb-ex-9.5R3.7/jail/var/run
 /var/jail/tmp           123M       122M      -9.4M      108%  /packages/mnt/jweb-ex-9.5R3.7/jail/var/tmp
 /var/tmp                314M       144K       289M        0%  /packages/mnt/jweb-ex-9.5R3.7/jail/var/tmp/uploads
 devfs                   1.0K       1.0K         0B      100%  /packages/mnt/jweb-ex-9.5R3.7/jail/dev 

The root cause can be identified by using the ls command. In the following output, WTMP already has over 125M; so the /var partition is full.

 root@EX4200:RE:0% ls -l /var/log/wtmp*
 -rw-rw-r--  1 root  wheel  125188228 Feb 11 06:11 wtmp
 -rw-rw-r--  1 root  wheel     211994 Jan 21 06:59 wtmp.0.gz
 -rw-rw-r--  1 root  wheel       1160 Dec 22 06:45 wtmp.1.gz
 -rw-rw-r--  1 root  wheel        864 Nov 23 05:16 wtmp.2.gz
 -rw-rw-r--  1 root  wheel        781 Oct 25 02:39 wtmp.3.gz
 -rw-rw-r--  1 root  wheel        737 Sep 27 02:39 wtmp.4.gz 

To resolve the issue, the WTMP log need to be cleared to release the space.
To clear the WTMP file, use the following Junos OS command:

root@EX4200> clear log wtmp
 

Note : In later Junos versions with the fix of PR681369 , the 'once every month' criteria for WTMP rotation is changed to once every month or if the file size is bigger than 10MB. If rollout is not happening automatically, then run the command set system syslog file wtmp archive files 10 size 1M binary-data to enable it.

2018-10-19: Updated the work-around in the solution to a simpler method.