Description

When trying to commit a new configuration, the following message is seen:

Please check space available in /var/.

The article will explain how to identify the space available in /var/ directory that might forbid any configuration commit.

Symptoms

The below error message is seen while trying to commit a new configuration: 

{MASTER}[edit]
root@ptx-re0# commit
re0:
error: Could not open configuration database (juniper.data+). Please check space available in /var/. To clear up space run the command 'request system storage cleanup'


Also seen in the log message: 

Apr 21 11:44:29  ptx-re0 mgd[2723]: LIBJNX_FILE_GENERIC_ERROR: System error: Failed to copy /var/rundb/juniper.db to /var/rundb/juniper.data+ -> No space left on device

Solution

This issue is due to the creation of a large number of subdirectories by an internal app. As a result, over a period of time, more than a million subdirectories were created.

First, use ' request system storage cleanup'. Then commit again. If the issue still exists, then thate the steps below.

  1. Check the IUse% value for the var directory

    user@router>start shell user root

    [vrf:none] user@router:~# df -hi

    Filesystem     Inodes IUsed IFree IUse% Mounted on
    /dev/sda6         64K    26   64K    1% /data/config
    /dev/sda1           0     0     0     - /boot
    /dev/sda2        2.0M  1.3K  2.0M    1% /soft
    /dev/sda5        192K  3.5K  189K    2% /data
    /dev/sda7         9.2M   9.2M     0    100% /var.   <-- 100%
    /dev/loop0       960K    13  960K    1% /data/var/external
    devtmpfs         7.8M  1.2K  7.8M    1% /dev
    uswitchd          476   476     0  100% /
    /tmp             7.8M    90  7.8M    1% /tmp
    run              7.8M  2.8K  7.8M    1% /run
    tmpfs            7.8M  1.3K  7.8M    1% /dev/shm
    tmpfs            7.8M    13  7.8M    1% /sys/fs/cgroup
    tmpfs            7.8M     5  7.8M    1% /run/user/0

  2. Check the No. of the free nodes included:

    [vrf:none] user@router :~# tune2fs -l /dev/sda7 | grep -i inode
    Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

    Inode count:              9576448
    Free inodes:                 0        <-- no free inodes


    This is due to the large amount of traces created. In the current example, it is due to the sysmanctl app.

    vrf:none] user@router :/var/log/traces# ls -lrt . |grep sysman |wc -l
    4580
    [vrf:none] user@router     :/var/log/traces# date  
    Thu Nov 12 01:14:17 PST 2020

    [vrf:none] user@router     :/var/log/traces# ls -lrt . |grep sysman |wc -l
    4906
    [vrf:none] user@router     :/var/log/traces# date  
    Thu Nov 12 01:18:56 PST 2020

    Approx. 1 subdirectory/sec * 60* 24* 24 = 86400 ~= 100 k subdirectories per day created by sysmanctl with new pids.


    As a workaround, use the 'clear trace' command to clear old traces.

    Starting with release  20.3R2-EVO and higher, a fix is included to hard limit the number of subdirectories created by an app inside /var/log/traces.