This article provides information about a filesystem full problem seen on the QFX10K devices and how to resolve the problem.

• Following logs seen on the device:

Dec 20 13:30:19.328 QFX10002-Switch kernel: pid 8415 (jdhcpd), uid 0 inumber 47379 on /.mount/var/log: filesystem full

Dec 20 13:30:20.230 QFX10002-Switch kernel: pid 8368 (rpd), uid 0 inumber 147 on /.mount/var/log: filesystem full

Dec 20 13:30:26.909 QFX10002-Switch kernel: pid 8368 (rpd), uid 0 inumber 147 on /.mount/var/log: filesystem full

Dec 20 13:30:27.883 QFX10002-Switch kernel: pid 8368 (rpd), uid 0 inumber 147 on /.mount/var/log: filesystem full

Dec 20 13:30:28.871 QFX10002-Switch kernel: pid 8368 (rpd), uid 0 inumber 147 on /.mount/var/log: filesystem full

Dec 20 13:30:29.955 QFX10002-Switch kernel: pid 8368 (rpd), uid 0 inumber 68 on /.mount/var/log: filesystem full

Dec 20 13:30:29.984 QFX10002-Switch kernel: pid 99364 (gzip), uid 0 inumber 136 on /.mount/var/log: filesystem full

Dec 20 13:30:31.011 QFX10002-Switch kernel: pid 8368 (rpd), uid 0 inumber 68 on /.mount/var/log: filesystem full

• The var log directory is full

{master}

User@QFX10002-Switch> start shell 

Jan 05 00:30:14

% df -h |grep /var/log

/dev/ada1p2      475M  474M  -37M  108%  /.mount/var/log >>>> showing 108% usage
 

• If the /var/log is full then the switch may stop all the logging in the /var/log directory owing to no space available.

The above issue might be seen due to the bloating of the utx.log file in the /var/log directory of the device.

{master}

User@QFX10002-Switch> file list /var/log detail | match utx

-rw-r--r-- 1 root wheel 213352446 Jan 5 00:30 utx.log >>> this file is almost 213 MB

We need to clear this log file to make space in the /var/log directory.

{master}

User@QFX10002-Switch>clear log utx.log 

After clearing the logs the /var/log utilization should come down and the logging will resume in the /var/log directory.

We can explicitly archive the utx.log file with the following setting:
 

set system syslog file utx.log archive size 64k

set system syslog file utx.log archive files 8