This article explains how, in certain scenarios, cron jobs may stop running on line cards. When this occurs, the file size in "/var/log" can increase because the "UKERN_GBL" file does not roll over into a gz, eventually causing the line card of the VC to run out of space.
Symptoms
In some scenarios, cron jobs are not respawned on line cards, which can lead "UKERN_GBL" file not rolling over into a gz file, eventually causing the line card of the VC to run out of space. Once this happens, a space cleanup action is required to ensure logging continues to function properly and to prevent potential problems caused by full storage. In this case, we can see that FPC5 is showing elevated storage utilization:
root@user> show system storage
fpc0:
--------------------------------------------------------------------------
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/junos 15G 1.3G 13G 9% /.mount
tmpfs 20M 48K 20M 0% /.mount/tmp
/var/jails/rest-api 15G 1.3G 13G 9% /.mount/packages/mnt/junos-runtime/web-api/var
tmpfs 609M 816K 608M 0% /.mount/mfs
fpc1:
tmpfs 609M 744K 608M 0% /.mount/mfs
fpc2:
/dev/gpt/junos 15G 1.1G 13G 7% /.mount
tmpfs 20M 44K 20M 0% /.mount/tmp
/var/jails/rest-api 15G 1.1G 13G 7% /.mount/packages/mnt/junos-runtime/web-api/var
tmpfs 609M 408K 608M 0% /.mount/mfs
fpc3:
/dev/gpt/junos 15G 998M 13G 7% /.mount
/var/jails/rest-api 15G 998M 13G 7% /.mount/packages/mnt/junos-runtime/web-api/var
fpc4:
/dev/gpt/junos 15G 1.4G 13G 10% /.mount
/var/jails/rest-api 15G 1.4G 13G 10% /.mount/packages/mnt/junos-runtime/web-api/var
fpc5:
/dev/gpt/junos 15G 15G -1.2G 109% /.mount
/var/jails/rest-api 15G 15G -1.2G 109% /.mount/packages/mnt/junos-runtime/web-api/var
tmpfs 609M 580K 608M 0% /.mount/mfs
When checking the files on FPC5, we can see that the "UKERN_GBL.log" file has grown to over 15 GB. The file is not rolling over into a .gz archive as expected, which is causing space on the line card to fill up:
"UKERN_GBL.log
.gz
% ls -l /var/log
total 30255104
-rw-rw---- 1 root wheel 15442371716 Feb 12 21:09 UKERN_GBL.log
-rw-r--r-- 1 root wheel 1862 Sep 4 18:23 __policy_names_rpdc__
When checking, cron job is not running on this line card:
root@user:LC:5% ps aux | grep cron
Expected output:
root@user:LC:3% ps aux | grep cron
root 10268 0.0 0.1 17968 3984 - Ss 22:33 0:00.03 /usr/sbin/cron -s
Here are the logs showing cron job rotating files on the members where the daemon is running:
Feb 12 18:51:00 user /usr/sbin/cron[95321]: (root) CMD ( newsyslog -X -f /var/etc/ukern_gbl.logrotate.conf)
Feb 12 18:54:00 user /usr/sbin/cron[95495]: (root) CMD ( newsyslog -X -f /var/etc/ukern_gbl.logrotate.conf)
Feb 12 18:57:00 user /usr/sbin/cron[95573]: (root) CMD ( newsyslog -X -f /var/etc/ukern_gbl.logrotate.conf)
As a restoration step, you can clean up the storage space and then manually trigger the cron job using "/usr/sbin/cron -s -n", however, this issue has already been addressed in the following software releases:
Resolved-In
junos:21.4R3-S12
junos:22.4R3-S8
junos:23.2R2-S5
junos:23.4R2-S5
junos:24.2R2-S1
junos:24.2R2-S2
junos:24.4R1-S2
junos:24.4R1-S3
junos:24.4R2
junos:25.2R1
junos:25.3R1