This article addresses the issue of disk usage alarms triggered by excessive log accumulation in the /.old/var/log directory, particularly after multiple chassis reboots on NG RE. These reboots, often due to power outages, cause the /.old partition to exceed 95% usage, leading to disk usage alarms. The /.old partition has only 2 GB of space, and frequent log writes after each reboot results in crossing the threshold.
After multiple chassis reboots, the system reports disk usage alarms due to the /.old partition exceeding 95% usage. This issue is characterized by the following symptoms:
Example Logs Indicating the Issue
Alarm seen after a system reboot
May 15 11:42:37 2024 eventd[18433]: %SYSLOG-5-SYSTEM_ABNORMAL_SHUTDOWN: System abnormally shut down May 15 11:42:37 2024 eventd[18433]: %SYSLOG-5-SYSTEM_OPERATIONAL: System is operational May 15 11:42:38 2024 kernel: %KERN-2: ---<<BOOT>>--- May 15 11:42:38 2024 kernel: %KERN-2: Copyright (c) 1998-2023 Juniper Networks, Inc. May 15 11:42:38 2024 kernel: %KERN-2: Copyright (c) 1992-2020 The FreeBSD Project. > show system alarms 1 alarms currently active Alarm time Class Description 2024-05-15 11:43:45 CEST Minor VMHost RE 0 Disk 1 Usage is above threshold > show vmhost resource-usage Compute cluster: rainier-re-cc Disk Usage ========== Filesystem Size Used Avail Use% Mounted on /dev/sda4 2.0G 1.8G 80M 96% /.old <--- more than 95 percent devtmpfs 15G 4.0K 15G 1% /dev tmpfs 16G 68K 16G 1% /dev/shm tmpfs 16G 2.2M 16G 1% /run <..>
Based on the investigation in PR 1815103, Engineering has identified the following solution to address this issue:
Remove the /.old/boot/vmlinux* files. These are uncompressed kernel files used for debugging and analysis purposes. Deleting these files can free up around 270 MB of space. This solution is implemented in the latest releases starting with LTS22 from the 23.4 release. Engineering will backport this fix to lower releases via the PR1815103.
root@node:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 2.0G 1.7G 140M 93% /.old devtmpfs 15G 4.0K 15G 1% /dev root@node:/.old# ls -ltr /.old/boot/vmlinux* -rw-r--r-- 1 root root 272081928 Dec 26 2022 /.old/boot/vmlinux-4.8.28-rt10-WR9.0.0.24_ovp lrwxrwxrwx 1 root root 40 Dec 26 2022 /.old/boot/vmlinux -> /boot/vmlinux-4.8.28-rt10-WR9.0.0.24_ovp
Remove the file :
root@node:/.old# rm /.old/boot/vmlinux* root@node:/.old# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 2.0G 1.5G 392M 79% /.old devtmpfs 15G 4.0K 15G 1% /dev tmpfs 16G 68K 16G 1% /dev/shm tmpfs 16G 2.2M 16G 1% /run if after removing the files, alarm is not cleared even though Use% is well within the limits, please proceed with the reboot of the corresponding RE.