Description

Customer reported that they observed some "UKERN_GBL*backup" log files keeps logging but not being overwritten which results /var/log directory full. This issue may occur after Junos upgrade to 21.4Rx or later.

Customer's output:

admin@QFX5120-48YM> file list detail /.mount/var/log

/.mount/var/log:
total blocks: 1440968
drwxrwxr-x 2 root operator 512 Nov 5 2022 .snap/
-rw-r--r-- 1 root wheel 18224 Sep 28 11:43 UKERN_GBL.log
-rw-r--r-- 1 root wheel 294078 Sep 28 11:43 UKERN_GBL.log.1-2023060114.backup
-rw-r--r-- 1 root wheel 300366 Sep 28 11:43 UKERN_GBL.log.1-2023060116.backup
-rw-r--r-- 1 root wheel 278677 Sep 28 11:44 UKERN_GBL.log.1-2023060121.backup
-rw-r--r-- 1 root wheel 2827311 Sep 28 11:44 UKERN_GBL.log.1-2023060123.backup
-rw-r--r-- 1 root wheel 277886 Sep 28 11:41 UKERN_GBL.log.1-2023060202.backup
-rw-r--r-- 1 root wheel 267181 Sep 28 11:41 UKERN_GBL.log.1-2023060204.backup
-rw-r--r-- 1 root wheel 271068 Sep 28 11:42 UKERN_GBL.log.1-2023060206.backup
-rw-r--r-- 1 root wheel 280746 Sep 28 11:42 UKERN_GBL.log.1-2023060207.backup
-rw-r--r-- 1 root wheel 355920 Sep 28 11:42 UKERN_GBL.log.1-2023060209.backup
-rw-r--r-- 1 root wheel 283546 Sep 28 11:42 UKERN_GBL.log.1-2023060213.backup
-rw-r--r-- 1 root wheel 278204 Sep 28 11:43 UKERN_GBL.log.1-2023060215.backup
-rw-r--r-- 1 root wheel 287218 Sep 28 11:43 UKERN_GBL.log.1-2023060216.backup
-rw-r--r-- 1 root wheel 304566 Sep 28 11:43 UKERN_GBL.log.1-2023060321.backup
-rw-r--r-- 1 root wheel 304845 Sep 28 11:43 UKERN_GBL.log.1-2023060323.backup
-rw-r--r-- 1 root wheel 306179 Sep 28 11:44 UKERN_GBL.log.1-2023060400.backup
-rw-r--r-- 1 root wheel 313153 Sep 28 11:44 UKERN_GBL.log.1-2023060405.backup
-rw-r--r-- 1 root wheel 311491 Sep 28 11:41 UKERN_GBL.log.1-2023060413.backup
-rw-r--r-- 1 root wheel 307684 Sep 28 11:41 UKERN_GBL.log.1-2023060418.backup
-rw-r--r-- 1 root wheel 310949 Sep 28 11:42 UKERN_GBL.log.1-2023060419.backup
-rw-r--r-- 1 root wheel 311188 Sep 28 11:42 UKERN_GBL.log.1-2023060421.backup
-rw-r--r-- 1 root wheel 309866 Sep 28 11:42 UKERN_GBL.log.1-2023060500.backup
-rw-r--r-- 1 root wheel 315136 Sep 28 11:42 UKERN_GBL.log.1-2023060508.backup
-rw-r--r-- 1 root wheel 312492 Sep 28 11:43 UKERN_GBL.log.1-2023060511.backup
-rw-r--r-- 1 root wheel 309298 Sep 28 11:43 UKERN_GBL.log.1-2023060519.backup
-rw-r--r-- 1 root wheel 314435 Sep 28 11:43 UKERN_GBL.log.1-2023060522.backup
-rw-r--r-- 1 root wheel 311490 Sep 28 11:43 UKERN_GBL.log.1-2023060606.backup
-rw-r--r-- 1 root wheel 310264 Sep 28 11:44 UKERN_GBL.log.1-2023060614.backup
-rw-r--r-- 1 root wheel 310147 Sep 28 11:44 UKERN_GBL.log.1-2023060617.backup
-rw-r--r-- 1 root wheel 311567 Sep 28 11:41 UKERN_GBL.log.1-2023060701.backup

Symptoms

Tons of "UKERN_GBL*backup" log files are filling up /var/log directory which results no space left to record normal logs.

{master:0}
root@QFX5120-48YM> show version | grep Junos: 
Junos: 22.2R3.15

{master:0}
root@QFX5120-48YM> file list detail /var/log | grep UKERN_GBL   
-rw-r--r--  1 root  wheel          0 Oct 5  15:28 UKERN_GBL.log
-rw-r--r--  1 root  wheel     267267 Aug 30 11:52 UKERN_GBL.log.1-2023082114.backup
-rw-r--r--  1 root  wheel     267564 Aug 30 11:52 UKERN_GBL.log.1-2023082211.backup
-rw-r--r--  1 root  wheel     267660 Aug 30 11:52 UKERN_GBL.log.1-2023082220.backup
-rw-r--r--  1 root  wheel     267236 Aug 30 11:52 UKERN_GBL.log.1-2023082518.backup
-rw-r--r--  1 root  wheel     267024 Aug 30 11:52 UKERN_GBL.log.1-2023082602.backup
-rw-r--r--  1 root  wheel     267130 Aug 30 11:52 UKERN_GBL.log.1-2023082611.backup
-rw-r--r--  1 root  wheel     268190 Aug 30 11:52 UKERN_GBL.log.1-2023082620.backup
-rw-r--r--  1 root  wheel     266716 Aug 30 11:52 UKERN_GBL.log.1-2023082705.backup
-rw-r--r--  1 root  wheel     267872 Aug 30 11:52 UKERN_GBL.log.1-2023082715.backup
-rw-r--r--  1 root  wheel    2532267 Aug 30 11:52 UKERN_GBL.log.1-2023082723.backup
-rw-r--r--  1 root  wheel     265250 Sep 12 13:45 UKERN_GBL.log.1-2023091203.backup
-rw-r--r--  1 root  wheel     264925 Sep 12 13:45 UKERN_GBL.log.1-2023091213.backup
-rw-r--r--  1 root  wheel     264295 Sep 25 06:15 UKERN_GBL.log.1-2023091417.backup
-rw-r--r--  1 root  wheel     266918 Sep 25 06:15 UKERN_GBL.log.1-2023091502.backup
-rw-r--r--  1 root  wheel     263353 Sep 25 06:15 UKERN_GBL.log.1-2023091620.backup
-rw-r--r--  1 root  wheel    2546246 Sep 25 06:15 UKERN_GBL.log.1-2023091803.backup
-rw-r--r--  1 root  wheel     263991 Sep 25 06:15 UKERN_GBL.log.1-2023091913.backup
-rw-r--r--  1 root  wheel     273843 Sep 25 06:15 UKERN_GBL.log.1-2023092008.backup
-rw-r--r--  1 root  wheel     267776 Sep 25 06:15 UKERN_GBL.log.1-2023092413.backup
-rw-r--r--  1 root  wheel    2520846 Sep 25 06:15 UKERN_GBL.log.1-2023092503.backup
-rw-r--r--  1 root  wheel     267918 Sep 25 06:15 UKERN_GBL.log.1-2023092506.backup
-rw-r--r--  1 root  wheel     266633 Oct 5  15:28 UKERN_GBL.log.1-2023092508.backup
-rw-r--r--  1 root  wheel     264337 Oct 5  15:28 UKERN_GBL.log.1-2023092514.backup
-rw-r--r--  1 root  wheel    2661030 Oct 5  15:28 UKERN_GBL.log.1-2023092520.backup
-rw-r--r--  1 root  wheel     267660 Oct 5  15:28 UKERN_GBL.log.1-2023092523.backup
-rw-r--r--  1 root  wheel     283104 Oct 5  15:28 UKERN_GBL.log.1-2023092615.backup
-rw-r--r--  1 root  wheel     263669 Oct 5  15:28 UKERN_GBL.log.1-2023092717.backup
-rw-r--r--  1 root  wheel     262160 Oct 5  15:28 UKERN_GBL.log.1-2023100504.backup
-rw-r--r--  1 root  wheel      20084 Oct 5  15:28 UKERN_GBL.log.1.gz
-rw-r--r--  1 root  wheel      20701 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082211.backup
-rw-r--r--  1 root  wheel      20563 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082220.backup
-rw-r--r--  1 root  wheel      20564 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082518.backup
-rw-r--r--  1 root  wheel         20 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082602.backup
-rw-r--r--  1 root  wheel      20531 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082611.backup
-rw-r--r--  1 root  wheel      20564 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082620.backup
-rw-r--r--  1 root  wheel      20677 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082705.backup
-rw-r--r--  1 root  wheel      20588 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082715.backup
-rw-r--r--  1 root  wheel     191802 Aug 30 11:52 UKERN_GBL.log.1.gz-2023082723.backup
-rw-r--r--  1 root  wheel      20601 Sep 12 13:45 UKERN_GBL.log.1.gz-2023091203.backup
-rw-r--r--  1 root  wheel      20499 Sep 12 13:45 UKERN_GBL.log.1.gz-2023091213.backup
-rw-r--r--  1 root  wheel      20510 Sep 25 06:15 UKERN_GBL.log.1.gz-2023091417.backup
-rw-r--r--  1 root  wheel      20468 Sep 25 06:15 UKERN_GBL.log.1.gz-2023091502.backup
-rw-r--r--  1 root  wheel      22435 Sep 25 06:15 UKERN_GBL.log.1.gz-2023091620.backup
-rw-r--r--  1 root  wheel     211193 Sep 25 06:15 UKERN_GBL.log.1.gz-2023091803.backup
-rw-r--r--  1 root  wheel      22650 Sep 25 06:15 UKERN_GBL.log.1.gz-2023092008.backup
-rw-r--r--  1 root  wheel     209649 Sep 25 06:15 UKERN_GBL.log.1.gz-2023092503.backup
-rw-r--r--  1 root  wheel      22920 Sep 25 06:15 UKERN_GBL.log.1.gz-2023092506.backup
-rw-r--r--  1 root  wheel      22543 Oct 5  15:28 UKERN_GBL.log.1.gz-2023092508.backup
-rw-r--r--  1 root  wheel      22947 Oct 5  15:28 UKERN_GBL.log.1.gz-2023092514.backup
-rw-r--r--  1 root  wheel     224886 Oct 5  15:28 UKERN_GBL.log.1.gz-2023092520.backup
-rw-r--r--  1 root  wheel      23764 Oct 5  15:28 UKERN_GBL.log.1.gz-2023092523.backup
-rw-r--r--  1 root  wheel      24235 Oct 5  15:28 UKERN_GBL.log.1.gz-2023092615.backup
-rw-r--r--  1 root  wheel      20040 Oct 5  15:28 UKERN_GBL.log.1.gz-2023100504.backup
-rw-r--r--  1 root  wheel      19997 Oct 5  15:28 UKERN_GBL.log.10.gz
-rw-r--r--  1 root  wheel      20039 Oct 5  15:28 UKERN_GBL.log.2.gz
-rw-r--r--  1 root  wheel      20195 Oct 5  15:28 UKERN_GBL.log.3.gz
-rw-r--r--  1 root  wheel      20042 Oct 5  15:28 UKERN_GBL.log.4.gz
-rw-r--r--  1 root  wheel      20083 Oct 5  15:28 UKERN_GBL.log.5.gz
-rw-r--r--  1 root  wheel      20542 Oct 5  15:28 UKERN_GBL.log.6.gz
-rw-r--r--  1 root  wheel     192685 Oct 5  15:28 UKERN_GBL.log.7.gz
-rw-r--r--  1 root  wheel      20171 Oct 5  15:28 UKERN_GBL.log.8.gz
-rw-r--r--  1 root  wheel      20056 Oct 5  15:28 UKERN_GBL.log.9.gz

Solution

Detect and remove these backup logs via a shell script as a workaround. These logs will reappear on each reboot so the script needs to be manually activated at each reboot.

Issue is resolved in the below releases:
junos:20.4R3-S9 junos:21.2R3-S7 junos:21.3R3-S5 junos:21.4R3-S5 junos:22.1R3-S4 junos:22.2R3-S3 junos:22.4R3 junos:23.2R2 junos:23.3R2 junos:23.4R1

Related KB - KB71045 [juniper.net]

Modification History

2023-10-26 : Article Created
2023-11-10 : Modified the solution contents
2024-07-16 : Added shell script example

2024-11-19: Added UKERN_GBL workaround example