We might see the CLI logs stops storing the information or stops working, when the eventd process utilisation reaches upto 100%.
When the eventd daemon reached the max limit of 1024 open files, as there is a leak in File Descriptor (FD) when "load replace" is used to update "system syslog source-address", we would be seeing the below syslog messages:
Nov 4 03:00:20.147 2024 MX eventd: %SYSLOG-3: /var/log/cli.log: Too many open files
Nov 4 03:30:20.350 2024 MX eventd: %SYSLOG-3: /var/log/link.up.down: Too many open files
Nov 4 04:00:21.168 2024 MX eventd: %SYSLOG-3: /var/log/cli.log: Too many open files
This can be verified by logging on to the shell user root and by executing the command:
root@labmx # procstat -a -f | grep " eventd .*:514"
9667 eventd 13 s - rw------ 1 0 UDP 0.0.0.0:514 0.0.0.0:0
9667 eventd 14 s - rw------ 1 0 UDP 0.0.0.0:514 0.0.0.0:0
9667 eventd 17 s - rw------ 1 0 UDP 0.0.0.0:514 0.0.0.0:0
9667 eventd 21 s - rw------ 1 0 UDP 0.0.0.0:514 0.0.0.0:0
9667 eventd 22 s - rw------ 1 0 UDP 0.0.0.0:514 0.0.0.0:0
9667 eventd 26 s - rw------ 1 0 UDP 0.0.0.0:514 0.0.0.0:0
9667 eventd 27 s - rw------ 1 0 UDP 0.0.0.0:514 0.0.0.0:0
9667 eventd 28 s - rw------ 1 0 UDP 0.0.0.0:514 0.0.0.0:0
This issue might be seen if the following conditions are met: 1. On all Junos platforms 2. Use "load replace" to update "system syslog source-address"
Eventd opens a new UDP socket every time when "load replace" is performed for the "system syslog source-address". After reaching the limit of 1024 open files, we will be able to see the error and the CLI might stop logging the information. Workaround is to not use 'load replace' for updating "system syslog source-address" and if the problem persists restart the eventd process during the MW time. This issue is explained on the PR1502499. https://prsearch.juniper.net/problemreport/PR1502499
https://prsearch.juniper.net/problemreport/PR1502499