Description

We might see the CLI logs stops storing the information or stops working, when the eventd process utilisation reaches upto 100%.

Symptoms

When the eventd daemon reached the max limit of 1024 open files, as there is a leak in File Descriptor (FD) when "load replace" is used to update "system syslog source-address", we would be seeing the below syslog messages:

 

Nov 4 03:00:20.147 2024 MX eventd: %SYSLOG-3: /var/log/cli.log: Too many open files

Nov 4 03:30:20.350 2024 MX eventd: %SYSLOG-3: /var/log/link.up.down: Too many open files

Nov 4 04:00:21.168 2024 MX eventd: %SYSLOG-3: /var/log/cli.log: Too many open files

 

This can be verified by logging on to the shell user root and by executing the command:

 

root@labmx # procstat -a -f | grep " eventd .*:514"

 9667 eventd       13 s - rw------ 1   0 UDP 0.0.0.0:514 0.0.0.0:0

 9667 eventd       14 s - rw------ 1   0 UDP 0.0.0.0:514 0.0.0.0:0

 9667 eventd       17 s - rw------ 1   0 UDP 0.0.0.0:514 0.0.0.0:0

 9667 eventd       21 s - rw------ 1   0 UDP 0.0.0.0:514 0.0.0.0:0

 9667 eventd       22 s - rw------ 1   0 UDP 0.0.0.0:514 0.0.0.0:0

 9667 eventd       26 s - rw------ 1   0 UDP 0.0.0.0:514 0.0.0.0:0

 9667 eventd       27 s - rw------ 1   0 UDP 0.0.0.0:514 0.0.0.0:0

 9667 eventd       28 s - rw------ 1   0 UDP 0.0.0.0:514 0.0.0.0:0

This issue might be seen if the following conditions are met:
1. On all Junos platforms
2. Use "load replace" to update "system syslog source-address"

 

Solution

Eventd opens a new UDP socket every time when "load replace" is performed for the "system syslog source-address". 
After reaching the limit of 1024 open files, we will be able to see the error and the CLI might stop logging the information.
Workaround is to not use 'load replace' for updating "system syslog source-address" and if the problem persists restart the eventd process during the MW time.
This issue is explained on the PR1502499.  https://prsearch.juniper.net/problemreport/PR1502499

Modification History

2024-11-20 : Article Created

Related Information

https://prsearch.juniper.net/problemreport/PR1502499