On EX4300 platform in some rare scenario it has been noticed that SNMP servers are not seeing traps correctly, and the logs are filled with below messages.
Oct X 0X:0X:XX xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KBOct X 0X:0X:XX xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x7522f441 PDU-Type: 162
This KB articulates the issue and means to overcome the issue if seen.
Below logs are continuously printing on the device. Oct 9 01:01:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 01:01:58 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x7522f441 PDU-Type: 162 Oct 9 01:01:59 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x69f609dd PDU-Type: 162 Oct 9 01:07:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 01:13:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 01:13:59 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x5d109da4 PDU-Type: 162 Oct 9 01:16:59 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x46a92149 PDU-Type: 162 Oct 9 01:22:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 01:22:59 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0xc787161 PDU-Type: 162 Oct 9 01:28:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 01:34:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 01:37:59 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x7874e9f6 PDU-Type: 162 Oct 9 01:43:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 01:49:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 01:55:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 02:01:58 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 231628 KB Max 262144 KB Oct 9 02:01:58 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x56816fb9 PDU-Type: 162 Oct 9 02:01:59 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x7ee50702 PDU-Type: 162 Oct 9 02:01:59 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x551fe444 PDU-Type: 162 Oct 9 02:13:59 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x28bbd033 PDU-Type: 162 Oct 9 02:24:47 xxxxxxxxxxx rpd[2323]: RPD_RT_TIME_AVRGD_WM_NOTICE: New time averaged watermark for Unique destination routes 11 [Old: 11] Oct 9 02:24:47 xxxxxxxxxxx rpd[2323]: RPD_RT_TIME_AVRGD_WM_NOTICE: New time averaged watermark for RIB routes 14 [Old: 14] Oct 9 02:24:47 xxxxxxxxxxx rpd[2323]: RPD_RT_TIME_AVRGD_WM_NOTICE: New time averaged watermark for FIB routes 2 [Old: 2] Oct 9 02:28:58 xxxxxxxxxxx snmpd[19532]: LIBJSNMP_NS_LOG_WARNING: WARNING: _sess_async_send@5436: TOO-LONG pdu->command: 162 orig_pdu_cmd:160 ReqID:0x50d8dff3 PDU-Type: 162 Oct 9 02:31:59 xxxxxxxxxxx /kernel: Process (19532,snmpd) has exceeded 85% of RLIMIT_DATA: used 232060 KB Max 262144 KB
This issue is result of a Memory leak happening in snmpd in the presence of SNMPv3 get-bulk queries.
Workaround:
Work-around is to use snmp v3 get-bulk request's 'Max-Repetitions' value as 10 or restart of snmpd process.
As per SNMP Best practices document, recommended value for ?Max-repetitions? is 10. (https://www.juniper.net/documentation/en_US/release-independent/nce/information-products/pathway-pages/nce/nce-snmp-best-practices.pdf), If user is sending SNMP v3 GET-BULK requests with ?Max-repetitions? set to 10, the code path that causes memory leak will not be traversed. On all Junos and Junos Evolved platforms, a memory leak is seen in the snmpd (SNMP daemon) during SNMP v3 get-bulk queries. When response PDUs exceed the incoming v3 msgMaxSize value, varbinds are truncated to fit, but subsequent cloning of structures leads to memory leaks due to overwritten pointers without proper freeing. If there is a continuous memory leak, this could result in process memory exhaustion and core.This could impact the system monitoring using SNMP. There will not be any other service impact, traffic loss or network outage due to this. This leak would be visible by watching the output of below command: user@device>show system processes extensive | match snmpd | refresh 30Also you can check the daemon usage from the below outputs:
user@m320-re0> show system processes extensivelast pid: 2814; load averages: 0.00, 0.00, 0.00 up 0+05:05:48 15:08:12108 processes: 3 running, 86 sleeping, 1 zombie, 18 waiting
Mem: 183M Active, 51M Inact, 144M Wired, 206M Cache, 69M Buf, 1413M FreeSwap: 2048M Total, 2048M Free
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND<snipping ...>1825 root 1 96 0 11812K 7728K select 0:01 0.00% mib2d1824 root 1 96 0 12576K 9368K select 0:01 0.00% snmpd<snipping ...>
user@m320-re0> file show /proc/1825/rlimitcpu -1 -1fsize -1 -1data 524288000 524288000 <-- 512M around reserved and 512M availstack 8388608 67108864core -1 -1rss 2092818432 -1memlock 697606144 2092818432nproc 532 532nofile 10500 10500sbsize -1 -1vmem -1 -1Not Available -1 -1
user@m320-re0> file show /proc/1824/rlimitcpu -1 -1fsize -1 -1data 268435456 2147483648 <-- Snmpd shared entire 2G memory, but limit in 256M aroundstack 8388608 67108864core -1 -1rss 2092818432 -1memlock 697606144 2092818432nproc 532 532nofile 10500 10500sbsize -1 -1vmem -1 -1Not Available -1 -1
Resolved-In
evo:21.2R3-S8-EVO evo:21.4R3-S7-EVO evo:22.2R3-S4-EVO evo:22.3R3-S3-EVO evo:23.2R2-EVO evo:23.4R2-EVO evo:24.1R1-EVO evo:24.2R1-EVO junos:21.2R3-S8 junos:21.4R3-S7 junos:22.1R3-S6 junos:22.2R3-S4 junos:22.3R3-S3 junos:22.4R3-S2 junos:23.2R2 junos:23.2R2-J14 junos:23.4R2 junos:24.2R1