Memory exhaustion observed for process ppmd-agent running on the backup Routing Engine and leading to process restart
Platform: PTX10K16
Release: 23.4R2-S3-EVO
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: ppmd-agent invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: CPU: 11 PID: 15077 Comm: ppmd-agent Kdump: loaded Tainted: G OE 5.2.60-yocto-standard-g25b2b83 #1
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: Hardware name: Juniper Networks Inc. 0CA3/0CA3, BIOS CBEP_P_VAL1_00.18.02 08/26/2020
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: Call Trace:
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: dump_stack+0x59/0x7a
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: dump_header+0x57/0x320
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: ? preempt_count_add+0x85/0xd0
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: ? _raw_spin_unlock_irqrestore+0x18/0x30
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: oom_kill_process.cold+0xb/0x10
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: out_of_memory+0x1c7/0x470
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: mem_cgroup_out_of_memory+0xbb/0xd0
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: try_charge+0x7be/0x830
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: ? preempt_count_add+0x5e/0xd0
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: mem_cgroup_try_charge+0xac/0x230
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: mem_cgroup_try_charge_delay+0x22/0x50
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: __handle_mm_fault+0xde8/0x1400
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: ? sched_clock_cpu+0x1a/0xe0
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: handle_mm_fault+0xc3/0x200
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: __do_page_fault+0x299/0x4a0
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: ? __audit_syscall_exit+0x249/0x2b0
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: do_page_fault+0x31/0x110
Nov 4 19:51:36.752 2024 labrouter-RE0-re1 kernel: page_fault+0x2f/0x40
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: RIP: 0033:0x7ff1bfb98d7e
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: Code: 41 55 41 54 53 48 83 ec 08 48 85 f6 0f 84 80 00 00 00 49 89 fd bf 28 00 00 00 48 89 f3 e8 4a ff ff ff bf 18 00 00 00 49 89 c4 <48> c7 40 08 00 00 00 00 48 c7 40 10 00 00 00 00 48 8d 05 83 16 01
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: RSP: 002b:00007ffce1cd8960 EFLAGS: 00010202
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: RAX: 00007fefa50a6a80 RBX: 0000000000010000 RCX: 00007fefa5004970
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: RDX: 00007fefa5000b20 RSI: 00007fefa5001000 RDI: 0000000000000018
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: RBP: 00007ffce1cd8980 R08: 0000000000000000 R09: 00000000000000ad
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: R10: 0000000000000000 R11: 0000000000000006 R12: 00007fefa50a6a80
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: R13: 00007ffce1cd89c0 R14: 00007fefa4fe3ca0 R15: 0000000000000032
Nov 4 19:51:36.753 2024 labrouter-RE0-re1 kernel: Memory cgroup out of memory: Killed process 15077 (ppmd-agent) total-vm:8968208kB, anon-rss:1031148kB, file-rss:133592kB, shmem-rss:130084kB
Nov 4 19:51:37.116 2024 labrouter-RE0-re1 systemd[1]: ppmdagent.service: Main process exited, code=killed, status=9/KILL
Nov 4 19:51:37.238 2024 labrouter-RE0-re1 systemd[1]: ppmdagent.service: Failed with result 'signal'.
Nov 4 19:51:37.246 2024 labrouter-RE0-re1 systemd[1]: ppmd.service: Succeeded.
Nov 4 19:51:37.362 2024 labrouter-RE0-re1 systemd[1]: ppmdagent.service: Service RestartSec=100ms expired, scheduling restart.
Nov 4 19:51:37.362 2024 labrouter-RE0-re1 systemd[1]: ppmdagent.service: Scheduled restart job, restart counter is at 1.
Nov 4 19:51:37.362 2024 labrouter-RE0-re1 systemd[1]: ppmd.service: Service RestartSec=100ms expired, scheduling restart.
Nov 4 19:51:37.362 2024 labrouter-RE0-re1 systemd[1]: ppmd.service: Scheduled restart job, restart counter is at 1.
Nov 4 19:51:37.362 2024 labrouter-RE0-re1 systemd[1]: Stopped "PPMD Daemon on RE".
Nov 4 19:51:37.363 2024 labrouter-RE0-re1 systemd[1]: Started "PPMD Daemon on RE".
Nov 4 19:51:37.121 2024 labrouter-RE0-re0 sysman[12956]: SYSTEM_APP_OFFLINE_EVENT: App went offline re1-ppmdagent
Nov 4 19:51:37.179 2024 labrouter-RE0-re0 sysman[12956]: SYSTEM_APP_OFFLINE_EVENT: App went offline re1-ppmd
Nov 4 19:51:37.369 2024 labrouter-RE0-re0 sysman[12956]: SYSTEM_APP_ONLINE_EVENT: App went online re1-ppmd
Nov 4 19:51:37.364 2024 labrouter-RE0-re1 systemd[1]: Stopped "PPMD agent on RE".
Nov 4 19:51:37.365 2024 labrouter-RE0-re1 systemd[1]: Started "PPMD agent on RE".
Nov 4 19:51:37.368 2024 labrouter-RE0-re1 sysman[13223]: SYSTEM_APP_READY: App is ready re1-ppmd
Nov 4 19:51:37.442 2024 labrouter-RE0-re1 ppmd[4488]: sysctl hw.re.slotid not supported
Nov 4 19:51:38.455 2024 labrouter-RE0-re1 ppmd-agent[4489]: SYSTEM_PSET_STATE_CHANGE: Production set [ppmdagent, re1, 50342029] has changed state to: standby at 19:51:38.455
For certain applications, the system caches state information on the standby instances to facilitate a faster reconciliation on an RE switch over. It was observed that certain stale information was not being cleared from this cache. Due to this, the memory usage on the standby instance of these applications continued to increase until a restart or an RE switch-over occurred. This is a slow leak and builds up over a period.
Only affect backup instances of applications like ifmand, ppmdagent, pfetokend. The issue is applicable from 23.1 onwards for all EVO platforms.
No functional impact given this only affects standby instances. The memory limits configured for these applications will restart the backup instance and reconciles fresh state. GRES/NSR switchovers will also release these memory and reconcile to the correct state of the system.