The customer reported that they are observing frequently cRPD container crashes in their network. The cRPD RR has been allocated 16G of memory. It was enough until the recent past and then we started seeing the containers getting killed due to them reaching the memory limit. Some of the memory increase is attributed to a recent external peering turn which adds additional prefixes to the network. However, at other sites, we see less memory has been used though the route count is not drastically different from the site that exhibits high memory usage.

The cRPD container will frequently crash with an Out-Of-Memory (OOM) signature

Nov 3 15:33:54 crpd : <13>1 2024-11-03T15:33:54.343Z crpd rpd - RPD_RT_HWM_NOTICE [junos@- message="New RIB highwatermark for routes:" maximum-value="11688537" time="2024-11-03 15:33:54"] New RIB highwatermark for routes: 11688537 [2024-11-03 15:33:54]

Nov 3 15:33:54 rpd : <13>1 2024-11-03T15:33:54.343Z crpd rpd - RPD_RT_HWM_NOTICE [junos@- message="New RIB highwatermark for routes:" maximum-value="11688537" time="2024-11-03 15:33:54"] New RIB highwatermark for routes: 11688537 [2024-11-03 15:33:54]

Nov 3 22:53:58 crpdk ernel: rpd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0

Nov 3 22:53:58 crpd kernel: Call Trace:

Nov 3 22:53:58 crpd kernel: dump_stack+0x50/0x6b

Nov 3 22:53:58 crpd kernel: dump_header+0x4a/0x200

Nov 3 22:53:58 crpd kernel: oom_kill_process+0xe8/0x130

Nov 3 22:53:58 crpd kernel: out_of_memory+0x113/0x560

Nov 3 22:53:58 crpd kernel: mem_cgroup_out_of_memory+0xb5/0xd0

Nov 3 22:53:58 crpd kernel: try_charge+0x810/0x850

Nov 3 22:53:58 crpd kernel: ? __alloc_pages_nodemask+0x154/0x320

Nov 3 22:53:58 crpd kernel: mem_cgroup_try_charge+0x70/0x190

Nov 3 22:53:58 crpd kernel: mem_cgroup_try_charge_delay+0x1c/0x40

Nov 3 22:53:58 crpd kernel: __handle_mm_fault+0x8e0/0x1270

Nov 3 22:53:58 crpd kernel: handle_mm_fault+0xb0/0x1e0

Nov 3 22:53:58 crpd kernel: __do_page_fault+0x278/0x4a0

Nov 3 22:53:58 crpd kernel: page_fault+0x2f/0x40

Nov 3 22:53:59 crpd kernel: Code: d1 0f 82 57 3a fa ff 0f 1f 80 00 00 00 00 c5 f9 6e c6 48 89 f8 c4 e2 7d 78 c0 48 83 fa 20 0f 82 a4 00 00 00 48 83 fa 40 77 0e <c5> fe 7f 44 17 e0 c5 fe 7f 07 c5 f8 77 c3 48 81 fa 00 08 00 00 77

Nov 3 22:53:59 crpd kernel: RSP: 002b:00007ffd8e30f458 EFLAGS: 00010246

Nov 3 22:53:59 crpd kernel: RAX: 000055ed433d5000 RBX: 0000000000000000 RCX: 0000000000000118

Nov 3 22:53:59 crpd kernel: RDX: 0000000000000040 RSI: 0000000000000000 RDI: 000055ed433d5000

Nov 3 22:53:59 crpd kernel: RBP: 00007ffd8e30f4a0 R08: 000055e950574280 R09: 000000000000002a

Nov 3 22:53:59 crpd kernel: R10: 0000000000000004 R11: 000055ecc9d32420 R12: 0000000000000000

Nov 3 22:53:59 crpd kernel: R13: 000055ed433d5000 R14: 000055e9525ee4b0 R15: 0000000000000000

Nov 3 22:53:59 crpd kernel: memory: usage 16777216kB, limit 16777216kB, failcnt 0

Nov 3 22:53:59 crpd kernel: memory+swap: usage 16777216kB, limit 16777216kB, failcnt 101298

Nov 3 22:53:59 crpd kernel: kmem: usage 99576kB, limit 9007199254740988kB, failcnt 0

it was determined that the high rate of route churn on the device, which followed the addition of the new peer, was a major contributing factor to the Out-Of-Memory (OOM) issue. Upon further investigation, we found that the customer had allocated 16GB of memory for the cRPD container. However, according to the documentation, the recommended memory allocation for this scenario should be over 24GB, as the RIB/FIB route scale exceeded 11.6 million entries. The customer has agreed to increase the memory allocation as suggested.