A device used for BGP flowspec controller experienced protocol rpd crash and restart due to the out of swap memory space. It has been confirmed that there's a memory leak when modifying flow route information. Fix was made and will be released.
When modifying flow route configuration and committing the change, memory leak is observed as follows.
labroot@jtac-ex3400> show system processes extensive | match rpd | refresh 180 ---(refreshed at 2023-06-15 09:30:48 IST)--- 17633 root 20 0 446M 108M kqread 1 0:22 0.00% rpd{rpd} 17633 root 20 0 446M 108M kqread 1 0:02 0.00% rpd{TraceThread} 17633 root 20 0 446M 108M kqread 1 0:00 0.00% rpd{bgpio-0-th} 17633 root 20 0 446M 108M kqread 0 0:00 0.00% rpd{krtio-th} ---(refreshed at 2023-06-15 09:33:48 IST)--- 17633 root 20 0 450M 111M kqread 0 0:23 0.00% rpd{rpd} 17633 root 20 0 450M 111M kqread 0 0:02 0.00% rpd{TraceThread} 17633 root 20 0 450M 111M kqread 1 0:00 0.00% rpd{bgpio-0-th} 17633 root 20 0 450M 111M kqread 0 0:00 0.00% rpd{krtio-th} ---snip--- ---(refreshed at 2023-06-15 10:12:48 IST)--- 17633 root 20 0 474M 135M kqread 1 0:34 0.00% rpd{rpd} 17633 root 20 0 474M 135M kqread 1 0:08 0.00% rpd{TraceThread} 17633 root 20 0 474M 135M kqread 1 0:01 0.00% rpd{bgpio-0-th} 17633 root 20 0 474M 135M kqread 0 0:00 0.00% rpd{krtio-th} ---(refreshed at 2023-06-15 10:15:48 IST)--- 17633 root 20 0 478M 138M kqread 0 0:35 0.00% rpd{rpd} 17633 root 20 0 478M 138M kqread 1 0:08 0.00% rpd{TraceThread} 17633 root 20 0 478M 138M kqread 0 0:01 0.00% rpd{bgpio-0-th} 17633 root 20 0 478M 138M kqread 1 0:00 0.00% rpd{krtio-th}
labroot@jtac-ex3400> show chassis routing-engine | refresh 180 ---(refreshed at 2023-06-15 09:31:05 IST)--- Routing Engine status: Slot 0: Current state Master Temperature 49 degrees C / 120 degrees F CPU temperature 49 degrees C / 120 degrees F DRAM 1940 MB (2048 MB installed) Memory utilization 38 percent ---snip--- ---(refreshed at 2023-06-15 10:13:05 IST)--- Routing Engine status: Slot 0: Current state Master Temperature 49 degrees C / 120 degrees F CPU temperature 49 degrees C / 120 degrees F DRAM 1940 MB (2048 MB installed) Memory utilization 41 percent
labroot@jtac-ex3400> show task memory | refresh 180 ---(refreshed at 2023-06-15 09:31:24 IST)--- Memory Size (kB) Percentage When Currently In Use: 33768 6% now Maximum Ever Used: 34824 6% 23/06/15 09:19:03 Available: 524288 100% now ---snip--- ---(refreshed at 2023-06-15 10:13:24 IST)--- Memory Size (kB) Percentage When Currently In Use: 63056 12% now Maximum Ever Used: 63056 12% 23/06/15 10:13:24 Available: 524288 100% now
Also, a large increase of 'RT_NEXTHOPS_TEMPLATE' from the 'show task memory detail' command can be observed.
labroot@jtac-ex3400> show task memory detail ------------------------ Allocator Memory Report ------------------------ Name Size Alloc DTXP Alloc Alloc MaxAlloc MaxAlloc Size Blocks Bytes Blocks Bytes ---snip--- RT_NEXTHOPS_TEMPLATE 376 384 61825 23740800 61825 23740800 history_db_entry_t 32 36 196 7056 196 7056 noblock_buffer_blk 4092 4096 12 49152 16 65536 ---snip---
The following logs will appear, eventually leading to the rpd crash and restart.
Jun 14 09:04:07 jtac-ex3400-48p-r2030 rpd[44053]: JTASK_OS_MEMHIGH: Using 620675 KB of memory, 115 percent of available Jun 14 09:05:07 jtac-ex3400-48p-r2030 rpd[44053]: JTASK_OS_MEMHIGH: Using 620691 KB of memory, 115 percent of available Jun 14 09:06:07 jtac-ex3400-48p-r2030 rpd[44053]: JTASK_OS_MEMHIGH: Using 620700 KB of memory, 115 percent of available Jun 14 09:15:56 jtac-ex3400-48p-r2030 kernel: real memory = 2147483648 (2048 MB) Jun 14 09:15:56 jtac-ex3400-48p-r2030 kernel: avail memory = 1983660032 (1891 MB) Jun 14 09:15:56 jtac-ex3400-48p-r2030 kernel: ihost_armpll0: <Broadcom iHost ARM PLL> mem 0-0xfff on simpleb ---snip--- Jun 14 15:37:04 jtac-ex3400-48p-r2030 kernel: pid 30145 (rpd), jid 0, uid 0, was killed: out of swap space Jun 14 15:37:05 jtac-ex3400-48p-r2030 jlaunchd[15704]: routing (PID 30145) terminated by signal number 9! Jun 14 15:37:05 jtac-ex3400-48p-r2030 jlaunchd[15704]: Registered PID 44053(routing): exec_command Jun 14 15:37:05 jtac-ex3400-48p-r2030 jlaunchd[15704]: routing (PID 44053) started Jun 14 15:37:05 jtac-ex3400-48p-r2030 jlaunchd[15704]: Registered PID 44053(routing): new process