Description

On the AS7816 device facing RPD crash due memory related issues, the following logs can be seen before the crash.

 

 

Jul 4 20:06:16 lab-as7816 kernel: swap_pager_getswapspace(10): failed

Jul 4 20:06:28 lab-as7816 jlaunchd[9209]: routing (PID 9740) terminated by signal number 6. Core dumped!

Jul 4 20:06:28 lab-as7816 jlaunchd[9209]: routing (PID 32324) started

Symptoms

High memory usage on the Juniper device.

 

root@lab-as7816> show system processes extensive no-forwarding 

 

last pid: 46066; load averages: 0.77, 0.75, 0.78 up 92+21:39:38  16:13:09 

371 threads:  3 running, 337 sleeping, 31 waiting 

CPU: 19.0% user, 0.0% nice, 9.8% system, 3.9% interrupt, 67.2% idle 

Mem: 2665M Active, 168M Inact, 203M Laundry, 671M Wired, 253M Buf, 81M Free 

Swap: 3072M Total, 1846M Used, 1226M Free, 60% Inuse 

 

 PID USERNAME  PRI NICE  SIZE  RES STATE  C  TIME  WCPU COMMAND 

  11 root    155 ki31   0B  32K CPU1   1 1517.2 59.77% idle{idle: cpu1} 

  11 root    155 ki31   0B  32K RUN   0 1511.1 50.88% idle{idle: cpu0} 

32324 root     30  0 3835M 2675M kqread  0 232:39 17.87% rpd{rpd} 

 

root@lab-as7816> show route summary 

 

Autonomous system number: 20940 

Router ID: 173.222.148.254 

 

Highwater Mark (All time / Time averaged watermark) 

  RIB unique destination routes: 1169225 at 2024-07-05 15:41:53 / 0 

  RIB routes          : 4144350 at 2024-07-05 09:50:07 / 0 

  FIB routes          : 109148 at 2024-07-05 09:50:38 / 0 

  VRF type routing instances  : 0 at 2024-07-04 20:06:32 

 

Route protocol flaps/instability of the routing protocol neighbors/issues with routing table entries and packet forwarding.

Solution

The issue is due to the increased route scale on the device. Check the following.

 

  1. Swap memory usage - Swap memory is often the victim and not the cause of the problem. Try to isolate the reason behind the usage of swap memory. In the above customer scenario, the customer was using an increased route scale which triggered the swap memory usage.
  2. Supported scaling on the device is:

 

  • RIB IPv4: 2M
  • RIB IPV6: 500K
  • FIB IPV4: 262K
  • FIB IPV6: 172K
These numbers have been tested on QFX5210, which is equivalent of AS7816 (both platforms use same chipset). 

Modification History

2024-07-08 - Initial Publication
2024-07-29 - Visibility Modified