Description

This article explains the reason behind the memory leak observed under the RPD daemon and steps to help us find the root cause.

 

37073 root          20   0 26702M 23791M kqread 3 65:05  0.00% rpd{rpd}

 

14 hours after

 

37073 root          20   0 36106M 32459M kqread 2 88:08  0.10% rpd{rpd}

Symptoms

Following logs could help us identify the issue:

 

Check show system process extensive to check RPD memory size on regular basis:

 

user@device> show system processes extensive | no-more

 

last pid: 41459; load averages: 0.22, 0.27, 0.30 up 23+14:45:50   21:03:07

 

319 processes: 5 running, 271 sleeping, 43 waiting

 

Mem: 1535M Active, 31G Inact, 1693M Wired, 749M Buf, 13G Free

 

Swap: 3072M Total, 314M Used, 2758M Free, 10% Inuse

 

 PID USERNAME     PRI NICE  SIZE   RES STATE  C  TIME   WCPU COMMAND

 

<Snipped>

 

37073 root          20   0 36106M 32459M kqread 2 88:08  0.10% rpd{rpd}

 

Solution

When mpls traffic-engineering database import is in place, or any similar condition such as ISIS flood prefixes into RIB lsdist.0 table, there will be memory leak. Using command '> show task memory detail | match so_mem | refresh 10' and check the usage of 'so_mem' on regular basis to identify this issue, and perform rpd live coredump for analysis. The fix is to free the socket address.
 

Check the usage of ‘so_mem’ on regular basis to identify the issue

 

user@device>show task memory detail | match so_mem | refresh 10 

 so_mem                 47971800 10744647880 47976642 10744802824 124740020

---(refreshed at 2023-10-12 01:51:37 EDT)---

 so_mem                 47971800 10744647880 47976642 10744802824 124741258

---(refreshed at 2023-10-12 01:51:47 EDT)---

 so_mem                 47972923 10744899432 47976642 10744899456 124743399

---(refreshed at 2023-10-12 01:51:57 EDT)---

 so_mem                 47972923 10744899432 47976642 10744899480 124744487

---(refreshed at 2023-10-12 01:52:07 EDT)---

 

WORKAROUND

avoid mpls traffic-engineering database knob if possible


To resolve the issue, move to a fixed version. Please refer PR1404829 for more details

Modification History

2023-10-13