Description

High rpd and jdhcpd CPU observed on MX10003 router and high swap memory is being used.

Symptoms

lab> show system processes extensive no-forwarding

 

last pid: 54561; load averages: 2.47, 1.87, 1.31 up 145+21:38:28   06:33:55

480 threads:  8 running, 424 sleeping, 2 zombie, 46 waiting

CPU: 3.1% user, 0.0% nice, 2.0% system, 0.3% interrupt, 94.5% idle

Mem: 6416M Active, 707M Inact, 34G Laundry, 2057M Wired, 756M Buf, 3851M Free

Swap: 3072M Total, 2990M Used, 82M Free, 97% Inuse

 

PID USERNAME   PRI NICE  SIZE   RES STATE   C  TIME   WCPU COMMAND

51003 root       103   0 6412M 5441M CPU2    2 12:11 75.00% rpd{rpd}

11637 root        23   0   10G 8504M kqread  1 481:16  15.96% rpd{rpd}


Oct 8 22:23:14 lab kernel: pid 12005 (jdhcpd), jid 0, uid 0, was killed: out of swap space

Oct 8 22:23:14 lab kernel: swap_pager_getswapspace(32): failed

Oct 8 22:23:25 lab rpdtmd[75090]: JTASK_SCHED_SLIP_KEVENT: 16 sec 437808 usec kevent block

Oct 8 22:23:25 lab overlayd[75086]: JTASK_SCHED_SLIP_KEVENT: 16 sec 384459 usec kevent block

Oct 8 22:23:25 lab mplsoamd[75064]: JTASK_SCHED_SLIP_KEVENT: 15 sec 338405 usec kevent block

Oct 8 22:23:25 lab dot1xd[75124]: JTASK_SCHED_SLIP_KEVENT: 15 sec 83283 usec kevent block

Oct 8 22:23:25 lab jlaunchd[11594]: dhcp-service (PID 12005) terminated by signal number 9!

Solution

The dhcp process is leaking memory and hence causing the SWAP memory to be used.


To recover from this state please follow the below procedure:

  1. Disable the GRES, NSR and commit synchronization between the REs.
  2. reboot the back RE.
  3. restart the dhcp process using the command "restart dhcp-service"
  4. Wait for 5 mins to settle things and then re-enable the GRES, NSR and commit synchronization between the REs.


Modification History

2024-10-17 : Article Created