CPU alarm set on Device
ouput showing SRRD high:
show system processes extensive no-forwarding
last pid: 52350; load averages: 4.05, 3.39, 3.23 up 104+13:31:56 19:21:30
384 processes: 10 running, 329 sleeping, 45 waiting
Mem: 8039M Active, 6309M Inact, 1974M Wired, 680M Buf, 3575M Free
Swap: 3072M Total, 3072M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
30387 root 100 0 2167M 1719M RUN 0 308.5H 97.27% srrd
29669 root 99 0 568M 41968K CPU2 2 496.4H 93.80% chassisd{chassisd}
30377 root 86 0 12307M 11285M CPU3 3 693.1H 48.49% rpd{rpd}
11 root 155 ki31 0K 64K RUN 2 1810.3 25.00% idle{idle: cpu2}
11 root 155 ki31 0K 64K RUN 3 1681.9 19.38% idle{idle: cpu3}
11 root 155 ki31 0K 64K RUN 1 1684.4 17.97% idle{idle: cpu1}
11 root 155 ki31 0K 64K RUN 0 1713.8 16.70% idle{idle: cpu0}
30418 root 26 0 509M 48740K CPU0 0 299.8H 11.57% mib2d
As per the design it is expected that SRRD takes 100% CPU for few minutes during huge route churn.
When a route churn happens and when it is huge, the following might happen
1. The FPC will be busy handling the route churn
2. The RPD sends the route updates to SRRD (that might include DELETE followed by an ADD if the link got up) and SRRD sends them back to FPC for JFlow functionality.
3. The PFE thread that handles updates from SRRD is a LOW PRIORITY thread in PFE. If FPC is busy with processing the route updates, the updates from SRRD are not processed and this will result in pileup of updates in SRRD. There is a threshold for number of updates per client, if the number crosses this threshold, we will close the client. The SRRD daemon during closing the connection, will first clear all the updates it holds for this client, this will keep SRRD in a loop and during this time it will be 100% for some time. Once the cleanup is done, the CPU usage of SRRD daemon becomes normal. When the client (the FPC in this case) notices that the connection is lost, it will retry to connect back to SRRD. This case is handled fine.