Description

The FPC CPU might go high during BGP convergence. The route convergence will take place and it will take FPC CPU to install the routes in FIB.

Symptoms

The FPC CPU might go high during route convergence and may remain high until the routes gets converged fully even after router convergence completed by RPD and PFE, SRRD may take time to converge the routes.

The SRRD takes FPC CPU until the routes fully converged by the daemon. The FPC CPU will go high for 5-10 minutes depending on the type of routes, the scale of routes, and the complexity of the next hop types.

 

Below is the test conducted in which total 1.07M IPv4 and 800K routes were installed repeatedly by clearing the bgp neighborship. It is observed that even after routes were installed in the RIB and FIB, a significant CPU utilization of FPC observed.

 

Router> show bgp summary   

Apr 02 22:59:50

Groups: 1 Peers: 1 Down peers: 0

Table     Tot Paths Act Paths Suppressed  History Damp State  Pending

inet.0        

         358311   358311     0     0     0   115995

inet6.0        

            0     0     0     0     0     0

Peer           AS   InPkt   OutPkt  OutQ  Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...

10.61.63.21       500   56311     5    0    6     59 Establ

 inet.0: 358311/358311/358311/0

 inet6.0: 0/0/0/0

  

 Router> show bgp summary   

Apr 02 23:01:33

Groups: 1 Peers: 1 Down peers: 0

Table     Tot Paths Act Paths Suppressed  History Damp State  Pending

inet.0        

         1070422  1070422     0     0     0     0

inet6.0        

         800001   800001     0     0     0   252917

Peer           AS   InPkt   OutPkt  OutQ  Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...

10.61.63.21       500   60639     8    0    6    2:42 Establ

 inet.0: 1070422/1070422/1070422/0

 inet6.0: 800001/800001/800001/0

 

Router> show chassis fpc   

Apr 02 23:01:35

           Temp CPU Utilization (%)  CPU Utilization (%) Memory  Utilization (%)

Slot State      (C) Total Interrupt   1min  5min  15min DRAM (MB) Heap   Buffer

 0 Empty      

 1 Online      42  100     5    98   72   54  2048    39     22

 2 Online      44  100     4    98   66   44  2048    37     22

 

Router> show chassis fpc   

Apr 02 23:03:48

           Temp CPU Utilization (%)  CPU Utilization (%) Memory  Utilization (%)

Slot State      (C) Total Interrupt   1min  5min  15min DRAM (MB) Heap   Buffer

 0 Empty      

 1 Online      42  100     6    99   83   60  2048    44     22

 2 Online      45  100     1    99   78   51  2048    46     22

 

Router> start shell command "cprod -A fpc2 -c \"show route all summary\"" | no-more 

Apr 02 23:04:12

 

 

Protocol Route Tables:

Protocol          Tables  Routes

-------------      -------- --------

IPv4              7  1070493---- 1.07M IPv4 Routes.

MPLS              2     8

IPv6              3  800038 -----800KIPv6 Routes

CLNP              2     3

DHCP-Snooping          1     1

 

Router> show chassis fpc                                

Apr 02 23:04:15

           Temp CPU Utilization (%)  CPU Utilization (%) Memory  Utilization (%)

Slot State      (C) Total Interrupt   1min  5min  15min DRAM (MB) Heap   Buffer

 0 Empty      

 1 Online      42  100     6    99   84   62  2048    46     22 ------------ FPC CPU remained high even after routes converged in RIB.

 2 Online      45  100     1    99   80   53  2048    49     22

 

Router> start shell command "cprod -A fpc2 -c \"show sample-rr summary\"" | no-more         

Apr 02 23:06:23

  IPv4 Routes   : 1070438

  IPv6 Routes   : 753644 ----- FPC CPU observed to be high as routes are still getting converged in Sample RR

 

 

 

 

 

 

Solution

- This is an expected behaviour that is when one does the "clear bgp neighbor". 

- During the route convergence process, we noticed that the FPC utilization remained high.

- Although the forwarding table had converged with all the IPv4 and IPv6 routes, the FPC CPU still remained high.

- Upon further investigation, we discovered that SRRD was still in the process of converging the routes.

-SRRD is an abreviation for Sampling Route Record Daemon, which is needed for jflow to work.

- Once the routes were converged in SRRD, the FPC utilization returned to normal. 

- SRRD, or Sampling Route Record Daemon, uses a low-priority thread for convergence. Once the route convergence is completed by RPD and PFE, SRRD takes the FPC CPU cycles to converge.

- Therefore, the high FPC CPU usage was observed until SRRD was converged.

- This is non-impacting and once a high-priority job pitches in, it will release the CPU cycles it holds.

 

 

Router> start shell command "cprod -A fpc2 -c \"show sample-rr summary\"" | no-more         

Apr 02 23:06:23

  IPv4 Routes   : 1070438

  IPv6 Routes   : 753644

Router> start shell command "cprod -A fpc2 -c \"show sample-rr summary\"" | no-more   

Apr 02 23:06:30

  IPv4 Routes   : 1070438

  IPv6 Routes   : 800015. ------> routes fully converged

 

Router> show chassis fpc                                      

Apr 02 23:06:38

Slot State      (C) Total Interrupt   1min 5min 15min DRAM (MB) Heap  Buffer

 0 Empty      

 1 Online      42  100     8   99  90  67  2048   54    22

 2 Online      45  83     1   97  87  59  2048   58    22 ------ FPC 2 utilization dropped

Router> show chassis fpc   

Apr 02 23:06:42

          Temp CPU Utilization (%) CPU Utilization (%) Memory  Utilization (%)

Slot State      (C) Total Interrupt   1min 5min 15min DRAM (MB) Heap  Buffer

 0 Empty      

 1 Online      42  100     8   99  90  67  2048   54    22 ----FPC 1 utilisation high

 2 Online      45  19     1   84  85  59  2048   58    22

 

Router> start shell command "cprod -A fpc1 -c \"show sample-rr summary\"" | no-more   

  IPv4 Routes   : 1070438

  IPv6 Routes   : 535787 ---- Routes for FPC1 still getting converged

Router> show chassis fpc                                

Apr 02 23:07:22

Slot State      (C) Total Interrupt   1min 5min 15min DRAM (MB) Heap  Buffer

 0 Empty      

 1 Online      42  100     7   99  91  68  2048   56    22

 2 Online      45  20     1   51  76  57  2048   58    22

Router> start shell command "cprod -A fpc1 -c \"show sample-rr summary\"" | no-more   

Apr 02 23:08:04

  IPv4 Routes   : 1070438

  IPv6 Routes   : 800015 ----- once the routes got converged in sampler RR

Router> show chassis fpc                                

Apr 02 23:08:12

          Temp CPU Utilization (%) CPU Utilization (%) Memory  Utilization (%)

Slot State      (C) Total Interrupt   1min 5min 15min DRAM (MB) Heap  Buffer

 0 Empty      

 1 Online      42  39     8   89  90  69  2048   58    22 ------ FPC utilization came to normal

 2 Online      45  20     1   32  68  55  2048   58    22

 

Modification History

2024-04-07 : Article Created