Description

On a scaled setup with extensive routes and BGP neighbors, high CPU is observed for RPD (new primary) post RE switchover, which may last as long as 20-30 minutes.

Solution

The following are tasks in RPD that are consuming most of the CPU post switchover (On a scale of ~ 8 Million  routes and ~9600 peers):

  1. Route Flash – Post switchover RPD Infra triggers flash processing for all the prefixes. This flash is consumed by BGP to detect any change that needs to be advertised to the BGP peer. Export policy is evaluated for the BGP routes as part of flash processing.

  2. BGP Import evaluation – BGP registers for switchover detection. Once BGP identifies that RPD (backup) switched to primary mode, it triggers Import policy evaluation for all established peers (in this case around 9600 peers). Import policy evaluation walks through all BGP RIBs and evaluates if any change is observed w.r.t to route’s nexthops, metrics or other relevant parameters. If BGP multipath is in use, this is required for BGP multipath processing to trigger multipath calculation (BGP multipath is not enabled on backup RPD by default).

  3. Rsync – Once the new backup RPD (old primary) comes up, it has to re-synchronize all the BGP specific data. This synchronization on scale will consume considerable CPU.

Based on the data collected, most of the RPD CPU consumption is due to the tasks/jobs as listed above.
CPU consumption is high due to the scale configured (8Million routes and around 9620 established peers). This is expected and BGP is working as per design.
No functional discrepancy is identified (no peer flaps, no interface flaps, no route deletes/changes to FIB).

Data collected:

  1. Rtsockmon -nrt (rtsockmon logs). This provides logs for any route or nexthop changes down to FIB (from RPD to kernel)
  2. BGP statistics – To check the number of routes processed for Import policy evaluation
  3. BGP summary – To check the connected peers/groups
  4. Show task jobs – Collected the task jobs while CPU is high (To provide the list of jobs running duting RPD high CPU condition)
 

Test Scenario

 Peers configured: 

Threading mode: BGP I/O
Groups: 4347 Peers: 9620 Down peers: 606
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending

 
Jobs running during high CPU condition: 

Background jobs:
Pri Task Job Name                         Runs Misses   Wait Flags
  0 bgpio-2/mgmt_resp_proc                   0      0      0
  0 bgpio-1/mgmt_resp_proc                   0      0      0
  3 OSPF/OSPF flood                          0      0      0 <--- SubJob
  4 KStat/krt_tx_to_kernel                   0      0      1
  4 Mirror Task.128.0.0.6+62307/mirror       0      0      0
  5 RT/Flash update INET                  2686   1037      1 <--- Starving SubJob
  5 RT/Flash update INET6                 2678    768      1 <--- SubJob
  5 KStat/Statistics                         8    512      1
  5 BGP_RT_Background/BGP Peer reconfi    2693      0      0
  6 BGP rsync/BGP rsync write                0   1280      1

Background jobs: 

Pri Task Job Name                         Runs Misses   Wait Flags
  0 bgpio-2/mgmt_resp_proc                   0      0      0
  0 bgpio-0/mgmt_resp_proc                   0      0      0
  4 Mirror Task.128.0.0.6+62307/mirror       0      0      0
  4 KStat/krt_tx_to_kernel                   0      0      0
  5 KStat/Statistics                         1    512      0
  5 RT/Flash update INET6                 4845    256      0 <--- SubJob
  5 BGP_RT_Background/BGP Peer reconfi    4860      0      0
  6 BGP rsync/BGP rsync write                0   1024      0

Import evaluation for all BGP peers: 

Total routes_count       0        Total jobs_count       0
last iter routes_count   8441600  last iter jobs_count   8508
Initialization done, Peer thread is not empty