Description

Fibd utilization on re0 (primary) reaches 99% every few seconds.

Symptoms

root@test-re0> show system processes extensive node re0 | grep fibd 
13736 root     13736  20     0 507540 292320 R 1-15:50:49  0.8 99.9 fibd{FIBD.EN_US}
13736 root     17261  20     0 507540 292320 S 1-15:50:49  0.8  0.0 fibd{FIBD.EN_US}
13736 root     17262  20     0 507540 292320 S 1-15:50:49  0.8  0.0 fibd{fibd

Solution

When a topology churn happens, a lower priority task in fibd reevaluates all of its cached connection requests to see if any lookup produces a different result from the churn (currently ppmd lookups are cached). This background processing does not block fibd's normal services (lookups and DDS updates). Given the churn circumstances, it is normal to see fibd reaching 99% for every few seconds.

A simple clear arp would trigger this:

vrf:none] root@bx02:~# while [ 1 ]; do sleep 1; cli clear arp interface et-7/0/3:0; done

>>>

>>> while ports:

...     s.sendto(b'Tom', 0, (' 20.0.0.10', ports))

...     ports = ports - 1

...

root@test-re0> show system processes extensive node re0

top - 19:13:21 up 1 day,  9:52,  8 users,  load average: 2.41, 2.14, 1.88
Tasks: 571 total,   4 running, 564 sleeping,   0 stopped,   3 zombie
%Cpu(s): 11.5 us,  1.8 sy,  0.0 ni, 86.1 id,  0.0 wa,  0.2 hi,  0.3 si,  0.0 st
KiB Mem : 65345096 total, 47812692 free,  7963912 used,  9568492 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used. 56478596 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20586 root      20   0 5917676 385064 148572 R  99.3  0.6 215:31.61 fibd
10006 root      20   0 5776392 228540 143792 S  16.5  0.3 314:42.60 fabspoked-fchip
10115 root      20   0 5774344 228788 144452 S  16.2  0.4 314:46.41 fabspoked-fchip
10249 root      20   0 5774344 227668 143364 S  16.2  0.3 314:37.82 fabspoked-fchip
9952 root      20   0 5774344 228872 144408 R  15.5  0.4 315:22.87 fabspoked-fchip  

The EVO fibd daemon services route lookup requests for egress host traffic.  This daemon receives network topology updates from the EVO DDS facility.  The daemon also services a few CLI commands such as "show route forwarding-table" .

Within fibd, low priority background processing is initiated when the network topology changes.  This processing does not block normal fibd processing but may show fibd as having high CPU utilization when there are frequent topology changes to the process. Within the fibd process, we run a low priority background activity that evaluates a subset of network flows for updates resulting from changes in network topology.  These flows represent peers that receive periodic packets sent from the ppmd daemon. The ppmd daemon continues to send packets without interruption and is not blocked in any way by this fibd processing.

This processing occurs as a background activity within fibd and yields after 8 msec to ensure service of pending higher priority fibd activities (such as egress route lookups).  This background processing does not impact the overall system health. It is not unexpected that the low-priority background fibd processing can take multiple iterations to complete. Processing is triggered when the network state changes. 

During this time the fibd process will show as having high CPU utilization. Despite this, it is able and ready to service normal requests. From the data collected on the 1/11/2022 we observed that fibd was spending the bulk of its processing time performing this low-priority work. Essentially, fibd is operating as it was designed to operate.  A high CPU utilization is expected in this circumstance.   The CPU cycles consumed by the low priority background activity in fibd will not impact any system functions.