Description

The Packet Process Engine UCODE Re-balancing (LKUP ASIC UCODE Rebalance Service) feature is enabled by default to improve forwarding performance. This microcode re-balance mechanism may lead to MPC CPU spikes seen on regular intervals in MX series routers with MPCs installed. This is harmless and does not impact service.

This mechanism increases overall packet forwarding performance or better throughput. The UCODE rebalancing improves packet processing by providing faster memory for the active UCODE instructions, which means latency per packet will be reduced. If lower latency is important to this APP/Service, then eliminating UCODE rebalancing may be noticeable.

Symptoms

Confirm if the CPU spikes are caused by LKUP ASIC UCODE rebalance service by running 'show sched' and 'show threads' from the FPC when the FPC CPU utilization is high. 

>start shell pfe network <fpc#>
#show sched
#show threads

Example: 

#show sched
Total system uptime 0+00:00:05, (21305264020 ms), 12333 thread dispatches
CPU load is 100% (1 second), 0% (5 second)
            0% (10 second), 0% (1 minute)
Total interrupt time 686 (ms)

 CPU       Name      Time(ms)
  1%       Idle      89
 85%    Threads      4431
 13%        ISR      686
 13%    Level 1      682
  0%    Level 2      3

Last Long Running Thread Event time 100 ms

 CPU       Name      Time(ms)    Count
 26%    Level 1      26      567

 Top Thread:
   pid      = 89
   name     = LKUP ASIC UCODE Rebalance Service  
   time     = 3413 ms
   cpu      = 67%

#show threads

PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
  1 H  asleep    Maintenance            368/2056   0/0/0 ms  0%
  2 L  running   Idle                   320/2056   0/0/91 ms  1%
  3 H  asleep    Timer Services         312/2048   0/0/0 ms  0%
  5 L  asleep    Ukern Syslog           312/4104   0/0/0 ms  0%
  6 L  asleep    Sheaf Background       400/2056   0/0/0 ms  0%
  7 H  asleep    IGMP                  1248/16392  0/0/1 ms  0%
  8 H  asleep    IPv4 PFE Control Background   296/8200   0/0/0 ms  0%
  9 M  asleep    OTN                    376/8200   0/0/0 ms  0%
 <...>
 89 L  asleep    LKUP ASIC UCODE Rebalance Service  2184/4104   0/2/3413 ms 67%
 <...>

Solution

MPC CPU spikes due to LKUP ASIC UCODE rebalance service are harmless and does not impact service. However, this feature can be disabled via a hidden knob if needed:

set chassis fpc <slot> ucode-rebalance disable