Description

This article explains about an issue seen on QFX10002 where the CPU utilization on the cli shows 100% although there is no impact on the device.

Symptoms

> High CPU seen on the device continuously. The CPU graph will show constant 100% no fluctuations.

> The CPU is 100% with Kernel utilization showing 100%.

 

 

root@QFX10K> show chassis routing-engine no-forwarding

Routing Engine status:
  Slot 0:
    Current state                  Master
    Temperature                 29 degrees C / 84 degrees F
    CPU temperature             29 degrees C / 84 degrees F
    DRAM                      3360 MB (4096 MB installed)
    Memory utilization          43 percent
    5 sec CPU utilization:
      User                      10 percent
      Background                 0 percent
      Kernel                   100 percent
      Interrupt                  0 percent
      Idle                       0 percent
    1 min CPU utilization:
      User                      11 percent
      Background                 0 percent
      Kernel                   100 percent
      Interrupt                  0 percent
      Idle                       0 percent
    5 min CPU utilization:
      User                      11 percent
      Background                 0 percent
      Kernel                   100 percent
      Interrupt                  0 percent
      Idle                       0 percent
    15 min CPU utilization:
      User                      11 percent
      Background                 0 percent
      Kernel                   100 percent          >>>>>>>> Kernel 100% continuously
      Interrupt                  0 percent
      Idle                       0 percent
    Model                          RE-QFX10002-72Q
    Serial ID                      BUILTIN
    Uptime                         2148 days, 20 hours, 21 minutes, 24 seconds
    Last reboot reason             0x1:power cycle/failure
    Load averages:                 1 minute   5 minute  15 minute
                                       0.01       0.02       0.00

> However "show system process" command shows Idle percent as 96%.

 

root@QFX10K> show system processes extensive 

last pid: 7777; load averages: 0.14, 0.03, 0.01 up 1858+09:29:24  03:37:10

144 processes: 4 running, 118 sleeping, 2 zombie, 20 waiting

 

Mem: 843M Active, 98M Inact, 255M Wired, 2032M Cache, 69M Buf, 39M Free

Swap: 1106M Total, 1106M Free

 

 

 PID USERNAME THR PRI NICE  SIZE  RES STATE  TIME  WCPU COMMAND

  10 root    1 171  52   0K  12K RUN    ??? 92.68% idle. >>>>> Idle percent is 92

 7741 ENG     1 96  0 29560K 27564K select  0:01 2.69% cli

 

 

> The top output from the Junos shell shows utilization to be normal.

 

 

CPU states: 0.0% user, 0.0% nice, 0.8% system, 1.9% interrupt, 97.4% idle

Mem: 845M Active, 98M Inact, 255M Wired, 2032M Cache, 69M Buf, 37M Free

Swap: 1106M Total, 1106M Free

last pid: 7781; load averages: 0.06, 0.03, 0.01                                                                  up 1858+09:30:1003:37:56

CPU states: 1.5% user, 0.0% nice, 1.9% system, 2.3% interrupt, 94.4% idle

Mem: 845M Active, 98M Inact, 255M Wired, 2032M Cache, 69M Buf, 37M Free

Swap: 1106M Total, 1106M Free

 

> The vmstat output from Junos shell shows wrong values in the kernel.

 

root@QFX10K>start shell

 

% vmstat

procs   memory   page          disks  faults   cpu

 faults   cpu

r b w   avm  fre flt re pi po fr sr ad0 ad1  in  sy cs us sy id

1 0 0 4581388 2082156  6  0  0  0 14  0  0  0  3  17 10 -206 -366 671 >>> The Idle percent showing 671 and the system and user percent is negative

 

 

Solution

This issue is seen on 15.x code.

This is usually seen with a device having high uptime.

The FreeBSD kernel is reporting wrong values and hence the kernel is showing 100%.

 

It is recommended to reboot the device to clear this issue and upgrade to the latest release.

Modification History

2024-01-28 : Article Created
2024-08-21 : Modified solution and making article public