Description

This article outlines a high cpu issue on QFX10002 device and how to mitigate the same.

Symptoms

> High CPU seen on the device continuously.

> The CPU is 100% with Kernel Utilization showing 100%.

 

root@QFX10K> show chassis routing-engine no-forwarding

Routing Engine status:
  Slot 0:
    Current state                  Master
    Temperature                 29 degrees C / 84 degrees F
    CPU temperature             29 degrees C / 84 degrees F
    DRAM                      3360 MB (4096 MB installed)
    Memory utilization          43 percent
    5 sec CPU utilization:
      User                      10 percent
      Background                 0 percent
      Kernel                   100 percent
      Interrupt                  0 percent
      Idle                       0 percent
    1 min CPU utilization:
      User                      11 percent
      Background                 0 percent
      Kernel                   100 percent
      Interrupt                  0 percent
      Idle                       0 percent
    5 min CPU utilization:
      User                      11 percent
      Background                 0 percent
      Kernel                   100 percent
      Interrupt                  0 percent
      Idle                       0 percent
    15 min CPU utilization:
      User                      11 percent
      Background                 0 percent
      Kernel                   100 percent          >>>>>>>> Kernel 100% continuously
      Interrupt                  0 percent
      Idle                       0 percent
    Model                          RE-QFX10002-72Q
    Serial ID                      BUILTIN
    Uptime                         2148 days, 20 hours, 21 minutes, 24 seconds
    Last reboot reason             0x1:power cycle/failure
    Load averages:                 1 minute   5 minute  15 minute
                                       0.01       0.02       0.00

> The user cpu percentage is only 10%.
> The caed process will show high utilization.

root@QFX10K> show system processes extensive no-forwarding 1637 root 1 132 0 33196K 7292K RUN ??? 96.34% caed<<<<<<<<<<<< CAED process high

> No other impact seen due to this high cpu.

 

This issue is seen if we have the following configuration on the device

root@QFX10K> show cloud-analytics linecard 
Slot     PFE Id    Port     Connection State
0      0      0      Established

 

This problem is seen on releases before 16.1R4.

 

 

Solution

CAED(Cloud Analytics Engine Daemon) is enabled by default in Junos OS for supported QFX Series switches in releases prior to 16.1R4. This feature is disabled by default in later releases 16.1R4, 16.2R1, 15.1X53-D70 and onwards.

 

# set system processes cloud-analytics-engine disable

Note: This change needs to be done during a maintenance window.

Modification History

Added PR to the KB
Made changes to include caed task