High CPU was observed on FPC and RE when the traffic was shifted away
>Customer observed High CPU based on jnxOperatingCPU metrics monitored .
The device was showing FPC CPU as 100% as seen from below command,
root@QFX5100-48s-6q> show chassis fpc no-forwarding
Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt 1min 5min 15min DRAM(MB) Heap Buffer
0 Online 31 100 0 0 0 0 1920 0 70
1 Empty
>Below is the "show chassis routing engine" command during issue time which shows high utilisation under user,
root@QFX5100-48s-6q> show chassis routing-engine no-forwarding
Routing Engine status:
Slot 0:
Current state Master
Temperature 31 degrees C / 87 degrees F
CPU temperature 31 degrees C / 87 degrees F
DRAM 1920 MB
Memory utilization 70 percent
5 sec CPU utilization:
User 89 percent
Background 0 percent
Kernel 11 percent
Interrupt 0 percent
Idle 0 percent
1 min CPU utilization:
User 68 percent <<<<<<<<<<<<<<<<
Kernel 14 percent
Idle 18 percent
5 min CPU utilization:
User 55 percent <<<<<<<<<<<<<<<<
Kernel 13 percent
Idle 32 percent
15 min CPU utilization:
User 57 percent <<<<<<<<<<<<<<<<
Idle 29 percent
Model QFX Routing Engine
Serial ID BUILTIN
Uptime 140 days, 20 hours, 24 minutes, 9 seconds
Last reboot reason 0x2000:hypervisor reboot
Load averages: 1 minute 5 minute 15 minute
2.69 1.89 1.99
>>Upon checking "show system process extensive | no-more" output we see "mgd" process is consuming more cpu as seen below
root@QFX5100-48s-6q> show system processes extensive no-forwarding
last pid: 1843; load averages: 2.71, 1.90, 2.00 up 140+20:26:57 03:10:41
172 processes: 3 running, 147 sleeping, 1 zombie, 21 waiting
Mem: 794M Active, 113M Inact, 487M Wired, 220M Cache, 69M Buf, 234M Free
Swap: 1106M Total, 1106M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
1753 root 70 0 72196K 34816K RUN 0:02 30.68% mgd <<<<<<<<<<<<<
11840 root 43 -1 1255M 562M select 803.3H 17.77% fxpc
10 root 155 52 0K 12K RUN 2322.2 13.96% idle
1697 root 46 0 72224K 35068K select 0:02 12.39% mgd
1759 root 53 0 65972K 52424K select 0:00 2.45% cli
313 root -8 0 0K 12K mdwait 0:01 1.37% md6
1730 remote 47 0 66000K 52516K select 0:00 1.35% cli
1692 remote 41 0 66008K 52520K select 0:00 0.74% cli
11840 root -52 -52 1255M 562M select 803.3H 0.59% fxpc
1760 root 71 0 72060K 34908K select 0:00 0.56% mgd
6 root -8 0 0K 12K - 12:19 0.34% g_down
332 root -8 0 0K 12K mdwait 0:01 0.34% md7
12013 root 40 0 53428K 39224K select 36.7H 0.20% mib2d
1318 root 41 0 15680K 8016K select 473:27 0.15% eventd
24 root -80 -183 0K 12K WAIT 2:46 0.10% irq14: ata0
273 root -8 0 0K 12K mdwait 0:00 0.05% md4
12040 root 40 0 28968K 22380K select 23.7H 0.00% snmpd
Some of the reasons on why "mgd" process might be consuming more can be due to below reason,
mgd
show chassis fpc no-forwarding
show chassis routing-engine
show system processes extensive | no-more
show interfaces extensive | save /var/tmp/interface-stats
request pfe execute target fpc0 command "show threads” <<<< check which thread occupy high cpu
request pfe execute target fpc0 command "show sched"
request pfe execute target fpc0 command "show pfe statistics traffic"
request pfe execute target fpc0 command "show pfe statistics notification"
Turn on task accounting and collect the task accounting detail output
(three times with a gap of 30 seconds).
Don't forget to turn it off once done.
set task accounting on
show task accounting detail
set task accounting off
show task memory detail
show task memeory summary
show task io
show task history
show task statistics
show task job
show task jobs
show krt queue
show krt state
2.Generate live core during issue state using root access
Login to the shell
ps -aux | grep mgd
gcore -c mgd.live-core.0 <pid>
If We are unable to collect the core and need to recover the device, kill the process consuming more cpu or reboot the box to resolve the issue