This article provide details regarding the high CPU utilization reported on Hypervisor for vFP VM.
In performance mode , typically one lcore (logical core) is assigned for one port and a multiple/single lcore is allocated to the worker thread. That lcore executes a function which runs in tight loop and tries to process packets even though it may or may not get any packets to process. Process packet means, it will either try to flush out TX Q, or try to receive packet from RX Q, etc. Even if it does not find packets to process, it will again go over and try to process packets. So that particular CPU will have 100% consumption. In addition, the CPU is allocated to the worker thread, so the CPU would be high.
However, in lite mode , the same infinite function is executed and tries to process packet (rx, tx). But if packet is not found in current iteration, then it goes again and tries to check for packet. The execution will go to sleep for some time. This means it will free up the CPU for some time. This is the reason we don’t observe this behavior in lite mode.
Sample outputs to verify the details:
Below are the two processes which will be running on vFP, and would consume CPU. The vFP is configured with 10CPU and 16GB memory with 2 NIC.
root@localhost:~# ps -aux | grep riot Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html root 1360 0.0 0.0 9368 916 ? S 12:04 0:00 sh /usr/share/pfe/start_dpdk_riot.sh 0x0BAA root 1373 0.0 0.0 9592 1340 ? S 12:04 0:00 sh start_riot.sh root 1495 676 0.4 44804808 74444 ? Sl 12:04 15:47 /home/pfe/riot/build/app/riot -c 0x3ff -n 2 --log-level=5 -w 13:00.0 -w 1b:00.0 -- --rx (0,0,0,3),(1,0,1,4), --tx (0,3),(1,4), --w 5,6,7,8,9 --f 1 --l2_mode 1 --rpio local,3000,3001 --hostif local,3002 --bsz (32,32),(64,64),(512,512) root 1914 0.0 0.0 4408 508 ttyS0 R+ 12:06 0:00 grep riot root@localhost:~# ps -aux | grep vmxt Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html root 1570 19.6 1.2 2199404 207700 ? Ssl 12:04 0:28 /usr/share/pfe/vmxt -N -C 4 root 1945 0.0 0.0 4408 504 ttyS0 S+ 12:07 0:00 grep vmxt root@localhost:~root@localhost:~#
root@localhost:~# ps -aux | grep riot
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html root 1360 0.0 0.0 9368 916 ? S 12:04 0:00 sh /usr/share/pfe/start_dpdk_riot.sh 0x0BAA root 1373 0.0 0.0 9592 1340 ? S 12:04 0:00 sh start_riot.sh root 1495 676 0.4 44804808 74444 ? Sl 12:04 15:47 /home/pfe/riot/build/app/riot -c 0x3ff -n 2 --log-level=5 -w 13:00.0 -w 1b:00.0 -- --rx (0,0,0,3),(1,0,1,4), --tx (0,3),(1,4), --w 5,6,7,8,9 --f 1 --l2_mode 1 --rpio local,3000,3001 --hostif local,3002 --bsz (32,32),(64,64),(512,512) root 1914 0.0 0.0 4408 508 ttyS0 R+ 12:06 0:00 grep riot root@localhost:~# ps -aux | grep vmxt Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html root 1570 19.6 1.2 2199404 207700 ? Ssl 12:04 0:28 /usr/share/pfe/vmxt -N -C 4 root 1945 0.0 0.0 4408 504 ttyS0 S+ 12:07 0:00 grep vmxt root@localhost:~root@localhost:~#
Next, when checking “top“ output for the above process, the lc can be seen:
root@localhost:~# top -H -p 1495 top - 12:07:36 up 4 min, 1 user, load average: 6.98, 3.28, 1.27 Tasks: 12 total, 7 running, 5 sleeping, 0 stopped, 0 zombie Cpu(s): 66.4%us, 1.6%sy, 0.0%ni, 32.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16431908k total, 13421092k used, 3010816k free, 18628k buffers Swap: 0k total, 0k used, 0k free, 2340312k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1512 root 20 0 42.7g 72m 2444 R 100 0.5 2:51.78 lcore-slave-3 1513 root 20 0 42.7g 72m 2444 R 100 0.5 2:51.98 lcore-slave-4 1514 root 20 0 42.7g 72m 2444 R 100 0.5 2:51.84 lcore-slave-5 1515 root 20 0 42.7g 72m 2444 R 100 0.5 2:51.87 lcore-slave-6 1516 root 20 0 42.7g 72m 2444 R 100 0.5 2:51.86 lcore-slave-7 1517 root 20 0 42.7g 72m 2444 R 100 0.5 2:51.87 lcore-slave-8 1518 root 20 0 42.7g 72m 2444 R 100 0.5 2:51.86 lcore-slave-9 1510 root 20 0 42.7g 72m 2444 S 12 0.5 0:26.95 lcore-slave-1 1495 root 20 0 42.7g 72m 2444 S 0 0.5 0:13.78 riot 1509 root 20 0 42.7g 72m 2444 S 0 0.5 0:00.00 eal-intr-thread 1511 root 20 0 42.7g 72m 2444 S 0 0.5 0:00.00 lcore-slave-2 1521 root 20 0 42.7g 72m 2444 S 0 0.5 0:00.00 riot root@localhost:~# top -H -p 1570 top - 12:08:02 up 5 min, 1 user, load average: 7.13, 3.61, 1.44 Tasks: 5 total, 0 running, 5 sleeping, 0 stopped, 0 zombie Cpu(s): 71.2%us, 1.6%sy, 0.0%ni, 32.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16431908k total, 13421092k used, 3010816k free, 18628k buffers Swap: 0k total, 0k used, 0k free, 2340312k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1570 root 20 0 2147m 202m 163m S 18 1.3 0:39.12 J-UKERN 1571 root 20 0 2147m 202m 163m S 0 1.3 0:00.01 J-UDOG 1572 root 20 0 2147m 202m 163m S 0 1.3 0:00.02 J-LOG 1573 root -14 0 2147m 202m 163m S 0 1.3 0:00.23 J-SCHED 1574 root -4 0 2147m 202m 163m S 0 1.3 0:00.00 J-REMOTE-PIO-EV
root@localhost:~#
As highlighted above, 7 CPU’s are seen running with 100% and core-slave-1 and J-UKERN combined using 30%( 12+18) . From the top output, approximately 66-72% of CPU is in use.