This article explores useful commands to check the various running status of vrouter DPDK threads.

KB36056 [juniper.net] explains the roles of different threads in dpdk-vrouter and shows how to use the 'ps' command to list them.

KB35517 [juniper.net] explains how CPU pinning works with examples.

The 'ps' command is a powerful, versatile tool that give all kinds of useful information about a running process.  With different parameters of the 'ps' command, we can check the load of each vrouter dpdk threads:

    $ ps -mo pid,tid,comm,psr,pcpu -p $(ps -ef | awk '$8=="/usr/bin/contrail-vrouter-dpdk" {print $2}')
      PID   TID COMMAND         PSR %CPU
     3638     - contrail-vroute   -  803
        -  3638 -                 0  2.2
        -  4026 -                15  0.0
        -  4027 -                23  2.8
        -  4028 -                34  0.0
        -  4029 -                21  0.0
        -  4030 -                12  0.2
        -  4031 -                 1 99.9        #<---forwarding thread
        -  4032 -                 2 99.9        #<---forwarding thread
        -  4033 -                13 99.9        #<---forwarding thread
        -  4034 -                14 99.8        #<---forwarding thread
        -  4035 -                25 99.9        #<---forwarding thread
        -  4036 -                26 99.9        #<---forwarding thread
        -  4037 -                37 99.9        #<---forwarding thread
        -  4038 -                38 99.9        #<---forwarding thread
        -  5941 -                20  0.0

The output of this command shows, the 8 forwarding threads are running with full CPU utilizations. The reason is that with DPDK vrouter design, each forwarding thread keeps polling the queue to see if any packets need to be received/sent in a indefinite loop, regardless of the actual traffic volumn. This is part of the performance boost method in the design. Other threads, instead, does not consume high CPU because they are not directly used to handle the user packets. Another useful information is the CPU pinning status. In KB35517 [juniper.net], the 'taskset' command is used to show the same result. For example, we can tell thread 4031 is pinned to CPU#1, 4032 is pinned to CPU#2, 4033 is pinned to CPU#13, etc. These matches to the CPU assignment configuration in '/etc/contrail/supervisord_vrouter_files/contrail-vrouter-dpdk.ini' .

Similar result can be displayed with the command, 'pidstat' as shown below:

    $ pidstat -t -p `pidof contrail-vrouter-dpdk`
    Linux 3.13.0-128-generic (bcomp80)      06/23/2020      _x86_64_        (48 CPU)

    02:56:41 PM   UID      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
    02:56:41 PM     0      3638         -  494.60  308.70    0.00  803.30     0  contrail-vroute
    02:56:41 PM     0         -      3638    0.69    1.52    0.00    2.21     0  |__contrail-vroute
    02:56:41 PM     0         -      4026    0.01    0.01    0.00    0.02    17  |__eal-intr-thread
    02:56:41 PM     0         -      4027    0.57    2.29    0.00    2.87    19  |__lcore-slave-1
    02:56:41 PM     0         -      4028    0.00    0.00    0.00    0.00    34  |__lcore-slave-2
    02:56:41 PM     0         -      4029    0.01    0.03    0.00    0.04    17  |__lcore-slave-8
    02:56:41 PM     0         -      4030    0.11    0.11    0.00    0.22    43  |__lcore-slave-9
    02:56:41 PM     0         -      4031   62.08   37.66    0.00   99.74     1  |__lcore-slave-10
    02:56:41 PM     0         -      4032   60.74   39.00    0.00   99.74     2  |__lcore-slave-11
    02:56:41 PM     0         -      4033   61.92   37.82    0.00   99.74    13  |__lcore-slave-12
    02:56:41 PM     0         -      4034   61.91   37.83    0.00   99.73    14  |__lcore-slave-13
    02:56:41 PM     0         -      4035   61.53   38.21    0.00   99.74    25  |__lcore-slave-14
    02:56:41 PM     0         -      4036   60.79   38.95    0.00   99.74    26  |__lcore-slave-15
    02:56:41 PM     0         -      4037   61.96   37.79    0.00   99.74    37  |__lcore-slave-16
    02:56:41 PM     0         -      4038   61.37   38.38    0.00   99.74    38  |__lcore-slave-17
    02:56:41 PM     0         -      5941    0.00    0.00    0.00    0.00    20  |__lcore-slave-9

It is better than the 'ps' command at least in one aspect that the tree structure of the threads and their parent process (dpdk-vrouter) is remained, along with the threads name, so you can easily tell who is who. The drawback is that this command does not come with the standard Linux distribution and requires to be be installed before it can be used.