Description

In a DPDK compute node, a common question that users ask is why does the dpdk-vrouter have an affinity to all available CPU cores in the system even if it is configured to bind to specific cores in the config file. This article explains this behavior.  

Symptoms

From the configuration below, we pin CPU cores 1, 25, 2, 26, 13, 37, 14, and 38 for DPDK module:
 
    $ cat /etc/contrail/supervisord_vrouter_files/contrail-vrouter-dpdk.ini|grep taskset
    command=taskset 
        -c 1,25,2,26,13,37,14,38 /usr/bin/contrail-vrouter-dpdk
        --no-daemon 
        --vdev "eth_bond_bond0,mode=4,xmit_policy=l34,socket_id=1,mac=14:02:ec:66:b8:dc,slave=0000:87:00.0,slave=0000:09:00.1"
        --vlan_tci "722" 
        --vlan_fwd_intf_name "bond0" 
        --socket-mem 1024,1024

Note: The output gives one long line which has been broken into multiple lines for better readability.

From the output shown above, the expected result should be dpdk-vrouter with CPU affinity set to `1,25,2,26,13,37,14,38`. However, the DPDK thread uses more than those 8 defined cores:

    $ ps -o pid,spid,comm,cpuid -Tp `pidof contrail-vrouter-dpdk`
    PID  SPID COMMAND         CPUID
    10416 10416 contrail-vroute    21
    10416 10665 eal-intr-thread     0
    10416 10666 lcore-slave-1       0
    10416 10667 lcore-slave-2      12
    10416 10668 lcore-slave-8      17
    10416 10669 lcore-slave-9      20
    10416 10670 lcore-slave-10      1
    10416 10671 lcore-slave-11      2
    10416 10672 lcore-slave-12     13
    10416 10673 lcore-slave-13     14
    10416 10674 lcore-slave-14     25
    10416 10675 lcore-slave-15     26
    10416 10676 lcore-slave-16     37
    10416 10677 lcore-slave-17     38
    10416 13674 lcore-slave-9      15

The `ps` command first calls another command, `pidof contrail-vrouter-dpdk` to get the process ID (PID) of DPDK process. With "thread view" (`-T`) it prints information about each thread in the same process separately. The `-o` option specifies what columns to look at. In this example, it will print four columns: 

* `PID`: process ID of current dpdk process
* `SPID`: thread process ID of each threads
* `comm`: name of each thread
* `CPUID`: CPU ID

In this setup, a total of 15 CPU cores are being used by DPDK threads. Furthermore, with `taskset` command, we can see that the DPDK process indeed utilizes all 48 cores in the system .

    $ taskset -c -p `pidof contrail-vrouter-dpdk`
    pid 10416's current affinity list: 0-47

Solution

This behavior is working as designed. With the following `-a` option added in the same `taskset` command, from the output, we can see that cores for forwarding are using what is defined in the configuration file `contrail-vrouter-dpdk.ini`. However, for control we can use any core from 0-47. 

    $ taskset -a -cp `pidof contrail-vrouter-dpdk `
    pid 10416's current affinity list: 0-47     #<---control
    pid 10665's current affinity list: 0-47     #<---control
    pid 10666's current affinity list: 0-47     #<---control
    pid 10667's current affinity list: 0-47     #<---control
    pid 10668's current affinity list: 0-47     #<---control
    pid 10669's current affinity list: 0-47     #<---control
    pid 10670's current affinity list: 1        #<---forwarding
    pid 10671's current affinity list: 2        #<---forwarding
    pid 10672's current affinity list: 13       #<---forwarding
    pid 10673's current affinity list: 14       #<---forwarding
    pid 10674's current affinity list: 25       #<---forwarding
    pid 10675's current affinity list: 26       #<---forwarding
    pid 10676's current affinity list: 37       #<---forwarding
    pid 10677's current affinity list: 38       #<---forwarding
    pid 13674's current affinity list: 0-47     #<---forwarding

The output of the `ps` command lists all the cores used by DPDK, including cores used both for forwarding and control purposes.  However, the `contrail-vrouter-dpdk.ini` configuration file only defines the cores allowed
to be used for forwarding purpose. In other words, the CPU pinning for `contrail-vrouter-dpdk` process is only for resource consuming threads that forward packets. The pinning doesn't apply to the threads doing control related tasks. To change this default behavior,  use taskset to pin the control threads if needed. Or pin the CPUs to certain VMs to prevent those threads from using the same CPUs.