Description

DPDK binary has been loaded successfully, but Contrail vrouter-agent is stuck in initializing state. The vhost0 is not reachable from the configured gateway, and if the gateway sends broadcast traffic such as ARP request, it is seen on the vhost0's packet sniffer but unicast packet from gateway such as ICMP request, is not received by vhost0. 

​This article explains what to look for you encounter this failure state and how to resolve the issue.

Symptoms

Running ' contrail-status' shows following state: 

# contrail-status -d
== Contrail vRouter ==
supervisor-vrouter:           active
contrail-vrouter-agent        initializing (XMPP:control-node:172.18.101.102, XMPP:control-node:172.18.101.103, XMPP:dns-server:172.18.101.101, XMPP:dns-server:172.18.101.103, Collector connection down)pid 12194, uptime 4 days, 17:18:01     
contrail-vrouter-dpdk         active              pid 12080, uptime 4 days, 17:18:04     
contrail-vrouter-nodemgr      initializing (Collector, Discovery:Collector[Subscribe - Connection Error] connection down)pid 12079, uptime 4 days, 17:18:04   
 
# vifdump -i 0 arp and host 172.29.21.195
vif0/0      PCI: 0000:00:00.0 (Speed 10000, Duplex 1)
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mon0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:25:32.591831 ARP, Request who-has 172.29.21.195 tell 172.29.21.214, length 46
18:25:33.595801 ARP, Request who-has 172.29.21.195 tell 172.29.21.214, length 46

The vhost0 is not reachable from the gateway or to the gateway. Traffic capture done on bond0 or vhost0 confirms ARP request from the gateway was received, but no ARP response is sent back.

Solution

The most common cause is due to a wrong MAC being configured for DPDK module, which is not the same as the vRouter agent's configured MAC or bond0/vhost0's MAC address.

  1. Check the exact MAC used by DPDK module in the memory.

    Example with mac=14:02:ec:6b:5f:4c used by DPDK: 

    # ps -aux | grep dpdk   
root     11472  709  0.0 4091000 7140 ?        Sl   17:16  41:01 /usr/bin/contrail-vrouter-dpdk --no-mrgbuf --no-daemon --vr_mpls_labels 6000 --vr_flow_entries=2097152 --vdev eth_bond_bond0,mode=4,xmit_policy=l34,socket_id=0,mac=14:02:ec:6b:5f:4c,slave=0000:08:00.0,slave=0000:81:00.1 --vr_mpls_labels=6000 --vlan_tci 2004 --vlan_fwd_intf_name bond0 --socket-mem 1024 1024
root     17316  0.0  0.0  10476   908 pts/3    S+   17:22   0:00 grep dpdk

  2. The above MAC must be consistent with vRouter agent's MAC and therefore vhost0's MAC. In the following output, we can see it is different: 

    # ifconfig vhost0
vhost0    Link encap:Ethernet  HWaddr 00:11:ac:1d:d4:58 
          inet addr:172.29.21.214  Bcast:172.29.21.255  Mask:255.255.255.192
          inet6 addr: fe80::686e:20ff:fefe:6e63/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9100  Metric:1
          RX packets:1554 errors:0 dropped:196 overruns:0 frame:0
          TX packets:539 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:95802 (95.8 KB)  TX bytes:67274 (67.2 KB)
    ​ ​​
    This is because the user may have entered a wrong MAC in DPDK's initial file. 
    cat /etc/contrail/supervisord_vrouter_files/contrail-vrouter-dpdk.ini
[program:contrail-vrouter-dpdk]
command=taskset -c  1,25,2,26,13,37,14,38 /usr/bin/contrail-vrouter-dpdk --no-mrgbuf --no-daemon --vr_mpls_labels 6000 --vr_flow_entries=2097152 --vdev "eth_bond_bond0,mode=4,xmit_policy=l34,socket_id=0,mac=14:02:ec:6b:5f:4c,slave=0000:08:00.0,slave=0000:81:00.1" --vr_mpls_labels=6000 --vlan_tci "2004" --vlan_fwd_intf_name "bond0" --socket-mem 1024,1024
priority=410
autostart=True
killasgroup=True
stdout_capture_maxbytes=1MB
redirect_stderr=True
stdout_logfile=/var/log/contrail/contrail-vrouter-dpdk-stdout.log
stderr_logfile=/var/log/contrail/contrail-vrouter-dpdk-stderr.log
exitcodes=0
loglevel=debug​

After fixing the error above and restarting supervisor-vrouter , the vrouter agent comes up as expected.