From time to time, a VM landed on a DPDK compute fails to get an IP address from the vRouter, even if the TAP interface is up and the virtual network has DHCP enabled.
This article describes a few scenarios when such a problem is seen and provides steps to troubleshoot it.
VM ports are created successfully on the compute node and all VRouter services are up and running. However, no DHCP IP is assigned inside the VM.
Compute:/var/lib/contrail/ports# vif --list |grep -A5 tapad6b0e1f-84 vif0/10 PMD: tapad6b0e1f-84 Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0 Vrf:3 Flags:PL3L2D MTU:9160 Ref:16 RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 RX packets:0 bytes:0 errors:0 TX packets:150 bytes:6300 errors:0
vifdump on the TAP interface does not show DHCP requests from the VM, but DHCP requests going out can be seen with a packet sniffer within the VM.
Virtual network has DHCP enabled.
VMs spawned on kernel mode vRouter with non-DPDK flavors in the same virtual network are getting DHCP IP address.
Confirm that the following conditions are met for a newly spawned VM on a DPDK compute.
1. Confirm that the HugePages are configured on the compute. In the the example below, the dpdk compute node has Hugepagesize set to 2048 kB. It should have a non-zero value; if it is zero, then it needs to be configured on the VM.
Compute:~# cat /proc/meminfo|grep -i huge AnonHugePages: 1249280 kB HugePages_Total: 48331 HugePages_Free: 42660 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB
2. Confirm the newly spawned VM has hugepage configured. The VM UUID can be found in the Contrail UI - Monitor- Networking - Instances. It should have a non-zero value; if it is zero, then it needs to be configured on the VM.
Compute:~#virsh dumpxml <vm_UUID> | grep page root@bcomp80:~# virsh dumpxml 2 | grep page <hugepages> <page size='2048' unit='KiB' nodeset='0'/>
3. Double check the dpdk flavor that customer uses has hw:mem_page_size meta_data set. On the Contrail Control node run the following commands:
Control:~#source /etc/contrail/openstackrc Control:~# nova flavor-list +--------------------------------------+----------------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +--------------------------------------+----------------+-----------+------+-----------+------+-------+-------------+-----------+ | 088bb630-7779-4254-8cd9-07e9c0d708c0 | m1.small.dpdk | 2048 | 20 | 0 | | 1 | 1.0 | True | | 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True | | 1af1af01-fa89-486f-9923-41b3476fae05 | m1.tiny.dpdk | 512 | 1 | 0 | | 1 | 1.0 | True | | 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True | | 4b7b8390-f26f-481d-a452-ce9f81ea3e6c | m1.medium.dpdk | 4096 | 40 | 0 | | 2 | 1.0 | True | | 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True | +--------------------------------------+----------------+-----------+------+-----------+------+-------+-------------+-----------+ Control:~# nova flavor-show m1.small.dpdk +----------------------------+--------------------------------------+ | Property | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | disk | 20 | | extra_specs | {"hw:mem_page_size": "large"} | | id | 64d73fe0-73cd-4d5b-bf88-88ee77064b2e | | name | m1.small.dpdk | | os-flavor-access:is_public | True | | ram | 2048 | | rxtx_factor | 1.0 | | swap | | | vcpus | 1 | +----------------------------+--------------------------------------+
4. On the Contrail Control node, check that the tap interface has the correct vif type of "vhostuser". For kernel mode vRouter, the vif type should be "vrouter", whereas for a DPDK mode vrouter the vif type should be "vhostuser".
Control:~# neutron port-list|grep <VM IP> | c15a31a0-7ba9-4f3e-8662-ddd448f5b788 | c15a31a0-7ba9-4f3e-8662-ddd448f5b788 | 02:c1:5a:31:a0:7b | {"subnet_id": "716f54a0-ea89-42bd-9d5d-159d4759365b", "ip_address": "<VM IP>"}| Control:~# neutron port-show c15a31a0-7ba9-4f3e-8662-ddd448f5b788 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | bcomp80 | | binding:vif_details | {"vhostuser_socket": "/var/run/vrouter/uvh_vif_tapc15a31a0-7b", "vhostuser_vrouter_plug": true, "vhostuser_mode": "client"} | | binding:vif_type | vhostuser | | binding:vnic_type | normal | | device_id | c41f5d58-9e08-4616-a72f-48c965bd5a1d | | device_owner | compute:bComp80-dpdk | | fixed_ips | {"subnet_id": "716f54a0-ea89-42bd-9d5d-159d4759365b", "ip_address": "192.168.1.9"} | | id | c15a31a0-7ba9-4f3e-8662-ddd448f5b788 | | mac_address | 02:c1:5a:31:a0:7b | | name | c15a31a0-7ba9-4f3e-8662-ddd448f5b788 | | network_id | b5c9db20-3d9b-46b6-9a61-b7e32ee64a10 | | port_security_enabled | True | | security_groups | bb92a4f7-9225-4302-8106-179a9144b082 |
If the vif type is incorrect, check whether the dpdk compute has been correctly configured. Check the compute node configuration in Contrail Configdb to see whether "virtual_router_dpdk_enabled" has been set to "true". If the knob is missing, it could be a deployment issue. The following API call on the Contrail API admin port can be used to fetch vrouter details; it can be run on the Contrail control/config node using the keystone admin credentials.
Control:~# curl -u admin:<password> http://localhost:8095/virtual-routers |python -mjson.tool <The o/p will list all the vrouters that are part of the deployment> Control:~# curl -u admin:<password> http://localhost:8095/virtual-router/<vrouter UUID> |python -mjson.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 997 100 997 0 0 189k 0 --:--:-- --:--:-- --:--:-- 194k "href": "http://localhost:8095/virtual-router/10d83960-cc68-4cdd-bcf4-241bf8ef6e13", "id_perms": { "created": "2018-03-28T03:17:46.518439", "creator": null, -- "uuid": "10d83960-cc68-4cdd-bcf4-241bf8ef6e13", "virtual_router_dpdk_enabled": true, <<< Make sure dpdk has been enabled on this compute. "virtual_router_ip_address": "172.18.102.80" }
2020-05-27: Minor, non-technical edits.