In a Contrail Networking (CN) with RedHat OpenStack (RHOSP) setup, the " contrail-status " command shows the vRouter status as "inactive" and states that " vRouter is NOT PRESENT ."
contrail-status
vRouter is NOT PRESENT
This article details the troubleshooting steps to perform when investigations show that this is a problem with DPDK port binding.
When the contrail-status command is run, the vRouter status does not come up and shows as "inactive." When you run the Python script that is available within /opt/contrail/bin to check the DPDK port binding, the result shows <None> .
/opt/contrail/bin
[root@DPDK-Compute ~]# contrail-status vRouter is NOT PRESENT == Contrail vRouter == supervisor-vrouter: inactive contrail-vrouter-agent inactive contrail-vrouter-nodemgr failed ========Run time service failures============= [root@DPDK-Compute ~]# [root@DPDK-Compute ~]# /opt/contrail/bin/dpdk_nic_bind.py -s Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== 0000:01:00.0 'I350 Gigabit Network Connection' if=em3 drv=igb unused=vfio-pci 0000:01:00.1 'I350 Gigabit Network Connection' if=em4 drv=igb unused=vfio-pci 0000:19:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=em1 drv=i40e unused=vfio-pci 0000:19:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=em2 drv=i40e unused=vfio-pci 0000:87:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=p7p1 drv=i40e unused=vfio-pci 0000:87:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=p7p2 drv=i40e unused=vfio-pci Other network devices ===================== 0000:5e:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci 0000:5e:00.1 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci 0000:5f:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci 0000:5f:00.1 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci [root@DPDK-Compute ~]#
In fact, the DPDK ports that need to be bound are not even bound to the kernel driver. The devices appear under "Other network devices." The MAC address for the DPDK ports are:
1. 0000:5e:00.0 2. 0000:5e:00.1 3. 0000:5f:00.0 4. 0000:5f:00.1
To troubleshoot this problem, perform the following:
Bind the DPDK ports by using the command " /opt/contrail/bin/dpdk_nic_bind.py -b vfio-pci <MAC-ADDR> " and then perform a " supervisor-vrouter " restart.
/opt/contrail/bin/dpdk_nic_bind.py -b vfio-pci <MAC-ADDR>
supervisor-vrouter
Note that this may fail, and not help restore the DPDK binding but remove the DPDK binding again, moving the devices to the "Other network devices" section. As you observed earlier, the ports that needed to be bound to the DPDK were not bound to the kernel driver.
Note that this may fail, and not help restore the DPDK binding but remove the DPDK binding again, moving the devices to the "Other network devices" section.
As you observed earlier, the ports that needed to be bound to the DPDK were not bound to the kernel driver.
To restore the DPDK binding, bind the DPDK ports back to a system Ethernet driver first by using the following commands:
dpdk_nic_bind.py -e 0000:5e:00.0 dpdk_nic_bind.py -e 0000:5e:00.1 dpdk_nic_bind.py -e 0000:5f:00.0 dpdk_nic_bind.py -e 0000:5f:00.1
After performing the above steps, the status of the ports displays as shown below:
[root@DPDK-Compute ~]# /opt/contrail/bin/dpdk_nic_bind.py -s Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== 0000:01:00.0 'I350 Gigabit Network Connection' if=em3 drv=igb unused= 0000:01:00.1 'I350 Gigabit Network Connection' if=em4 drv=igb unused= 0000:19:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=em1 drv=i40e unused= 0000:19:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=em2 drv=i40e unused= 0000:5e:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=p3p1 drv=i40e unused= 0000:5e:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=p3p2 drv=i40e unused= 0000:5f:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=p2p1 drv=i40e unused= 0000:5f:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=p2p2 drv=i40e unused= 0000:87:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=p7p1 drv=i40e unused= 0000:87:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=p7p2 drv=i40e unused= Other network devices ===================== <none> [root@DPDK-Compute ~]#
All ports are now attached to the Ethernet driver and start showing under the kernel section.
Now restart " supervisor-vrouter ", so that the ports are directly taken up from the kernel and assigned under DPDK during the restart operation.
[root@DPDK-Compute ~]# systemctl restart supervisor-vrouter
While the supervisor-vrouter is being restarted, you should see the DPDK ports getting bound properly under the DPDK section. After the restart is completed, you should see all the four DPDK ports properly attached.
Now let us look at the bind status again:
[root@DPDK-Compute ~]# /opt/contrail/bin/dpdk_nic_bind.py -s Network devices using DPDK-compatible driver ============================================ 0000:5e:00.0 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e 0000:5e:00.1 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e 0000:5f:00.0 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e 0000:5f:00.1 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e Network devices using kernel driver =================================== 0000:01:00.0 'I350 Gigabit Network Connection' if=em3 drv=igb unused=vfio-pci 0000:01:00.1 'I350 Gigabit Network Connection' if=em4 drv=igb unused=vfio-pci 0000:19:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=em1 drv=i40e unused=vfio-pci 0000:19:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=em2 drv=i40e unused=vfio-pci 0000:87:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=p7p1 drv=i40e unused=vfio-pci 0000:87:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=p7p2 drv=i40e unused=vfio-pci Other network devices ===================== <none> [root@DPDK-Compute ~]#
When you monitor the contrail-status command output, you might see Failed status for " contrail-vrouter-nodemgr " as shown below:
contrail-vrouter-nodemgr
[root@DPDK-Compute ~]# contrail-status vRouter is NOT PRESENT == Contrail vRouter == supervisor-vrouter: inactive contrail-vrouter-agent initializing contrail-vrouter-dpdk active contrail-vrouter-nodemgr failed ========Run time service failures============= /var/crashes/core.contrail-vroute.4174.DPDK-Compute.1595296411 /var/crashes/core.contrail-vroute.228285.DPDK-Compute.1595854123 /var/crashes/core.lcore-slave-9.10723.DPDK-Compute.1631422810 /var/crashes/core.contrail-vroute.10724.DPDK-Compute.1631422823 /var/crashes/core.lcore-slave-9.12148.DPDK-Compute.1631422916 /var/crashes/core.contrail-vroute.12674.DPDK-Compute.1631422929 [root@DPDK-Compute ~]#
To determine the cause, check supervisord-vrouter.log :
supervisord-vrouter.log
[root@DPDK-Compute ~]# tail -f /var/log/contrail/supervisord-vrouter.log 2021-09-12 08:06:55,070 INFO exited: contrail-vrouter-nodemgr (exit status 0; not expected) 2021-09-12 08:06:56,703 INFO spawned: 'contrail-vrouter-nodemgr' with pid 15756 2021-09-12 08:06:56,703 INFO success: contrail-vrouter-dpdk entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-12 08:06:57,022 INFO exited: contrail-vrouter-nodemgr (exit status 0; not expected) 2021-09-12 08:06:59,025 INFO spawned: 'contrail-vrouter-nodemgr' with pid 15768 2021-09-12 08:06:59,343 INFO exited: contrail-vrouter-nodemgr (exit status 0; not expected) 2021-09-12 08:07:00,344 INFO success: contrail-vrouter-agent entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) 2021-09-12 08:07:02,347 INFO spawned: 'contrail-vrouter-nodemgr' with pid 15775 2021-09-12 08:07:02,691 INFO exited: contrail-vrouter-nodemgr (exit status 0; not expected) 2021-09-12 08:07:02,691 INFO gave up: contrail-vrouter-nodemgr entered FATAL state, too many start retries too quickly Wait for around 10 minutes, and then restart " contrail-vrouter-nodemgr ." Monitor for " contrail-vrouter-nodemgr " to come up without any issue. [root@DPDK-Compute ~]# systemctl restart contrail-vrouter-nodemgr
[root@DPDK-Compute ~]# tail -f /var/log/contrail/supervisord-vrouter.log 2021-09-12 08:06:55,070 INFO exited: contrail-vrouter-nodemgr (exit status 0; not expected) 2021-09-12 08:06:56,703 INFO spawned: 'contrail-vrouter-nodemgr' with pid 15756 2021-09-12 08:06:56,703 INFO success: contrail-vrouter-dpdk entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-12 08:06:57,022 INFO exited: contrail-vrouter-nodemgr (exit status 0; not expected) 2021-09-12 08:06:59,025 INFO spawned: 'contrail-vrouter-nodemgr' with pid 15768 2021-09-12 08:06:59,343 INFO exited: contrail-vrouter-nodemgr (exit status 0; not expected) 2021-09-12 08:07:00,344 INFO success: contrail-vrouter-agent entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) 2021-09-12 08:07:02,347 INFO spawned: 'contrail-vrouter-nodemgr' with pid 15775 2021-09-12 08:07:02,691 INFO exited: contrail-vrouter-nodemgr (exit status 0; not expected) 2021-09-12 08:07:02,691 INFO gave up: contrail-vrouter-nodemgr entered FATAL state, too many start retries too quickly
Wait for around 10 minutes, and then restart " contrail-vrouter-nodemgr ." Monitor for " contrail-vrouter-nodemgr " to come up without any issue.
[root@DPDK-Compute ~]# systemctl restart contrail-vrouter-nodemgr
Now restart "supervisord-vrouter" again and confirm that the vRouter has come up without any problems.
[root@DPDK-Compute ~]# systemctl restart supervisord-vrouter
Monitor the contrail-status output:
[root@DPDK-Compute ~]# contrail-status == Contrail vRouter == supervisor-vrouter: active contrail-vrouter-agent active contrail-vrouter-dpdk active contrail-vrouter-nodemgr active ========Run time service failures============= /var/crashes/core.contrail-vroute.4174.DPDK-Compute.1595296411 /var/crashes/core.contrail-vroute.228285.DPDK-Compute.1595854123 /var/crashes/core.lcore-slave-9.10723.DPDK-Compute.1631422810 /var/crashes/core.contrail-vroute.10724.DPDK-Compute.1631422823 /var/crashes/core.lcore-slave-9.12148.DPDK-Compute.1631422916 /var/crashes/core.contrail-vroute.12674.DPDK-Compute.1631422929 [root@DPDK-Compute ~]#
Note: If the above steps do not resolve the problem, contact Support .