Article Detail

Description

This article explains the procedure for identifying NIC firmware on a dpdk-vrouter. There can be instances when NIC firmware is upgraded and further DPDK vrouter fails to start up. This article will be helpful in planning for an upgrade of NIC firmware to verify firmware support and troubleshoot instances where DPDK vrouter fails to come up after NIC firmware upgrade.

This article is applicable for Intel NIC X710/XL710/XXV710 and X722.

Symptoms

DPDK vrouter is loosely coupled with NIC firmware and i40e kernel driver. In case of mismatch or upgrades of NIC firmware, which may lead to the failing of DPDK vrouter initializing state.

Solution

This procedure checks the firmware version of a NIC and how to unbind a DPDK NIC which is required to check firmware.

6. Once the NIC is bound to kernel driver i40e driver, the NIC name can be seen present in lshw:

[root@overcloud-contraildpdk-1 heat-admin]# lshw -c network -businfo
Bus info          Device     Class          Description
=======================================================
pci@0000:01:00.0  em3        network        I350 Gigabit Network Connection
pci@0000:01:00.1  em4        network        I350 Gigabit Network Connection
pci@0000:19:00.0  em1        network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:19:00.1  em2        network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:3b:00.0  p1p1       network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:3b:00.1  p1p2       network        Ethernet Controller X710 for 10GbE SFP+
                  vlan710    network        Ethernet interface
                  vlan730    network        Ethernet interface
                  docker0    network        Ethernet interface

Identify the NIC used for DPDK on the compute node with the following command from the DPDK container. Here, the X710 NIC is used:

(vrouter-agent-dpdk)[root@overcloud-contraildpdk-1 /]$ /opt/contrail/bin/dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver
============================================
0000:3b:00.0 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e

Network devices using kernel driver
===================================
0000:01:00.0 'I350 Gigabit Network Connection' if=em3 drv=igb unused=vfio-pci 
0000:01:00.1 'I350 Gigabit Network Connection' if=em4 drv=igb unused=vfio-pci 
0000:19:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=em1 drv=i40e unused=vfio-pci 
0000:19:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=em2 drv=i40e unused=vfio-pci *Active*
0000:3b:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=p1p2 drv=i40e unused=vfio-pci 

Other network devices
=====================
<none>

lspi and lshw commands can also be use to check the number of NIC cards on a node and device name. The device below, which is on PCI address 3b:00.0 is bound to DPDK driver and the NIC name is not shown.

[heat-admin@overcloud-contraildpdk-1 ~]$ lspci | grep Ethern
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
19:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
19:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
3b:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

[root@overcloud-contraildpdk-1 heat-admin]# lshw -c network -businfo
Bus info          Device     Class          Description
=======================================================
pci@0000:01:00.0  em3        network        I350 Gigabit Network Connection
pci@0000:01:00.1  em4        network        I350 Gigabit Network Connection
pci@0000:19:00.0  em1        network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:19:00.1  em2        network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:3b:00.0             network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:3b:00.1  p1p2       network        Ethernet Controller X710 for 10GbE SFP+
                  vlan710    network        Ethernet interface
                  vlan730    network        Ethernet interface
                  vhost0     network        Ethernet interface
                  docker0    network        Ethernet interface
[root@overcloud-contraildpdk-1 heat-admin]#

Stop all vrouter containers

[root@overcloud-contraildpdk-1 heat-admin]# docker ps | grep contrail
057057e97d5d        192.168.24.1:8787/contrail/contrail-vrouter-agent:1908.47-rhel-queens        "/entrypoint.sh /u..."   2 months ago        Up 15 minutes                               contrail_vrouter_agent
c8f32071f7e5        192.168.24.1:8787/contrail/contrail-vrouter-agent-dpdk:1908.47-rhel-queens   "/entrypoint.sh /u..."   2 months ago        Up 15 minutes                               contrail-vrouter-agent-dpdk
30a59ab6c762        192.168.24.1:8787/contrail/contrail-nodemgr:1908.47-rhel-queens              "/entrypoint.sh /b..."   11 months ago       Up 15 minutes                               contrail_vrouter_agent_nodemgr
[root@overcloud-contraildpdk-1 heat-admin]# docker stop contrail_vrouter_agent
contrail_vrouter_agent
[root@overcloud-contraildpdk-1 heat-admin]# docker stop contrail-vrouter-agent-dpdk
contrail-vrouter-agent-dpdk
[root@overcloud-contraildpdk-1 heat-admin]# docker stop contrail_vrouter_agent_nodemgr
contrail_vrouter_agent_nodemgr

Locate dpdk_nic_bind.py script. Check script and check status. The NIC is not bound to any NIC driver after containers have been stopped:

[root@overcloud-contraildpdk-1 ~]# find / -name dpdk_nic_bind.py
/var/lib/docker/overlay2/88fe9e8a8590ac665f8ebab0301f57a4eb766a84c10139eec42217dd39545f21/diff/opt/contrail/bin/dpdk_nic_bind.py
/var/lib/docker/overlay2/b2b66fe1e5f8011eebbe1663b8c80f02f50fce9389a8339ffc79b9187cea8253/diff/opt/contrail/bin/dpdk_nic_bind.py
[root@overcloud-contraildpdk-1 ~]# cd /var/lib/docker/overlay2/88fe9e8a8590ac665f8ebab0301f57a4eb766a84c10139eec42217dd39545f21/diff

[root@overcloud-contraildpdk-1 diff]# opt/contrail/bin/dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:01:00.0 'I350 Gigabit Network Connection' if=em3 drv=igb unused=vfio-pci 
0000:01:00.1 'I350 Gigabit Network Connection' if=em4 drv=igb unused=vfio-pci 
0000:19:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=em1 drv=i40e unused=vfio-pci 
0000:19:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=em2 drv=i40e unused=vfio-pci *Active*
0000:3b:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=p1p2 drv=i40e unused=vfio-pci 

Other network devices
=====================
0000:3b:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci
[root@overcloud-contraildpdk-1 diff]#

Execute the command below to bind the NIC to the kernel default driver i40e:

[root@overcloud-contraildpdk-1 diff]# opt/contrail/bin/dpdk_nic_bind.py -e 3b:00.0
p1p1
[root@overcloud-contraildpdk-1 diff]#  opt/contrail/bin/dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:01:00.0 'I350 Gigabit Network Connection' if=em3 drv=igb unused=vfio-pci 
0000:01:00.1 'I350 Gigabit Network Connection' if=em4 drv=igb unused=vfio-pci 
0000:19:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=em1 drv=i40e unused=vfio-pci 
0000:19:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=em2 drv=i40e unused=vfio-pci *Active*
0000:3b:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=p1p1 drv=i40e unused=vfio-pci 
0000:3b:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=p1p2 drv=i40e unused=vfio-pci 

Other network devices
=====================
<none>
[root@overcloud-contraildpdk-1 diff]#

Use the ethtool utility to check the driver version and firmware version of the NIC:

[root@overcloud-contraildpdk-1 heat-admin]# ethtool -i p1p1
driver: i40e
version: 2.4.20-k
firmware-version: 6.10 0x800051a4 19.0.12
expansion-rom-version: 
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

[root@overcloud-contraildpdk-1 heat-admin]# modinfo i40e
filename:       /lib/modules/3.10.0-1160.25.1.el7.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
version:        2.4.20-k
license:        GPL v2
description:    Intel(R) Ethernet Connection XL710 Network Driver
author:         Intel Corporation, <[email protected]>
retpoline:      Y
rhelversion:    7.9
srcversion:     3CBB6A60ECD75BC75264662

We can check dpdk version from contrail-vrouter-dpdk logs, here we aare using dpdk version "DPDK 18.05.1"

[root@overcloud-contraildpdk-1 diff]# grep -i dpdk -C1 /var/log/containers/contrail/contrail-vrouter-dpdk.log  | more
2021-07-01 16:36:22,831 VROUTER: vRouter version: {"build-info": [{"build-time": "2019-08-21 05:31:36.552268", "build-hostname": "rhel-7-builder-juniper-contrail-ci
-0000280663", "build-user": "zuul", "build-version": "1908"}]}
2021-07-01 16:36:22,831 VROUTER: DPDK version: DPDK 18.05.1
2021-07-01 16:36:23,268 VROUTER: Log file : /var/log/contrail/contrail-vrouter-dpdk.log
2021-07-01 16:36:23,268 VROUTER: Bridge Table limit:          262144

DPDK forum maintains supported NIC firmware and kernel driver i40e versions per DPDK version in the below link for intel NICs, we can verify the required NIC firmware version per release of DPDK.

Note: DPDK honors some level of forward and backward compatibility but in case of major mismatches DPDK would fail to recognize and bind the NIC.

https://doc.dpdk.org/guides/nics/i40e.html
```
Example
DPDK version     Kernel driver version     Firmware version

21.02 2.14.13 8.00 20.11 2.14.13 8.00 20.08 2.12.6 7.30 20.05 2.11.27 7.30 20.02 2.10.19 7.20 19.11 2.9.21 7.00 19.08 2.8.43 7.00 19.05 2.7.29 6.80 19.02 2.7.26 6.80 18.11 2.4.6 6.01 18.08 2.4.6 6.01 18.05 2.4.6 6.01 18.02 2.4.3 6.01 17.11 2.1.26 6.01
```
In case NIC firmware is upgraded and DPDK vrouter is failing to be up and you find the upgraded firmware version is not matching the table on DPDK forum you will need to downgrade the NIC firmware as peer supportability for DPDK to start working on the given compute node.

Modification History

2022-09-02: Initial publication.

Related Information