Description

When vBNG is deployed on RHOSP13 cluster with vMX (performace-mode) SRIOV NIC, the vMX SRIOV interface is not connecting properly to the physical CPE (with single/double vlan-tags). 

In this example:
  • Physical CPE's are connected via Openstack SRIOV Network ( 'vbng-cpe-sriov' ) and coming to vMX SRIOV interface with stacked vlan [outer:3321 inner:100,10]
  • 2 CPEs configured to generate traffic with single vlan tag 100 & 101. Outer vlan is added by the L2 Switch which is connecting the CPEs to vBNG SRIOV interface.
  • DC Gateway is connected to vMX VirtIO interface via Openstack VirtIO network 'vbng-int-virtio' with single vlan tag 3320.

Symptoms

Topology:

DC Gateway<--->([vlan:3320]ge-0/0/2) MX (ge-0/0/3[vlan:3321.x)<--->(q-in-q)L2 switch<--->Physical CPE(IPoE/PPPoE Subs.)
  • ge-0/0/2  : Is configured in VirtIO mode connecting to Openstack network 'vbng-int-virtio' (vlan:3320, connected via openstack's internal OVS) 
  • ge-0/0/3 : Is configured in SRIOV mode connecting to Openstack network 'vbng-cpe-sriov' (vlan:3321, connected via VF function of SRIOV NIC port)

Note:

Physical CPEs are generating traffic with single tag (vlan 100,101) & L2-Switch is adding Outer tag 3321. So When traffic reaches to vMX SRIOV port, it's q-in-q (stacked vlan) for IPoE/PPPoE Subscriber.   

Configuration:

interfaces {
    ge-0/0/2 {
        inactive: hierarchical-scheduler maximum-hierarchy-levels 2;
        flexible-vlan-tagging;
        speed 10g;
        unit 3320 {
            vlan-id 3320;
            family inet {
                address 10.0.31.10/28;
            }
            family inet6 {
                address 2004:2004::1/64;
            }
        }
    }
    ge-0/0/3 {
        inactive: hierarchical-scheduler maximum-hierarchy-levels 2;
        flexible-vlan-tagging;
        vlan-offload;
        speed 10g;
        auto-configure {
            stacked-vlan-ranges {
                dynamic-profile AUTO-VLAN-STACK {
                    accept [ dhcp-v4 dhcp-v6 ];
                    ranges {
                        3000-4000,any;
                    }
                }
            }
        }
        encapsulation flexible-ethernet-services;
    }
}

Core Interface ge-0/0/2 (connecting to DC Gateway) is working fine with virtio (openstack network 'vbng-int-virtio')

root@vmx-v1> show route 10.0.31.1

inet.0: 27 destinations, 27 routes (27 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.31.0/28       *[Direct/0] 5d 01:56:18
                    >  via ge-0/0/2.3320

root@vmx-v1> show ospf neighbor
Address          Interface              State           ID               Pri  Dead
10.0.31.1        ge-0/0/2.3320          Full            10.240.32.128      1    31

root@vmx-v1> ping 10.0.31.1
PING 10.0.31.1 (10.0.31.1): 56 data bytes
64 bytes from 10.0.31.1: icmp_seq=0 ttl=64 time=34.395 ms
64 bytes from 10.0.31.1: icmp_seq=1 ttl=64 time=16.894 ms
64 bytes from 10.0.31.1: icmp_seq=2 ttl=64 time=11.529 ms
^C
--- 10.0.31.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 11.529/20.939/34.395/9.763 ms


But with SRIOV interface (ge-0/0/3), there are No IPoE/PPPoE subscribers. The CPE is UP and vlans are properly configured in L2-Switch for the IPoE/PPPoE traffic to hit VMX SRIOV interface with 3321.x vlan tagging.
On further troubleshooting with packet capture on ge-0/0/3, there are no packets captured with 3321 Vlan ID on ge-0/0/3.

root@vmx-v1> show subscribers
Interface             IP Address/VLAN ID                      User Name                      LS:RI

root@vmx-v1>

Solution

The SRIOV NICs are installed with native Intel i40e driver instead of Juniper provided modified SRIOV driver. For Native Intel SRIOV Driver, use 'set interface <interface-name> vlan-offload' command to offload the VLAN filtering to unmodified PF (Intel stock) driver.

Note:  vMX was built using HEAT Template in RHOSP13 platform, and each compute node in RHOSP13 cluster has 2 Intel SRIOV NIC (XXV710 - 25GbE SFP28) p1p1 & p2p1 with 64 VFs per NIC.

[root@test-compute-1]# ethtool -i p2p1
driver: i40e
version: 2.3.2-k
firmware-version: 6.00 0x80003785 18.5.17
expansion-rom-version:
bus-info: 0000:5e:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[root@test-compute-1]# lspci -nn | grep Ether
01:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
01:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
19:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
19:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
5e:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)
5e:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)
5e:02.0 Ethernet controller [0200]: Intel Corporation Ethernet Virtual Function 700 Series [8086:154c] (rev 02)
. . .
5e:09.2 Ethernet controller [0200]: Intel Corporation Ethernet Virtual Function 700 Series [8086:154c] (rev 02)   <--SRIOV Port of vMX(ge-0/0/3)
. . .
5e:09.7 Ethernet controller [0200]: Intel Corporation Ethernet Virtual Function 700 Series [8086:154c] (rev 02)
d8:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)
d8:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)

[root@test-compute-1]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
. . .
8: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:bf:83:e0 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    . . .
    vf 58 MAC fa:16:3e:5f:0c:c5 , vlan 3321 , spoof checking off, link-state enable, trust off     <--SRIOV Port of vMX(ge-0/0/3)
    . . .
    vf 63 MAC c6:2d:13:33:87:2a, spoof checking off, link-state auto, trust off

[root@test-compute-1]# virsh dumpxml instance-000009ba
<domain type='kvm' id='29'>
  <name> instance-000009ba
  <uuid> $ABC123
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="17.0.7-2.el7ost"/>
      <nova:name> Instance_RSP_In_A_Box-vmx-Test_fpc0
    . . .
    <interface type='vhostuser'>
      <mac address='fa:16:3e:fa:9a:73'/>
      <source type='unix' path='/var/lib/vhost_sockets/vhua1c82c0a-d9' mode='server'/>
      <target dev='vhua1c82c0a-d9'/>
      <model type='virtio'/>
      <driver rx_queue_size='1024' tx_queue_size='1024'/>
      <alias name='net1'/>
     

   
    <interface type='hostdev' managed='yes'>
      <mac address='<strong>
      <driver name='<strong>                   <--SRIOV Interface ge-0/0/3
      <source>
       

     
      <vlan>
        <tag id='<strong>
     
      <alias name='hostdev1'/>
     

   
     . . .

To solve this problem, add the 'vlan-offload ' command under ge-0/0/3 (SRIOV) interface as below, then commit.

root@vmx-v1# set interfaces ge-0/0/3 ?
. . .
  vlan-offload         Offload VLAN filtering to NIC HW

[edit]
root@vmx-v1# set interfaces ge-0/0/3 vlan-offload
root@vmx-v1# commit and-quit


Final Configuration:

interfaces {
    ge-0/0/2 {
        inactive: hierarchical-scheduler maximum-hierarchy-levels 2;
        flexible-vlan-tagging;
        speed 10g;
        unit 3320 {
            vlan-id 3320;
            family inet {
                address 10.0.31.10/28;
            }
            family inet6 {
                address 2004:2004::1/64;
            }
        }
    }
    ge-0/0/3 {
        inactive: hierarchical-scheduler maximum-hierarchy-levels 2;
        flexible-vlan-tagging;
        vlan-offload;
        speed 10g;
        auto-configure {
            stacked-vlan-ranges {
                dynamic-profile AUTO-VLAN-STACK {
                    accept [ dhcp-v4 dhcp-v6 ];
                    ranges {
                        3000-4000,any;
                    }
                }
            }
        }
        encapsulation flexible-ethernet-services;
    }
    lo0 {
        unit 0 {
            family inet {
                address 10.1.100.1/32;
            }
            family inet6 {
                address 2004::1/128;
            }
        }
    }
}

root@vmx-v1> show route 10.0.31.1

inet.0: 27 destinations, 27 routes (27 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.31.0/28       *[Direct/0] 5d 02:19:34
                    >  via ge-0/0/2.3320

root@vmx-v1> show ospf neighbor
Address          Interface              State           ID               Pri  Dead
10.0.31.1        ge-0/0/2.3320          Full            10.240.32.128      1    33

root@vmx-v1> ping 10.0.31.1
PING 10.0.31.1 (10.0.31.1): 56 data bytes
64 bytes from 10.0.31.1: icmp_seq=0 ttl=64 time=73.270 ms
64 bytes from 10.0.31.1: icmp_seq=1 ttl=64 time=4.010 ms
^C
--- 10.0.31.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 4.010/38.640/73.270/34.630 ms

root@vmx-v1> show subscribers
Interface             IP Address/VLAN ID                      User Name                      LS:RI
demux0.3221225472     0x8100.3321 0x8100.100                                            default:default
demux0.3221225476     0x8100.3321 0x8100.101                                            default:default

root@vmx-v1> show subscribers extensive
Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221225472
Interface type: Dynamic
Underlying Interface: ge-0/0/3
Dynamic Profile Name: AUTO-VLAN-STACK
Dynamic Profile Version: 1
State: Active
Session ID: 1
PFE Flow ID: 13
Stacked VLAN Id: 0x8100.3321
VLAN Id: 0x8100.100
Login Time: 2021-01-20 00:01:38 UTC

Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221225476
Interface type: Dynamic
Underlying Interface: ge-0/0/3
Dynamic Profile Name: AUTO-VLAN-STACK
Dynamic Profile Version: 1
State: Active
Session ID: 5
PFE Flow ID: 19
Stacked VLAN Id: 0x8100.3321
VLAN Id: 0x8100.101
Login Time: 2021-01-20 05:24:12 UTC