Description

In a contrail cluster, tenants often want jumbo non-fragmented packets to pass from end to end due to the requirement of certain applications. This requires all network interfaces along the L3 forwarding path to have a proper MTU value (for example, 9160) configured to support jumbo frames. This article describes, for contrail version 3.2.x and 3.0.x, how to change the default MTU size from 1500 to 9160 on a DPDK compute node so that jumbo PING can succeed without fragmentation. We assume that all compute nodes, switches or routers along the forwarding path have already had MTU configured properly on the corresponding network interfaces. 

Symptoms

Tenant starts a jumbo non-fragmented PING from a VM to another VM located in a different compute node. The following error is observed: 

ubuntu@ubuntu-charging-dpdk:~$ ping 10.85.190.104 -M do -s 8000
PING 10.85.190.104 (10.85.190.104) 8000(8028) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500

​
This is because the local network interface has a default value of 1500 configured. 

ubuntu@ubuntu-charging-dpdk:~$ ifconfig ens3
ens3      Link encap:Ethernet  HWaddr 02:3d:d8:d5:6d:ca 
          inet addr:10.85.190.105  Bcast:10.85.190.127  Mask:255.255.255.224
          inet6 addr: fe80::3d:d8ff:fed5:6dca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:613 errors:0 dropped:0 overruns:0 frame:0
          TX packets:392 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:476148 (476.1 KB)  TX bytes:30904 (30.9 KB)

​
But even after increasing the MTU size to 9160, the PING still failed with a similar error message: 

ubuntu@ubuntu-charging-dpdk:~$ sudo ifconfig ens3 mtu 9160
ubuntu@ubuntu-charging-dpdk:~$ ifconfig ens3
ens3      Link encap:Ethernet  HWaddr 02:3d:d8:d5:6d:ca 
          inet addr:10.85.190.105  Bcast:10.85.190.127  Mask:255.255.255.224
          inet6 addr: fe80::3d:d8ff:fed5:6dca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9160  Metric:1
          RX packets:935 errors:0 dropped:0 overruns:0 frame:0
          TX packets:581 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:503536 (503.5 KB)  TX bytes:53018 (53.0 KB)

ubuntu@ubuntu-charging-dpdk:~$ ping 10.85.190.104 -M do -s 8000
PING 10.85.190.104 (10.85.190.104) 8000(8028) bytes of data.
ping: local error: Message too long, mtu=1454
ping: local error: Message too long, mtu=1454

Solution

From the compute node's prospective, we need to make sure physical interfaces in the underlay network supports jumbo packets with a proper MTU configured. In the following example, we have two physical interfaces p1p2 and p2p1 configured in LACP bundle bond0, where we need to set MTU value to 9160 on both NIC ports as well bond0, bond0. <vlan> and vhost0. Please note that a vrouter restart is necessary, i.e.,  service supervisor-vrouter restart, for DPDK compute to load MTU settings correctly. The command, 'ifconfig <eth> mtu <mtu_size> ' does not work for physical interfaces p1p2 or p2p1 which has DPDK driver instead of OS driver loaded.

Snippet before specifying MTU in /etc/network/interfaces file: 

root@bcomp80:/etc/network# grep bond0 -C2 /etc/network/interfaces
iface p2p1 inet manual
    down ip addr flush dev p2p1
    bond-master bond0

--
iface p1p2 inet manual
    down ip addr flush dev p1p2
    bond-master bond0

​auto bond0

iface bond0 inet manual
    hwaddress 14:02:ec:66:b8:dc
    down ip addr flush dev bond0
    bond-slaves none
    bond-miimon 100
-- 
auto bond0.722
iface bond0.722 inet manual
    vlan-raw-device bond0


Snippet after specifying large MTU values on p2p1, p1p2, vhost0, bond0 and bond0. <vlan> 

root@bcomp80:/etc/network# grep 9160 -C3 /etc/network/interfaces

iface p2p1 inet manual
    down ip addr flush dev p2p1
    bond-master bond0
    mtu 9160

auto p1p2
iface p1p2 inet manual
    down ip addr flush dev p1p2
    bond-master bond0
    mtu 9160

auto bond0
iface bond0 inet manual
--
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit_hash_policy layer3+4
    mtu 9160
   

auto bond0.722
iface bond0.722 inet manual
    vlan-raw-device bond0
    mtu 9160

auto vhost0
iface vhost0 inet static
--
    network_name contrail
    dns-nameservers 172.22.199.11
    post-up ip link set vhost0 address 14:02:ec:66:b8:dc
    mtu 9160


After vrouter restarts and loads the jumbo MTU on all those interfaces, jumbo packets PING test now passes from the tenant side: 

​ubuntu@ubuntu-charging-dpdk:~$ ping 10.85.190.104 -M do -s 8000
PING 10.85.190.104 (10.85.190.104) 8000(8028) bytes of data.
8008 bytes from 10.85.190.104: icmp_seq=1 ttl=64 time=2.34 ms
8008 bytes from 10.85.190.104: icmp_seq=2 ttl=64 time=0.724 ms
8008 bytes from 10.85.190.104: icmp_seq=3 ttl=64 time=0.540 ms
8008 bytes from 10.85.190.104: icmp_seq=4 ttl=64 time=0.494 ms
8008 bytes from 10.85.190.104: icmp_seq=5 ttl=64 time=0.557 ms
8008 bytes from 10.85.190.104: icmp_seq=6 ttl=64 time=0.667 ms
^C
--- 10.85.190.104 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5001ms
rtt min/avg/max/mdev = 0.494/0.887/2.341/0.655 ms