SSH/TCP traffic fails in-between virtual machines hosted on different compute nodes due to checksum errors.

SSH/TCP traffic fails in-between:

• Virtual machines (VMs) on different compute nodes
• Compute nodes and the MX Gateway router

The following is a simple use case to demonstrate the issue:

You might observe the following:

The SSH traffic from vm1 to vm2 and vm3 to vm4 passes through as the traffic gets handled at the vRouter within the compute node, and won't pass through the physical NIC of the compute node as the VM pair is hosted on the same compute node.

However, SSH traffic from vm1 to vm3 and vm2 to vm4 has to pass through the physical NIC of the compute nodes. In the case of SSH traffic passing through the physical node, cases are observed in which the traffic drops.

This is mostly observed (but not limited to) the following scenarios:

• Nested VMs
• VMs running in ESXi Hypervisor
• NIC cards that do not support checksum and generate an incorrect checksum (e.g., Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe NIC cards).

Check if there are any incorrect checksum errors on the host and the virtual machine using the following command:

If you observe any errors as the output of the above command, you can turn off the tx-checksumming parameter. In Contrail, the vrouter makes use of the NIC cards to perform checksum offload for the inner packet (after the header has been added). Some NIC cards do not perform this operation correctly. In such cases, it is advisable to make a note of the driver and the hardware being used before turning off the tx-checksumming parameter. 

To get around this problem, it is advised to turn off the tx-checksumming parameter on the compute node data interface using the ethtool utility from the CLI.

By default, the ethtool utility is not installed on the servers. It must be installed on the compute nodes. 

If the OS is CentOS, it can be installed using the following command:

yum install ethtool

If the OS is Ubuntu, it can be installed using the following command:

sudo apt-get install ethtool

Once the ethtool utility is installed, perform the following steps to disable tx-checksumming:

1. Determine the interface used for data traffic by mapping the MAC address of vhost0 to the physical interface's MAC address on the compute node.
2. Enter the following command: ethtool -K eth1 tx off (assuming that eth1 is the data interface)
3. Repeat step 1 and step 2 on all compute nodes. 

You can then check the status of tx-checksumming using the ethtool -k <interface_name> command. The output indicates whether tx-checksumming is on or off.

Disabling tx-checksumming should allow the SSH/TCP traffic (in the failed use cases) to be allowed without any problems.