This article explains a problem that could be encountered in an EVPN/VxLAN CRB deployment where Spine switches are the default gateway for hosts connected to the leaf switches.
In a CRB design, Leaf switches are configured with virtual gateway information such as Spine’s IP and MAC address to be used during ARP refresh process. Leaf switches act as a proxy for Spine switches by generating and responding to ARP messages on Spine’s behalf. Leaf switches are supposed to use Spine’s IP and MAC address in ARP requests and replies. But due to a software bug, Leaf switches will use their own IRB MAC address in ARP requests instead of virtual gateway MAC. The hosts (such as Linux) that use ARP requests to update their ARP tables will encounter this problem. Below is an example of a host that could be impacted by this problem.
As shown in the diagram below, a host is dual homed to two Leaf switches, Leaf1 and Leaf2. When host sends ARP request for the default gateway, receiving Leaf will send ARP reply with Spine’s IRB MAC, which is correct behavior. However, when Leaf switches send ARP requests to refresh their ARP tables, they incorrectly use their respective IRB MAC instead of Spine's VGW MAC. Host will update default gateway’s MAC address with either Leaf1 or Leaf2’s IRB MAC, say Leaf1 in this case. When host sends IP packet and if they are hashed to a Leaf2, they will be dropped.
Topology
+----------+----------+
| |
| Spine |
+----/----------\------+
/ \
+-------------/-----+ +--------\-----------+
| | | |
| Leaf1 | | Leaf2 |
+---\----------------+ +----/---------------+
\ /
\======== ESI ===========/
+---\--------------/--+
| Dual Homed |
| Host |
| VLAN=3111 |
+---------------------+
Configuration
Leaf1 and Leaf2 switches are configured with virtual gateway information
Leaf1 Configuration
The irb.3111 has following configuration:
unit 3111 {
virtual-gateway-accept-data;
description "** L3 interface for VL-3111 in RI-CONTRAIL-TENANT-UNDERLAY";
family inet {
address 100.98.156.6/25 {
preferred;
virtual-gateway-address 100.98.156.1;
}
virtual-gateway-v4-mac 00:a0:a0:a0:31:11; }
Leaf2 Configuration
address 100.98.156.7/25 {
virtual-gateway-v4-mac 00:a0:a0:a0:31:11;
Workaround
Either of the following changes can prevent the issue occurence
1. Configure IRB-MAC to be same as VGA-MAC (or)2. Configure VGA-IP greater than IRB-IP address
Below are the packet captures of ARP exchange with correct MAC address
1) Host sends ARP request
Frame 1: 42 bytes on wire (336 bits), 42 bytes captured (336 bits)
Ethernet II, Src: MS-NLB-PhysServer-32_08:c9:00:00:48 (02:28:c9:00:00:48), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request)
Hardware type: Ethernet (1)
Protocol type: IPv4 (0x0800)
Hardware size: 6
Protocol size: 4
Opcode: request (1)
Sender MAC address: MS-NLB-PhysServer-32_08:c9:00:00:48 (02:28:c9:00:00:48)
Sender IP address: 100.98.168.46
Target MAC address: 00:00:00_00:00:00 (00:00:00:00:00:00)
Target IP address: 100.98.168.1
2) Leaf1 responds with VGW MAC
Frame 2: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Ethernet II, Src: CompactData_a0:31:41 (00:a0:a0:a0:31:41), Dst: MS-NLB-PhysServer-32_08:c9:00:00:48 (02:28:c9:00:00:48)
Address Resolution Protocol (reply)
Opcode: reply (2)
Sender MAC address: CompactData_a0:31:41 (00:a0:a0:a0:31:41) <==== VGW MAC
Sender IP address: 100.98.168.1
Target MAC address: MS-NLB-PhysServer-32_08:c9:00:00:48 (02:28:c9:00:00:48)
Target IP address: 100.98.168.46
Later Leaf1 sends an ARP request to refresh its ARP table. The ARP request from TOR switch contains VGW IP address, but IRB MAC address. Host uses this MAC address to update its own ARP table.
Below is ARP refresh packet with incorrect source MAC causing Host to change its ARP table incorrectly
Frame 3: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Ethernet II, Src: JuniperNetwo_a1:ae:60 (88:e6:4b:a1:ae:60), Dst: MS-NLB-PhysServer-32_08:c9:00:00:48 (02:28:c9:00:00:48)
Sender MAC address: JuniperNetwo_a1:ae:60 (88:e6:4b:a1:ae:60) <==== IRB MAC
Host has changed default gateway’s MAC address with IRB MAC address.
[user@linux-host-1:~]$ site=xxx; for node in ${site}ccctrl-{0..2} ${site}cactrl-{0..2}; do echo "arp on node: $node" ; ssh $node 'arp | grep "\.1 "' ; done
arp on node: xxxx
100.98.156.1 ether 88:e6:4b:a1:81:60 C enp5s0
Following link specifies which releases have the fix for this issue.
https://prsearch.juniper.net/problemreport/PR1909786