Consider a requirement when you want the Anycast Gateways (AGWs) to be extended across two data centers (DCs) by using MPLS data center interconnects (DCI) in order to ensure high availability across two geographic locations.
As part of this requirement, two DCs were deployed with EVPN/VXLAN, and these DCs were interconnected over MPLS DCI by using EVPN for MPLS logical-tunnel stitching (lt-stitching).
However, when AGWs with the same IP address were added across the MPLS DCI, several issues emerged. These issues included network slowdowns and outages when failover attempts were made, resulting in the isolation of DC subnets. ARP-related problems were also identified.
Detailed information regarding these issues can be found in the Symptoms section, along with their respective causes.
To achieve this requirement without encountering the specified issues, the recommended solution is to opt for seamless EVPN-VXLAN with EVPN-MPLS stitching by using interconnected-vlan-list and encapsulation MPLS.
In accordance with the documentation on lt-stitching, it is designed to establish connections between two data centers (DCs) while utilizing distinct virtual-gw-IP-addresses (VGA). Specifically, VGA should be x.x.x.x on DC1 and y.y.y.y on DC2.
It's worth noting that the documentation does not cover the scenario wherein the VGA is assigned x.x.x.x in both DC1 and DC2. This configuration would ideally provide High Availability for AGWs across the Data Center Interconnect (DCI), facilitating end-host communication between host1 and host2. However, implementing the same IP address across the DCI using lt-stitching presents a host of issues, including suboptimal configuration leading to potential unicast flooding.
When setting up a VGA IP address on an Integrated Routing and Bridging (IRB) interface, the Layer 3 gateway automatically generates an IPv4 media MAC address of 00:00:5E:00:01:01. However, the spines fail to advertise local VGA information within BGP's mpls-evpn switch-instance when the same VGA IP address is employed across all spines in both DC1 and DC2.
user@dc2-spine1# run show route advertising-protocol bgp 5.5.5.5 table mpls-evpn.evpn.0 mpls-evpn.evpn.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path 2:10.42.1.13:1::101::2c:6b:f5:07:7c:f0/304 MAC/IP * Self 100 I 2:10.42.1.13:1::101::dc:22:dc:22:dc:22/304 MAC/IP * Self 100 I 2:10.42.1.13:1::102::2c:6b:f5:07:7c:f0/304 MAC/IP * Self 100 I 2:10.42.1.13:1::102::dc:22:dc:22:dc:22/304 MAC/IP * Self 100 I 2:10.42.1.13:1::101::2c:6b:f5:07:7c:f0::10.41.101.248/304 MAC/IP * Self 100 I 2:10.42.1.13:1::101::dc:22:dc:22:dc:22::10.41.101.4/304 MAC/IP * Self 100 I 2:10.42.1.13:1::102::2c:6b:f5:07:7c:f0::10.41.102.248/304 MAC/IP * Self 100 I 2:10.42.1.13:1::102::dc:22:dc:22:dc:22::10.41.102.4/304 MAC/IP * Self 100 I 3:10.42.1.13:1::101::10.42.1.13/248 IM * Self 100 I 3:10.42.1.13:1::102::10.42.1.13/248 IM * Self 100 I
The output from "show route advertising-protocol bgp 5.5.5.5 table mpls-evpn.evpn.0" highlights the absence of a Type 2 EVPN route for the MAC address 00:00:5E:00:01:01. This anomaly is not observed when using different VGA IP addresses. The lack of a backup route for the VGA obstructs the achievement of High Availability over the DCI.
Furthermore, during a spine deactivation/reactivation of IRB, the local VGA information is advertised in BGP's mpls-evpn switch-instance. A PR was initiated to comprehend this behavior, eventually confirming the incompatibility of this requirement with the lt-stitching approach.
It was observed that when configuring a static VGA MAC address for the AGWs, the Type 2 route for the static MAC address does indeed get advertised in BGP. However, this introduces its own set of problems, which will be further discussed.
The Layer 3 gateway doesn't incorporate the automatically generated virtual MAC address (00:00:5E:00:01:01) as the source MAC address in the packets it generates. Instead, the device includes the IRB MAC address in two ways:
In data packets
In the source MAC address field within the outer Ethernet header of:
Address Resolution Protocol (ARP) replies
Neighbor advertisement packets
The implication of this behavior in our context is evident during a failover scenario when we simulate failover by deactivating the IRBs on the spines of, let's say, DC1. Subsequently, the EVPN database of DC1's leaf is purged of the auto-generated VGA destination MAC address (00:00:5E:00:01:01).
The traffic originating from Host1 and destined for Host2, which necessitates routing, is then unintentionally flooded as unknown unicast within DC1. Moreover, any future ARP requests for the gateway from the hosts result in DC2 spines sending ARP replies sourced from the IRB MAC address rather than the VGA MAC address. Consequently, the leaf remains unaware of the VGA destination MAC address (00:00:5E:00:01:01) information from the DC2 spines.
While inter-vlan pinging from Host1 to Host2 would still function, this operation would be marred by unicast floods within the network, rendering it suboptimal and subsequently leading to network slowdown.
Normal Scenario
user@Leaf-1_re# run show evpn database Instance: vxlan-evpn VLAN DomainId MAC address Active source Timestamp IP address 101 00:00:5E:00:01:01 05:00:00:01:2C:00:00:00:65:00 Jan 04 22:54:06 10.41.101.254 101 2C:6B:F5:07:7C:F0 10.41.1.1 Jan 04 22:54:18 10.41.101.248 101 2C:6B:F5:A7:03:F0 10.41.1.1 Jan 04 22:54:06 10.41.101.250 101 DC:11:DC:11:DC:11 ge-0/0/0.0 Jan 04 22:54:24 10.41.101.1 102 00:00:5E:00:01:01 05:00:00:01:2C:00:00:00:66:00 Jan 04 22:54:06 10.41.102.254 102 2C:6B:F5:07:7C:F0 10.41.1.1 Jan 04 22:54:18 10.41.102.248 102 2C:6B:F5:A7:03:F0 10.41.1.1 Jan 04 22:54:06 10.41.102.250 102 DC:11:DC:11:DC:11 ge-0/0/0.0 Jan 04 22:54:24 10.41.102.1 102 DC:22:DC:22:DC:22 10.41.1.1 Jan 04 22:45:49
After deactivating IRBs on Spine1
user@Leaf-1_re# run show evpn database Instance: vxlan-evpn VLAN DomainId MAC address Active source Timestamp IP address 101 2c:6b:f5:07:7c:f0 10.41.1.1 Jan 04 22:59:25 10.41.101.248 101 dc:11:dc:11:dc:11 ge-0/0/0.0 Jan 04 22:59:24 10.41.101.1 102 2c:6b:f5:07:7c:f0 10.41.1.1 Jan 04 22:59:25 10.41.102.248 102 dc:11:dc:11:dc:11 ge-0/0/0.0 Jan 04 22:59:24 10.41.102.1
This scenario presents a variation of the problem elucidated in the document discussing MAC Addresses for a default VGA within an EVPN-VXLAN or EVPN-MPLS Overlay Network. For an in-depth understanding of this matter, refer to Understanding the MAC Addresses For a Default Virtual Gateway in an EVPN-VXLAN or EVPN-MPLS Overlay Network.
In this overlay network, when host 1 initiates an ARP request, irb.1 on the MX Series router becomes engaged. Within its ARP reply, the MX Series router incorporates the following:
Source MAC address in the outer Ethernet header: 00:05:85:00:53:01 (IRB MAC) → the intermediary Layer 2 switch EX1 learns this MAC address.
Sender MAC address within the ARP reply packet: 00:00:5E:00:01:01 (virtual MAC) → the intermediary Layer 2 switch EX1 remains unaware of this MAC address and therefore does not learn it.
The solution described in this document involves the use of either virtual-gateway-v4-mac or virtual-gateway-v6-mac. When configuring static VGA-MAC statements, this solution accomplishes two key things:
The configured static virtual MAC address supersedes the automatically generated virtual MAC address.
When the Layer 3 gateway MX1 sends data packets, ARP replies, and neighbour advertisement packets, it employs the statically configured virtual MAC address in the outer Ethernet header of these packets. Consequently, the intermediary Layer 2 switch EX1 also acquires knowledge of the statically configured virtual MAC address. This prevents the possibility of the switch flooding the domain with unknown unicast packets.
However, when we attempted to implement this solution, it led to a partial outage where one of the data centers became isolated. We applied static VGA-MAC statements in our lab settings, configuring the datacenters in two distinct ways:
Configuring the same static VGA MAC address on all the spines of both DC1 and DC2
Configuring a unique static VGA MAC address across the data centers while maintaining the same MAC address within each data center
Subsequently, we will delve into how both approaches resulted in a partial outage and introduced ARP-related issues. As a consequence, the implementation of static VGA MAC addresses could not serve as a viable solution for the customer's requirements.
When we configure the same static VGA MAC address on the spines of both DC1 and DC2, such as using aa.aa.aa.aa.aa.aa for IRB.101 and deadbeefdead for IRB.102, we observed that the VGA IP addresses/MAC address were indeed being advertised by the spines in BGP's mpls-evpn switch-instance. However, a significant issue emerged where all traffic requiring routing was dropped. Allow me to explain this in detail.Imagine that we have Host1 with IP address 10.41.101.1 and MAC address dc:11:dc:11:dc:11, requiring communication with Host 2 with IP address 10.41.102.4 and MAC address dc:22:dc:22:dc:22. This situation involves inter-vlan traffic, leading to routing. Consequently, routing will be handled by dc1-spine-1, following the asymmetric routing model.For any inter-vlan traffic that needs to traverse the Data Center Interconnect (DCI), dc1-spine-1 will take charge of packet routing. The routed traffic will be sourced by using the statically configured VGA MAC address, present both locally on dc1-spine-1 and remotely on dc2-spine1. However, the microcode within the Packet Forwarding Engine (PFE) of the line card carries out a check under this circumstance, resulting in packet drops. This drop occurs due to the locally configured VGA MAC address being identical, causing the microcode to classify the packets as a form of spoofed packet. This scenario is what led to the isolation of the data center.
This approach becomes unfeasible due to two scenarios that trigger ARP-related issues:
ARP requests flooding across DCI: ARP requests for the VGA MAC address from a host flood across the Data Center Interconnect (DCI). This means that both the local spine and the remote spine receive the ARP request. Consequently, both respond with their respective unique static VGA MAC addresses in the ARP replies. The issue arises when the host selects the ARP reply for the VGA MAC address from the remote spine. Since the remote spine's ARP reply is generally the last received, it's prioritized.
ARP entry clearance during local AnycastGW unavailability: During failover scenarios when the local Anycast Gateway (AnycastGW) becomes unavailable, the ARP entry for the AnycastGW MAC address on a host clears only when it ages out. This leads to a situation where traffic from the host, destined for the unavailable AnycastGW MAC address, eventually faces drops until the ARP entry ages out.
In summary, attempting to configure a unique static VGA MAC address across different data centers while maintaining consistency within each data center introduces these complex ARP-related challenges.
Seamless EVPN-VXLAN with EVPN-MPLS stitching using interconnected-vlan-list and encapsulation mpls is the recommended solution.
It provides HA across the DCI.
This feature is VLAN-based and includes a VLAN-aware bundle and VLAN bundle support by using a VLAN list.
It is a cleaner and more scalable approach – The MPLS routing-instance is no longer required, as well as the LT interfaces.
The addition of interconnected-vlan-list and encapsulation mpls is part of Junos OS Release 21.4R1
user@dc1-spine1# show routing-instances vxlan-evpn instance-type virtual-switch; protocols { evpn { encapsulation vxlan; default-gateway no-gateway-community; extended-vni-list [ 101 102 ]; interconnect { vrf-target target:1:2; route-distinguisher 100:110; esi { 00:00:00:22:22:22:22:22:22:22; all-active; } interconnected-vlan-list [ 101 102 ]; encapsulation mpls; } } } vtep-source-interface lo0.0; bridge-domains { bd-101 { vlan-id 101; routing-interface irb.101; vxlan { vni 101; } } bd-102 { vlan-id 102; routing-interface irb.102; vxlan { vni 102; } } } route-distinguisher 10.41.1.1:12; vrf-target target:1:1; user@dc2-spine1# show routing-instances vxlan-evpn instance-type virtual-switch; protocols { evpn { encapsulation vxlan; default-gateway no-gateway-community; extended-vni-list [ 101 102 ]; interconnect { vrf-target target:1:2; route-distinguisher 100:120; esi { 00:00:00:33:33:33:33:33:33:33; all-active; } interconnected-vlan-list [ 101 102 ]; encapsulation mpls; } } } vtep-source-interface lo0.0; bridge-domains { bd-101 { vlan-id 101; routing-interface irb.101; vxlan { vni 101; } } bd-102 { vlan-id 102; routing-interface irb.102; vxlan { vni 102; } } } route-distinguisher 10.42.1.13:12; vrf-target target:1:3; user@dc1-spine1# show interfaces irb unit 101 { proxy-macip-advertisement; virtual-gateway-accept-data; virtual-gateway-esi { 00:00:00:aa:aa:aa:aa:aa:aa:aa; all-active; } family inet { address 10.41.101.250/24 { virtual-gateway-address 10.41.101.254; } } } unit 102 { proxy-macip-advertisement; virtual-gateway-accept-data; virtual-gateway-esi { 00:00:00:bb:bb:bb:bb:bb:bb:bb; all-active; } family inet { address 10.41.102.250/24 { virtual-gateway-address 10.41.102.254; } } } user@dc2-spine1# show interfaces irb unit 101 { proxy-macip-advertisement; virtual-gateway-accept-data; virtual-gateway-esi { 00:00:00:cc:cc:cc:cc:cc:cc:cc; all-active; } family inet { address 10.41.101.248/24 { virtual-gateway-address 10.41.101.254; } } } unit 102 { proxy-macip-advertisement; virtual-gateway-accept-data; virtual-gateway-esi { 00:00:00:dd:dd:dd:dd:dd:dd:dd; all-active; } family inet { address 10.41.102.248/24 { virtual-gateway-address 10.41.102.254; } } }
How does this solution ensure high availability (HA)? First, all the spines advertise their respective local VGA MAC/IP information through BGP.
In the context of seamless stitching, dc1-spine1 is equipped with two entries in the EVPN database for the AnyCastGW - 10.41.101.254 and 10.41.102.254. One entry is locally originated, designated as Rank 1, whereas the other is received from the remote DC2 interconnect with ESI 00:00:00:33:33:33:33:33:33:33, marked as Rank 2.
The locally originated route with Rank 1 holds precedence and is both active and preferred. However, when the local IRB is deactivated, the VGA route with Rank 2 takes precedence, becoming active. This dynamic switching ensures the provision of AnyCastGW high availability over the DCI.
root@dc1-spine1> show evpn database mac-address 00:00:5e:00:01:01 extensive Instance: vxlan-evpn VN Identifier: 101, MAC address: 00:00:5e:00:01:01 State: 0x0 Source: 00:00:00:aa:aa:aa:aa:aa:aa:aa, Rank: 1, Status: Active Remote origin: 10.41.1.2 Remote state: <Mac-Only-Adv Pinned> Mobility sequence number: 0 (minimum origin address 10.41.1.1) Timestamp: Jan 17 05:55:39.858069 (0x63c6a8db) State: <Local-Virtual-Gateway Local-To-Remote-Adv-Allowed Remote-To-Local-Adv-Done Local-Pinned Remote-Pinned> MAC advertisement route status: Created Interconn advertisement route status: DCI route created IP address: 10.41.101.254 Flags: <Local-Adv> Local origin: irb.101 Remote origin: 10.41.1.2 Remote state: <Sent-to-l2ald> Interconn advertisement route status: DCI route created Source: 00:00:00:33:33:33:33:33:33:33, Rank: 2, Status: Inactive Remote origin: 10.42.1.13 Remote state: <Mac-Only-Adv Pinned Interconnect-DC> Remote origin: 10.42.1.14 Remote state: <Mac-Only-Adv Pinned Interconnect-DC> Mobility sequence number: 0 (minimum origin address 10.42.1.13) Timestamp: Jan 17 04:42:35.243955 (0x63c697bb) State: <Remote-Pinned> MAC advertisement route status: Not created (inactive source) IP address: 10.41.101.254 Flags: <Interconnect-DC> Remote origin: 10.42.1.13 Remote state: <Interconnect-DC> Remote origin: 10.42.1.14 Remote state: <Interconnect-DC> Interconn advertisement route status: Not created History db: <No entries> VN Identifier: 102, MAC address: 00:00:5e:00:01:01 State: 0x0 Source: 00:00:00:bb:bb:bb:bb:bb:bb:bb, Rank: 1, Status: Active Remote origin: 10.41.1.2 Remote state: <Mac-Only-Adv Pinned> Mobility sequence number: 0 (minimum origin address 10.41.1.1) Timestamp: Jan 17 05:55:39.857205 (0x63c6a8db) State: <Local-Virtual-Gateway Local-To-Remote-Adv-Allowed Remote-To-Local-Adv-Done Local-Pinned Remote-Pinned> MAC advertisement route status: Created Interconn advertisement route status: DCI route created IP address: 10.41.102.254 Flags: <Local-Adv> Local origin: irb.102 Remote origin: 10.41.1.2 Remote state: <Sent-to-l2ald> Interconn advertisement route status: DCI route created Source: 00:00:00:33:33:33:33:33:33:33, Rank: 2, Status: Inactive Remote origin: 10.42.1.13 Remote state: <Mac-Only-Adv Pinned Interconnect-DC> Remote origin: 10.42.1.14 Remote state: <Mac-Only-Adv Pinned Interconnect-DC> Mobility sequence number: 0 (minimum origin address 10.42.1.13) Timestamp: Jan 17 04:42:35.243354 (0x63c697bb) State: <Remote-Pinned> MAC advertisement route status: Not created (inactive source) IP address: 10.41.102.254 Flags: <Interconnect-DC> Remote origin: 10.42.1.13 Remote state: <Interconnect-DC> Remote origin: 10.42.1.14 Remote state: <Interconnect-DC> Interconn advertisement route status: Not created History db: <No entries>
When selecting a MAC from the EVPN database received from two sources, the following priorities are established:
A locally originated MAC is assigned the highest priority and is assigned Rank 1.
If a MAC address is received via BGP EVPN and carries the pinned flag, it is given the next level of priority.
In cases where both routes possess the pinned flag, the route originating from the source with the lowest source IP address is granted priority.
Next, we will explore how these rules impact certain setups. We will examine a scenario in which a data center has an IRB on only one spine, which results in asymmetric states on spines without IRB:
In DC1
dc1-spine1 has IRB.
The AGW MAC address is locally generated from irb.101/102 =>Local origin => Rank 1.
It receives an AGW pinned MAC address over I-ESI from dc2-spine1 => Rank 2.
dc1-spine2 has no IRB.
It receives an AGW pinned MAC address from dc1-spine1 => Rank 1.
This is because the source IP address of dc1-spine1 is lower than dc2-spine1.
In DC2
dc2-spine1 has IRB.
It receives an AGW pinned MAC address over I-ESI from dc1-spine1 => Rank 2.
dc2-spine2 has no IRB.
It receives an AGW pinned MAC address from dc2-spine1 => Rank 2.
It receives an AGW pinned MAC address over I-ESI from dc1-spine1 => Rank 1.
Thus, we observe that dc2-spine2 does not choose its local spine for l3-routing, unlike dc1-spine2.
We will revisit the same setup in which the data center has an IRB on only one spine, which results in asymmetric states on leaf switches.
dc1-spine1 advertises itself as Anycast GW with pinned flag.
dc1-spine2 re-advertises dc1-spine1’s BGP route with pinned flag. Since EBGP is used in the overlay with no-next-hop-change, adds its own AS to the BGP route.
For the dc1-leaf, dc1-spine1 is the only Anycast GW =>Rank 1.
dc2-spine1 advertises itself as Anycast GW with pinned flag.
dc2-spine2 regenerates I-ESI dc1-spine1’s BGP route without pinned flag.
It advertises its local I-ESI as Anycast GW.
2023-09-08: Initial version
2023-09-11: Made KB visible to all