This article explained one possibility that causes communication failure between hosts in one VLAN in DC-1 and their anycast gateway in the same VLAN in DC-2.

Customer reported communication failure between hosts in one VLAN in DC-1 and their anycast gateway in the same VLAN in DC-2.

>>> Customer topology looks like below.

Hosts --- (VLAN_1, L2) Leaf-1 --- Spine-1 --- Border-Leaf-1 --- (DCI) --- Border-Leaf-2 --- Spine-2 --- Leaf-2 (VLAN_1 and irb.1)

*** In customer's topology, one MAC address is configured for all anycast gateway (IRB) interfaces, for example, 00:1c:XX:00:00:01.

*** On Leaf-1 in DC-1, VLAN_1 is a pure L2 bridge domain with NO irb/GW interfaces configured. However, there are other VLANs with anycast irb/GW interfaces configured with the same MAC in other VRFs.

*** On Leaf-2 in DC-2, anycast gateway (irb.1) is configured for VLAN_1 as gateway of hosts in VLAN_1.

*** Communication between hosts on Leaf-1 (in DC-1) and that on Leaf-2 (in DC-2) is always fine.

>>> Workaround suggested by JTAC:

*** Change MAC address of irb.1 on Leaf-2 to a different one from 00:1c:XX:00:00:01.

*** Communication started working between hosts on Leaf-1 and anycast gateway on Leaf-2 after that.

>>> Explanation given by engineering team:

*** As per PR1878933 (confidential), engineer team confirmed that Leaf-1 installs the anycast MAC address on its PFE (my_station_tcam table) and sends all packets destined to this MAC to its RE (CPU) regardless the MAC address is for other VLANs/IRBs in VRFs.

*** As a result, packets sourced from host in DC-1 destined to its gateway in DC-2 will be punted to RE/CPU of DC-1 leaf node instead of being forwarded.

*** Anycast gateway needs to be deployed either across all Leaf/Border-Leaf nodes (with irb enabled as L3 GW) or none of them (with NO irb enables as L2 bridge). It will not work if we mix them.

Change MAC address of anycast gateway in DC-2 to a different one from that of in DC-1.