Description

In Multichassis link aggregation groups (MC-LAG) topology, traffic failure is seen due to ARP failure on one of the  devices.

This article describes the scenario when its occurs and provides a workaround to avoid it.

Symptoms

In active-active MC-LAG topology, traffic loss could be seen in certain topology and scenario due to ARP failure on one of the MC-LAG devices.

For example, MC-LAG interface down and up could create such a scenario.

(1)
No ARP learing initially
Example MC-LAG topology:
  • MC-LAG is enabled on QFX-A and QFX-B.
  • Each device has IRB interface.
  • QFX-A and QFX-B are connected to EX through MC-LAG interface.
  • On QFX-A and QFX-B, there is no ARP entry for EX initially.
(2)
ARP reply goes to QFX-B and sync through ICCP
In case QFX-B sends ARP request to EX(192.168.100.254), EX will send ARP reply to one of the MC-LAG interfaces.
In this example, EX sends ARP reply to QFX-B directly.
When QFX-B receives ARP reply from EX, it learns ARP entry on ae1 interface and QFX-A learns the same via ICCP.
The ping to EX would be succeeded from QFX-A as well.
--------------------
test@QFX-A> ping 192.168.100.254
PING 192.168.100.254 (192.168.100.254): 56 data bytes
64 bytes from 192.168.100.254: icmp_seq=0 ttl=64 time=23.164 ms
64 bytes from 192.168.100.254: icmp_seq=1 ttl=64 time=12.136 ms
--------------------
test@QFX-A> show arp
MAC Address       Address         Name                      Interface               Flags
84:b5:cc:ff:ff:00 192.168.100.254    192.168.100.254              irb.100 [ae1.0]         none

test@QFX-B> show arp
MAC Address       Address         Name                      Interface               Flags
84:b5:cc:ff:ff:00 192.168.100.254    192.168.100.254              irb.100 [ae1.0]         none

test@EX> show arp
MAC Address       Address         Name                      Interface               Flags
84:b5:99:cc:33:88 192.168.100.1      192.168.100.1                irb.100 [ae1.0]         none
--------------------

(3)
Disable mc-lag interface on QFX-A
As an issue scenario, remove cable of MC-LAG interface between QFX-A and EX.
Then the associated interface of ARP entry of EX would be changed from ae1 to interchassis link(ICL) interface, ae0 on QFX-A.
At that time, ping is still working fine from QFX-A to EX.
--------------------
test@QFX-A> ping 192.168.100.254
PING 192.168.100.254 (192.168.100.254): 56 data bytes
64 bytes from 192.168.100.254: icmp_seq=0 ttl=64 time=13.561 ms
64 bytes from 192.168.100.254: icmp_seq=1 ttl=64 time=11.976 ms
--------------------
test@QFX-A> show arp
MAC Address       Address         Name                      Interface               Flags
84:b5:cc:ff:ff:00 192.168.100.254    192.168.100.254              irb.100 [ae0.0]         none

test@QFX-B> show arp
MAC Address       Address         Name                      Interface               Flags
84:b5:cc:ff:ff:00 192.168.100.254    192.168.100.254              irb.100 [ae1.0]         none

test@EX> show arp
MAC Address       Address         Name                      Interface               Flags
84:b5:99:cc:33:88 192.168.100.1      192.168.100.1                irb.100 [ae1.0]         none
--------------------

(4)
Enable mc-lag interface on QFX-A and ARP reply goes to QFX-B but sync doesn’t happen
As a last step, insert the cable again of MC-LAG interface between QFX-A and EX.
Since QFX-A recognized its MC-LAG interface, ARP entry belonging to ICL(ae0) would be removed.
Then ARP failure could be seen on QFX-A, and cannot ping to EX.
 
--------------------
test@QFX-A> ping 192.168.100.254
PING 192.168.100.254 (192.168.100.254): 56 data bytes
^C
--- 192.168.100.254 ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
--------------------
test@QFX-A> show arp
MAC Address       Address         Name                      Interface               Flags
none

test@QFX-B> show arp
MAC Address       Address         Name                      Interface               Flags
84:b5:cc:ff:ff:00 192.168.100.254    192.168.100.254              irb.100 [ae1.0]         none

test@EX> show arp
MAC Address       Address         Name                      Interface               Flags
84:b5:99:cc:33:88 192.168.100.1      192.168.100.1                irb.100 [ae1.0]         none
--------------------

Solution

As an expected behavior, ARP reply cannot be passed through ICL, but QFX-A can get ARP entry via ICCP when QFX-B learns the entry.
However, in the issue scenario above, QFX-B already learned ARP entry of EX so QFX-A cannot get ARP entry via ICCP from QFX-B.
Therefore, if you clear ARP entry of EX on QFX-B, QFX-A can get ARP entry via ICCP when QFX-B newly learns it.

You can find the description below for ARP reply which cannot be passed through ICL.

Address Resolution Protocol Active-Active MC-LAG Support Methodology

Dynamic ARP resolution over the ICL interface is not supported. Consequently, incoming ARP replies on the ICL are discarded. However, ARP entries can be populated on the ICL interface through ICCP exchanges from a remote MC-LAG peer.

To overcome this behavior, you can have a VRRP in MC-LAG devices so that ARP sync will happen and can avoid traffic loss.
Here is the sample config and reference.

<QFX-A>
set interfaces irb unit 100 family inet address 192.168.100.1/24 vrrp-group 1 virtual-address 192.168.100.100

<QFX-B>
set interfaces irb unit 100 family inet address 192.168.100.2/24 vrrp-group 1 virtual-address 192.168.100.100

How to Configure MC-LAG Active-Active Bridge Domains

To ensure that Layer 3 operates properly, instead of dropping the Layer 3 packet, the VRRP backup attempts to perform routing functions if the packet is received on an MC-LAG. A VRRP backup sends and responds to Address Resolution Protocol (ARP) requests.

Modification History

.