This article describes a Layer 2 loop that appears when two QFX leafs in a collapsed EVPN-VXLAN fabric are connected to a pair of Cisco Nexus switches running vPC, and the links are grouped into two separate ESI-LAGs instead of one.

Topology used in this article:

• Leaf-1 and Leaf-2: QFX5120-48Y, collapsed EVPN-VXLAN fabric, joined by Layer 3 links.
• Nexus-A and Nexus-B: Layer 2 top-of-rack switches in one vPC domain, joined by the vPC peer-link.
• Each leaf has one 25G link to Nexus-A and one 25G link to Nexus-B.

interfaces ae1 (member et-0/0/0, to Nexus-A)

esi <esi-id-1>, all-active

lacp system-id <lacp-system-id-1>

interfaces ae2 (member et-0/0/1, to Nexus-B)

esi <esi-id-2>, all-active

lacp system-id <lacp-system-id-2>

Both bundles are trunks carrying the same VLAN list.

• Constant interface flapping and loop messages as soon as a VLAN is allowed on both port-channels on the Cisco side and on both ae interfaces on the Juniper side.
• The network becomes stable again when the VLAN is pruned from one of the port-channels.
• Enabling RSTP on the ae interfaces puts ports into blocking state (BLK) and the links do not come up as expected.
• MAC addresses keep moving between the local bundle and the remote Ethernet segment:

vlan_name <vlan-name> mac <mac-address> was moved from ae1.0 to <interface> with flags: 0x2101f

vlan_name <vlan-name> mac <mac-address> was moved from ae1.0 to <interface> with flags: 0x2101f

vlan_name <vlan-name> mac <mac-address> was deleted from <interface> with flags: 0x1000

In this output, <interface> is the esi.x interface of the second Ethernet segment.

• show evpn instance extensive shows two different Ethernet segments toward the same neighbor pair, each with a single local member:

Interface name ESI Mode Status AC-Role

ae1.0 <esi-id-1> all-active Up Root

ae2.0 <esi-id-2> all-active Up Root

• show lacp interfaces shows the leaf presenting two different actor system identifiers to a single vPC partner:

LACP info: Role System System

priority identifier

et-0/0/0 Actor 127 <lacp-system-id-1>

et-0/0/0 Partner 32667 <mac-address>

et-0/0/1 Actor 127 <lacp-system-id-2>

et-0/0/1 Partner 32667 <mac-address>

• Both members see the same partner system identifier, which is the vPC.
• During the instability, bundle and BFD or BGP messages may also be seen in /var/log/messages:

lag_bundlestate_ifd_change: bundle ae2: bundle IFD minimum bandwidth or minimum links not met

LACP_INTF_MUX_STATE_CHANGED: ae2: et-0/0/1: Lacp state changed from COLLECTING_DISTRIBUTING to DETACHED

BFDD_STATE_UP_TO_DOWN: BFD Session <ip-address> (IFL 902) state Up -> Down

The two Nexus switches act as one logical switch through vPC, and their peer-link bridges the same VLANs. When the Juniper side presents two separate Ethernet segments, the same VLAN has two parallel Layer 2 paths:

fabric --> ae1 --> Nexus-A --> vPC peer-link --> Nexus-B --> ae2 --> fabric

EVPN-VXLAN does not run STP to block the second path, so broadcast and unknown unicast traffic circulates and MAC addresses oscillate between the two Ethernet segments.

Enabling RSTP blocks one of the paths, which confirms the loop but is not the correct fix: EVPN-VXLAN relies on split horizon and designated forwarder election, not on STP, so the link grouping itself has to be corrected.

Configuration change

Build a single ESI-LAG on each leaf facing the vPC pair. In the example below, ae1 is kept (it already carries the ESI, the LACP system identifier and the VLAN list) and the second member is moved into it. Apply the same change on both leafs:

delete interfaces et-0/0/1 ether-options 802.3ad

delete interfaces ae2

set interfaces et-0/0/1 ether-options 802.3ad ae1

set interfaces ae1 description to-nexus-vpc

commit confirmed 5

Use commit confirmed so the change is rolled back automatically if connectivity is lost during the cutover. On a fabric managed by an external controller, apply the change through the controller as well, so it is not overwritten on the next deploy.

On the Cisco side, the four links must be members of a single vPC port-channel with the same allowed VLANs.

Verification

After the change, each leaf has one bundle with both members, and both leafs present the same identifiers to the vPC:

user@Leaf-1> show lacp interfaces ae1 extensive

Aggregated interface: ae1

LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity

et-0/0/0 Actor No No Yes Yes Yes Yes Fast Active

et-0/0/1 Actor No No Yes Yes Yes Yes Fast Active

LACP protocol: Receive State Transmit State Mux State

et-0/0/0 Current Slow periodic Collecting distributing

et-0/0/1 Current Slow periodic Collecting distributing

LACP info: Role System System

priority identifier

et-0/0/0 Actor 127 <lacp-system-id-1>

et-0/0/0 Partner 32667 <mac-address>

et-0/0/1 Actor 127 <lacp-system-id-1>

et-0/0/1 Partner 32667 <mac-address>

One actor system identifier on both leafs, one partner system identifier for all members.

user@Leaf-1> show evpn instance extensive

Number of local interfaces: 5 (3 up)

Interface name ESI Mode Status AC-Role

ae1.0 <esi-id-1> all-active Up Root

ESI: <esi-id-1>

Status: Resolved by IFL ae1.0

Local interface: ae1.0, Status: Up/Forwarding

Number of remote PEs connected: 1

DF Election Algorithm: MOD based

Also confirm that MAC addresses are learned on the bundle and stop moving:

user@Leaf-1> show ethernet-switching mac-learning-log

Tue Sep 22 23:00:05 vlan_name <vlan-name> mac <mac-address> was learned on ae1.0 with flags: 0x2001f