This article explains why a mismatched state may be seen for LACP bundles on QFX5K devices by demonstrating the working of the core isolation feature with LACP bundles in a setup with EVPN configuration. It also suggests a couple of workarounds that could be used to prevent the mismatched LACP bundle state.
Note: The problem was observed in Junos OS release 17.3R3-S2.
Consider the following topology:
Spines 01 spine 02 | | | | | | Leaf 01 ---------------| ae | | | Host
In the above topology, Leaf 01 on the QFX5100 device is connected to two spines. When the leaf loses connectivity to both spines, the EVPN core isolation feature is triggered and the LACP bundles (ae) are brought down (moving the Mux state to “Waiting”) so that no traffic enters Leaf 01 (basically isolating the QFX5100 device).
Note: By default, the core isolation feature is enabled on QFX5100 devices.
However, if the LACP configuration in the leaf has the minimum-links configuration enabled, the leaf will keep the ae bundle up and in the "collecting and distributing" state, a behavior that indicates a mismatched LACP bundle state. For on the peer device (works fine) as shown below, the state is “Attached,” which means no traffic is entering the leaf.
show lacp interfaces Feb 11 17:36:50 Aggregated interface: ae1 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-0/0/4 CDN Actor No No Yes Yes No Yes Fast Active xe-0/0/4 CDN Partner No No No No Yes Yes Fast Active <<<<<<<<<<<<<<< xe-0/0/5 CDN Actor No No Yes Yes No Yes Fast Active xe-0/0/5 CDN Partner No No No No Yes Yes Fast Active xe-0/0/6 CDN Actor No No Yes Yes No Yes Fast Active xe-0/0/6 CDN Partner No No No No Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-0/0/4 Current Fast periodic Collecting distributing xe-0/0/5 Current Fast periodic Collecting distributing xe-0/0/6 Current Fast periodic Collecting distributing Aggregated interface: ae2 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-0/0/17 CDN Actor No No Yes Yes No Yes Fast Active xe-0/0/17 CDN Partner No No No No Yes Yes Fast Active xe-0/0/18 CDN Actor No No Yes Yes No Yes Fast Active xe-0/0/18 CDN Partner No No No No Yes Yes Fast Active xe-0/0/19 CDN Actor No No Yes Yes No Yes Fast Active xe-0/0/19 CDN Partner No No No No Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-0/0/17 Current Fast periodic Collecting distributing xe-0/0/18 Current Fast periodic Collecting distributing xe-0/0/19 Current Fast periodic Collecting distributing
LACP peer showing attached state to indicate no traffic entering the leaf
show lacp interfaces ae2 Aggregated interface: ae2 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-0/0/6 Actor No No No No Yes Yes Fast Active xe-0/0/6 Partner No No Yes Yes No Yes Fast Active xe-0/0/7 Actor No No No No Yes Yes Fast Active xe-0/0/7 Partner No No Yes Yes No Yes Fast Active xe-0/0/8 Actor No No No No Yes Yes Fast Active xe-0/0/8 Partner No No Yes Yes No Yes Fast Active xe-0/0/9 Actor No No No No Yes Yes Fast Active xe-0/0/9 Partner No No Yes Yes No Yes Fast Active xe-0/0/10 Actor No No No No Yes Yes Fast Active xe-0/0/10 Partner No No Yes Yes No Yes Fast Active xe-0/0/11 Actor No No No No Yes Yes Fast Active xe-0/0/11 Partner No No Yes Yes No Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-0/0/6 Current Fast periodic Attached xe-0/0/7 Current Fast periodic Attached xe-0/0/8 Current Fast periodic Attached
ae bundle configuration on both devices with minimum-links enabled
QFX
set interfaces ae1 esi 00:00:00:00:00:00:00:50:33:01 set interfaces ae1 esi all-active set interfaces ae1 aggregated-ether-options minimum-links 2 set interfaces ae1 aggregated-ether-options lacp active set interfaces ae1 aggregated-ether-options lacp periodic fast set interfaces ae1 aggregated-ether-options lacp system-id 01:01:01:01:01:01 set interfaces ae1 unit 0 family ethernet-switching interface-mode trunk set interfaces ae1 unit 0 family ethernet-switching vlan members V388
Peer device
set interfaces ae2 aggregated-ether-options lacp active set interfaces ae2 aggregated-ether-options lacp periodic fast set interfaces ae2 unit 0 family ethernet-switching interface-mode trunk set interfaces ae2 unit 0 family ethernet-switching vlan members V1141 set interfaces ae2 unit 0 family ethernet-switching vlan members V1135 set interfaces ae2 unit 0 family ethernet-switching vlan members V1115
The fix to this problem will be made available in a future Junos OS release.
Meanwhile, a couple of workarounds that can be used are as follows:
Delete interfaces ae1 aggregated-ether-options minimum-links
OR
set protocols evpn no-core-isolation