This article is applicable to all QFX5K platforms.
When an interface under ECMP undergoes a flap within a short time interval (2 to 3 seconds), it can lead to mis-programming in the Packet Forwarding Engine (PFE) and the interface might not get programmed in the hardware. This will cause traffic congestion on the device as one of the interfaces will not pass any traffic through it.
Outgoing traffic through the affected interface might drop to zero pps. However, no issues will be reported at the control plane end.
Example
ae118,ae119, ae128,ae129, ae138, ae139, ae148, and ae149 are 8 LAGs on the device configured for ECMP.
All the AE interfaces in the device have a single child member interface.
In the problem state, the routing table and forwarding table will show the affected interface in the next hop.
Routing Table
user@device > show route 120.0.0.10/24 inet.0: 30041 destinations, 270096 routes (30040 active, 0 holddown, 30035 hidden) + = Active Route, - = Last Active, * = Both 120.0.0.0/24 *[BGP/170] 04:34:52, localpref 100, from 100.92.240.17 AS path: 16509 400 I, validation-state: unverified to 100.92.240.17 via ae118.0 to 100.92.240.19 via ae119.0 to 100.92.240.21 via ae128.0 to 100.92.240.23 via ae129.0 > to 100.92.240.25 via ae138.0 to 100.92.240.27 via ae139.0 to 100.92.240.29 via ae148.0 to 100.92.240.31 via ae149.0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Forwarding Table
user@device> show route forwarding-table destination 120.0.0.10/24 Routing table: default.inet Internet: Enabled protocols: Bridging, Destination Type RtRef Next hop Type Index NhRef Netif 120.0.0.0/24 user 0 ulst 524291 30004 indr 524286 2 100.92.240.17 ucst 1836 4 ae118.0 indr 524294 2 100.92.240.19 ucst 1845 4 ae119.0 indr 524287 2 100.92.240.21 ucst 1838 4 ae128.0 indr 524289 2 100.92.240.23 ucst 1839 4 ae129.0 indr 524288 2 100.92.240.25 ucst 1840 4 ae138.0 indr 524290 2 100.92.240.27 ucst 1841 4 ae139.0 indr 524295 2 100.92.240.29 ucst 1844 4 ae148.0 indr 524293 2 100.92.240.31 ucst 1843 4 ae149.0>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
However, at the hardware level, the interface will not be present under the ECMP group:
% cprod -A fpc0 -c ‘set dcbcm bcmshell "l3 defip show”’ 4048 1 120.0.0.10/24 00:00:00:00:00:00 200010 0 0 0 0 n (ECMP) % cprod -A fpc0 -c 'set dcbcm bcmshell "l3 multipath show 200010"' HW (unit 0) Multipath Egress Object 200010 Interfaces: 100038 100047 100040 100041 100042 100043 100046 >>>>>>>>>>>>>>>>> 7 entries out of 8 Reference count: 30003
This issue is more likely to happen in scaled scenarios. Due to frequent down and up events for the interface, programming fails in the PFE leading to installation failure on the affected interface in the ECMP group. This issue can be prevented by configuring a hold-down timer on the physical interfaces that are configured for ECMP.
Mitigation: To prevent this issue, configure a hold-down timer only for the up event for an interface. Below is the required configuration:
set interfaces et-0/0/10 hold-time up 2000 set interfaces et-0/0/10 hold-time down 0
You need to configure this only on the physical interface and only the up timer is required.
This configuration provides sufficient time for the PFE to program all interfaces correctly in the hardware, and thus prevent mis-programming.
There is no software fix because this is a timing issue between the AE member/child interface creation in the PFE and the individual next-hop addition to the ECMP next-hop. At the time of adding individual next-hops to the ECMP next-hop, the AE member/child interface is not created in the PFE.To fix this, from the software, you need to initiate/invoke the population of the ECMP next-hop again and again from the AE member/child interface. In the case of a scaled setup with multiple AE interfaces and multiple ECMP next-hops, it is an overhead to keep adding individual next-hops to the ECMP next-hop and it can cause performance impact. This can lead to packet loss in some cases if anything fails during the addition of a next-hop to the ECMP next-hop. You will need to track it and perform clean-up correctly. This will make the software heavily used during scaled setup/scenario.
This issue will occur when there is a single-member interface within the LAG. If the LAG interface has more than one member interface and only one of the member interfaces goes down and comes up without any flap of the LAG interface, then this issue will not be seen.
However, if the member interface flap causes a LAG flap because of minimum links configuration, then the issue can be seen.
2023-04-04: Added details when LAG has multiple members