BUM traffic loop is seen on mc-lag topology after rebooting one of mc-lag node and when it comes up.This behavior could be caused during linkup timing of ICL/ICCP interface and mc-ae interface, and init-delay-time on mc-ae interface could solve the issue if the same scenario in this KB was applied.
After one of mc-lag node was rebooted and it comes up, BUM traffic loop is seen.The issue is seen when mc-ae comes up first and then ICL/ICCP interface comes up.At that time, BUM traffic coming to mc-ae interface will be forwarded to ICL interface immediately and peer mc-lag node will forward that traffic back to the same mc-ae interface which peer has.Issue scenario:1) TopologyIn this mc-lag topology, ae10 is mc-ae interface and ae9 is sharing ICL and ICCP.Target QFX and peer mc-lag node are sharing ae10(mc-ae).2) When BUM traffic is passing through target device, target device will flood it on ICL. But peer device will discard it to avoid traffic loop.3) While rebooting target QFX node, BUM traffic is passing through peer node.4) After target QFX came up, BUM traffic loop is seen.
The issue is seen when ae10(mc-ae) comes up first, followed by the ae9(ICL/ICCP).Once ae10(mc-ae) received the traffic, target device will flood it in the same segment including ICL. But when ae9(ICL/ICCP) comes up late, peer mc-lag node won't be able to handle BUM traffic properly because it doesn't know if original traffic is coming from mc-ae which is sharing with. The BUM traffic loop could be seen until ICCP convergence was done.Increasing the value of init-delay-time to have a delay between ICL/ICCP and mc-ae would be a solution.
set protocols iccp peer X.X.X.X session-establishment-hold-time 420 set interfaces aeXX aggregated-ether-options mc-ae init-delay-time 320
*NOTE: On QFX and EX Series switches, the default session establishment hold time is 300 seconds. However, the session establishment time must be at least 100 seconds higher than the init delay time. You can optionally update the session establishment time to be 340 seconds and the init delay time to be 240 seconds.session-establishment-hold-timeThe init-delay-time on mc-ae will be initiated by lag_ifdconfig_change, you can find this log in syslog message.
Aug 22 06:23:45.941 2023 Switch /kernel: %KERN-6: lag_ifdconfig_change: IFD ae10 ifd_specific_flag 0x0 changed to 0x0 by process dcd pid 2161
Please check this log and if init-delay-time has enough delay till mc-ae interface comes up. In this example, ae10 came up just after child interfaces of ae10 turned up. This indicates a need to adjust init-delay time as there is a possibility that ae10 might come up before ae9(ICL/ICCP).
Aug 22 06:26:48.394 2023 Switch mib2d[2179]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 524, ifAdminStatus up(1), ifOperStatus up(1), ifName et-0/0/50 Aug 22 06:26:48.394 2023 Switch mib2d[2179]: %DAEMON-6-SNMP_TRAP_LINK_UP: ifIndex 533, ifAdminStatus up(1), ifOperStatus up(1), ifName et-0/0/50.0 Aug 22 06:26:48.395 2023 Switch mib2d[2179]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 525, ifAdminStatus up(1), ifOperStatus up(1), ifName et-0/0/51 Aug 22 06:26:48.395 2023 Switch mib2d[2179]: %DAEMON-6-SNMP_TRAP_LINK_UP: ifIndex 534, ifAdminStatus up(1), ifOperStatus up(1), ifName et-0/0/51.0 Aug 22 06:26:48.424 2023 Switch mib2d[2179]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 587, ifAdminStatus up(1), ifOperStatus up(1), ifName ae10 Aug 22 06:26:48.424 2023 Switch mib2d[2179]: %DAEMON-6-SNMP_TRAP_LINK_UP: ifIndex 623, ifAdminStatus up(1), ifOperStatus up(1), ifName ae10.0
In this output, child interfaces came up 180 sec after lag_ifdconfig_change. So you can adjust init-delay-time to 240 sec to avoid a BUM traffic loop.The time it takes for the physical interface to come up after a device reboot varies depending on the number of interfaces, device spec, software version and configuration.Therefore, it's necessary to customize while reviewing the logs on the actual device.