ISSUE: Unexpected BFD/BGP flaps causing network outage

ISSUE TRIGGER: No recent changes made to the environment. QFX5120 were upgraded to 23.4R2-S2.1 on December 30th

ISSUE IMPACT: BFD/BGP flaps causing network outage

ENVIRONMENT: Production

MIST/APSTRA MANAGED?: MIST managed

DEVICES:

DISTRIBUTION (Leaf) AND CORE (Spine) SWITCHES

Model: qfx5120-48y-8c

Junos: 23.4R2-S2.1

TOPOLOGY: (attached to the case)

Endpoint Switches < --- > LEAF 0/1 < --- > SPINE 0/1

TROUBLESHOOTING SUMMARY:

1. Customer is running EVPN/VXLAN ERB setup where unexpected BFD/BGP overlay/underlay flaps seen in leafs (0/1) and spines (0/1) causing network outages.

2. From analyzing the logs, sessions are being torn down due to BFD hello packets exchanges are not being received on time between the peers, leading to hello timer expiring and eventually state changes to down with hard reset.

3. No cpu processes or memory spikes seen in both the switches.

4. There were L3NHOP, TTL and L3MTU DDOS violations being set and cleared consistently in both the switches. These violations looks like a side effect of the BFD/BGP flaps.

5. Customer worked with L2 JTAC engineer initially and based on the JTAC engineer suggestion, they changed the ARP DDOS threshold to 50000 as customer was seeing multiple ARP violations. BFP/BGP flaps did not get resolved post this change.

6. Upon reviewing the configuration and logs, we noticed DDOS threshold for ARP (50000), VXLAN (50000) and DHCP (10000) were set to high values.

7. Performed ukern dump for VXLAN, ARP and DHCP queues which showed lot of ARP request packets along with DHCP broadcast packets. Monitor irb interfaces of wireless vlan, wired vlan and server vlan showed multiple ARP request packets with the majority of the wireless client vlan.

Customer made some changes to the wireless controller and loadbalancer to curb/remove the offender IP's. Number of ARP and VXLAN packets processed by leaf and spines were still on the higher side.

8. Suggested customer to review the offender IP's and work on to remove those IP's from the network and update the DDOS threshold values to an acceptable lesser range which currently is on higher side which is suspected to be causing this BFP/BGP flap and network instability.

9. Customer updated the VXLAN and ARP DDOS threshold values to 5000. Post this change, we noticed one BGP flap with BGP DDOS violation. Ukern dump for BGP showed legitimate BGP packets.

10. Customer will further worked to make DHCP changes to the network by removing the DHCP network traffic offender IPs, and re-enabling BFD on the leaf and spine switches. Following these changes, BFD has remained stable.

Random and frequent BFD flaps in the network

Customer made DHCP changes to the network by removing the DHCP network traffic offender IPs, and re-enabling BFD on the leaf and spine switches. Following these changes, BFD has remained stable.