Event Summary:

• QFX-2 switch rebooted on 3rd Feb and recovered on its own
• Afterwards, NOC received multiple end-user complaints about slow browsing in addition to significant unusual traffic on connected links. It’s clear that this unusual traffic is a loop. 
• Couple of VLANs was found to be the source of the loop. At first, turning off two of the VLANs didn't work. 
• To restore the issue state, one 100G interface (part of the AE bundle) was turned down.
• TAC suggested turning the interface back ON during MW to examine the loop's live behaviour and the participating Macs. 
• Customer reporting loop again now with many end users complaints of browsing issues

Initial Findings:

• Problem seems to be triggered post QFX-2 switch reboot but not immediately
• QFX-2 reboot triggered programming issue in QFX-2 switch
• As per traffic flow during problematic duration, traffic entered AE1 QFX1 and forwarded to 4 different ae interfaces upstream
• As per the design, QFX2 AE2 AE3 should block this traffic but it is forwarding due to filter programming issue. As a result, they are experiencing high traffic in the network and are currently facing an issue.

Here we see the programming issue on QFX-2 post reboot..

Non-working QFX-2:

FPC0(QFX-2 vty)# show shim virtual evpn_filter    

EVPN is-config flag is TRUE

Default Filter Not Installed

BUM Traffic Block Filter ID: -1 -1 

The ports to block for bum traffic are none

EVPN Local Bias Filter Info:

EVPN SVP GPORT 0xb0000011 

Filter ID: -1

Higig Filter ID: -1

BUM bit Filter ID: -1

Higig BUM bit Filter ID: -1

EVPN IGMP Filter ID: -1

The Ports in the Bitmap are xe1,ce3-ce6,ce11-ce15

FPC0(MP-BPL-GP-QFX5120-32C-SW02 vty)# exit

Working QFX-1:

FPC0(QFX-1 vty)# show shim virtual evpn_filter    

EVPN is-config flag is TRUE

EVPN Default Filter Info:

EVPN Default Filter ID: 3129

EVPN Higig Default Filter ID: 3131

EVPN Default BUM bit Filter ID: 3133

EVPN Higig Default BUM bit Filter ID: 3135

EVPN Default Filter Ref Count: 1075

EVPN MCAST Default Filter ID: 3137

EVPN MCAST Default Ref Cnount: 1075

BUM Traffic Block Filter ID: -1 -1 

The ports to block for bum traffic are none

EVPN Local Bias Filter Info:

EVPN SVP GPORT 0xb000000f 

Filter ID: 6005

Higig Filter ID: 6007

BUM bit Filter ID: 6009

Higig BUM bit Filter ID: 6011

EVPN IGMP Filter ID: 6013

The Ports in the Bitmap are xe1,ce4,ce11-ce15

• We noticed TCAM memory issue on QFX-2
• Further we restart the FPC of the QFX-2, post that the filter are programmed.
• However, the EVPN filter still not programmed, and we noticed no space available TCAM memory the show filter hw groups output shows there is space in TCAM; however, we are noticing this logs
• Further we decided to delete the protect RE filter and restart the FPC and check the status. Customer removed the loopback RE filter and restarted the FPC to resolve the TCAM and filter programming issues. Post the FPC restart EVPN filter was programmed in the PFE as expected which filters out the broadcast EVPN traffic. End user issue was resolved after filter programming.
• QFX-2 switch generated dcpfe core dump when executing operational commands. Customer is currently unable to attach this core dump. JTAC decoded the DCPFE core dump and attempted to match the decoded backtrace to existing PRs. Some internal PRs match this core dump backtrace; however, they were closed as the issue was unreproducible in the JTAC lab, and no PR fix was implemented. Switch is currently running 21.4R3-S5.17, which is already end of engineering and scheduled for end of support in June of this year. Due to this, JTAC was unable to open a PR to engage the DEV team for further investigation and recommend upgrading the switches to the latest recommended code.

Application slowness and traffic loop

Restart FPC and delete RE loopback filter reprogrammed the EVPN filter which fixed the application slowness issue