Description

This KB provides one possible cause of issue of traffic blackhole occurred on QFX5120 leaf nodes after several times of configuration changes (conducted either by CLI or scripts) on its downstream ports.

Symptoms

No MAC addresses of downstream clients are learned on the QFX5120-48Y leaf node. Bounce the downstream port does not help.

Upon tracing back logs there are several configuration add/delete on downstream aggregate interface (aeX.x) which is to add "vlan-id-list <vlan-id>" and delete them.

This activity caused VPORT stale entries on PFE so new entries stopped being learned.

PFE errors look like below will fill up "messages" file.

,,,"<131>2024-03-21T10:58:02+11:00 QFX5120-48Y_Leaf : %PFE-3: fpc0 0:_bcm_esw_stat_flex_attach_ingress_table_counters1:

,,,"<131>2024-03-21T10:58:02+11:00 QFX5120-48Y_Leaf : %PFE-3: fpc0 0:_bcm_esw_stat_flex_attach_ingress_table_counters1:


,,,"<131>2024-03-21T10:58:02+11:00 QFX5120-48Y_Leaf : %PFE-3: fpc0 Table:SVP_ATTRS_2 Has already allocated with index:158base 229 mode 1.First dealloc it

,,,"<131>2024-03-21T10:58:02+11:00 QFX5120-48Y_Leaf : %PFE-3: fpc0 Table:SVP_ATTRS_2 Has already allocated with index:158base 229 mode 1.First dealloc it


Moreover, VxLAN global error counters as below will also increment.
 

# cprod -A fpc0 -c show shim virtual error-counters

 

VXLAN GLOBAL ERROR COUNTERS

---------------------------

VPORT Lookup Failure              : 18950

Egress Interface Create Failure   : 49017

VPORT Delete Failure              : 20516             <<<<<<<<<<<

Egress Interface Delete Failure   : 106239

VPORT Stat Detach Failure         : 45               <<<<<<<<<<<

#times Long Route Installed       : 1

Dest UDP Port Install Success     : 1

Unknown Sip Install Passed        : 1

Venh without RVTEP ifl            : 1

VxLAN L3 MC Node Del req : 75387

VxLAN L3 MC Node Del lookup fail : 75387

Solution

1. Do NOT use “vlan-id-list” on QFX5120 leaf nodes for its SP-style ports either via CLI or automation scripts unless there is an actual Q-in-Q scenario. Use “vlan-id” instead.

2. Avoid using CLI command “clear bgp neighbors all”, “clear ethernet-switching table” or “clear ethernet-switching mac-ip-table”. Define specific neighbors or addresses when we have to use them.

Modification History

2024-06-03 Draft and publication