Customer reports intermittent connection between clients connected to an EVPN VXLAN setup, traffic drops are seen intermittently for a specific device while others work.
Customer may have exhausted its resources for type 5 tunnel NH entries on LEAF devices:
> show pfe vxlan nh-usage Slot 0 ------------------OVERLAY NH------------------ Overlay bank size : 8192. <<<< Current usage : Reserved entries : 1 ARP NDP/Type-5 tunnel NH entries : 8190 <<<< Available entries : 1 <<<< Expected usage : Reserved entries : 1 ARP NDP/Type-5 tunnel NH entries : 5637 Available entries : 2554 -----------------UNDERLAY NH----------------- Underlay bank size : 57344 Current usage : Reserved entries : 5 Underlay NH entries : 182 Available entries : 57157 Expected usage : Reserved entries : 5 NHs to CPU (IRB + L3 intf) entries : 26 NH ARPs of underlay L3 intfs : 52 Total tagged and access ports : 21 VTEP Flood NHs : 26 Available entries : 57214
And logs may be filled with these messages:
%PFE-3: fpc0 BRCM-VIRTUAL,brcm_virtual_tunnel_nh_install(),2561:Tunnel_nh_hw install failed for nh(9412)
%PFE-3: fpc0 BRCM-VIRTUAL,brcm_virtual_tunnel_create_overlay_nh(),1966:Failed egress if create (Table full) nh-id = 0x14341 gport = 0x80001eb7
The issue is related to default overlay NH scale limitation of 8192 NHs. This can be easily fixed by increasing the NH entries available with this command:
# set forwarding-options vxlan-routing next-hop <integer> <<<<<<<<< Where integer can go from 0 through 61440. The specified value must be a multiple of 4096—for example, 4096, 8192, 12.288, and so on.
NOTE: A restart of the PFE process is required for this change to take precedence.
Those 2 steps (configuration and PFE restart) should be performed across all LEAF devices for consistency and avoiding future events.
After this, traffic is stable and drops stopped.