Description

On QFX and EX series switch which is working in EVPN vxlan network, you may observe syslog message below. This KB describes this message and provide a solution.

Oct 8 19:26:33 Switch fpc0 BRCM-VIRTUAL,brcm_vxlan_riot_create_nh(),2016:Failed egress if create (Table full) nh-id = 12361 gport = 0x0 vplag = 2113 l3_egress_if.intf:132 l3_egress_if.flags:200 l3_egress_if.trunk:2113 l3_egress_if.port:80000841 egr = 0
Oct 8 19:26:33 Switch fpc0 BRCM-VIRTUAL,brcm_vxlan_riot_arp_ndp_install(),2521:IRB_ARP_NDP hw install failed for nh-id 12361

 

Symptoms

On QFX and EX series switch which is working in EVPN vxlan network, traffic loss might seen with error message below.
Oct 8 19:26:33 Switch fpc0 BRCM-VIRTUAL,brcm_vxlan_riot_create_nh(),2016:Failed egress if create (Table full) nh-id = 12361 gport = 0x0 vplag = 2113 l3_egress_if.intf:132 l3_egress_if.flags:200 l3_egress_if.trunk:2113 l3_egress_if.port:80000841 egr = 0
Oct 8 19:26:33 Switch fpc0 BRCM-VIRTUAL,brcm_vxlan_riot_arp_ndp_install(),2521:IRB_ARP_NDP hw install failed for nh-id 12361

The issue is seen when the switch has vtep interfaces and learning many ARP entries from that interfaces.

Solution

The ARP entries are stored in PFE as next hops and QFX and EX switch can store those next hops for underlay network and EVPN vxlan overlay network separately.
The default size of maximum number of next hops is 8K for overlay network, so if there are many ARP entries learning through vteps, there is a chance to exhaust allocated next hops.
This issue is more likely to occur, especially when the device is working as a core switch, as it tends to consume overlay next hop entries more easily.

As a solution, you can change the maximum number of reserved next hops for overlay network.
*Please note, this will restart PFE and brings service outage. Please have a maintenance window.

 

1) Check the current number of entries of "OVERLAY NH". This is the number of ARP entries learning from vteps.
You can use below commands:

user@Switch> show pfe vxlan nh-usage
or
user@Switch> request pfe execute target fpc0 command "show shim virtual nh-usage"

------------------OVERLAY NH------------------
Overlay bank size                  : 8192

Current usage                      :
Reserved entries                   : 1
ARP NDP/Type-5 tunnel NH entries   : 3835
Available entries                  : 4356

Expected usage                     :
Reserved entries                   : 1
ARP NDP/Type-5 tunnel NH entries   : 0
Available entries                  : 8191

-----------------UNDERLAY NH-----------------
Underlay bank size                 : 57344

Current usage                      :
Reserved entries                   : 5
Underlay NH entries                : 143
Available entries                  : 57196

Expected usage                     :
Reserved entries                   : 5
NHs to CPU (IRB + L3 intf) entries : 2
NH ARPs of underlay L3 intfs       : 4
Total tagged and access ports      : 0
VTEP Flood NHs                     : 2
Available entries                  : 57331

 

2) If there are enough "Available entries" for UNDERLAY NH, change the maximum number of OVERLAY NH.

set forwarding-options vxlan-routing next-hop 12288

*The integer you specify must be a multiple of 4096—for example, 4096, 8192, 12288, and so on.

 

3) Check the output again.

------------------OVERLAY NH------------------
Overlay bank size                  : 12288  << Increased from 8192

Current usage                      :
Reserved entries                   : 1
ARP NDP/Type-5 tunnel NH entries   : 9027
Available entries                  : 3260

Expected usage                     :
Reserved entries                   : 1
ARP NDP/Type-5 tunnel NH entries   : 0
Available entries                  : 12287

-----------------UNDERLAY NH-----------------
Underlay bank size                 : 53248  << Decreased from 57344

Current usage                      :
Reserved entries                   : 5
Underlay NH entries                : 143
Available entries                  : 53100

Expected usage                     :
Reserved entries                   : 5
NHs to CPU (IRB + L3 intf) entries : 2
NH ARPs of underlay L3 intfs       : 4
Total tagged and access ports      : 0
VTEP Flood NHs                     : 2
Available entries                  : 53235

 

Changing allocation will cause PFE to restart. You can see below message at that time.

Oct 8 19:30:35 Switch fpc0 FXPC_PLANNED_RESTART_REASON: INFO: fxpc_pfe_planned_restart_process: Preparing for planned restart reason 2

 

Modification History

2024-10-08 : Article Created

Related Information

https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/next-hop-edit-forwarding-options-vxlan-routing.html