In EVPN VXLAN deployments with any of the QFX5110, QFX5120, and EX4400 device models, the default allocated resources for next-hops and interface-nums could get exhausted, leading to connectivity issues in the Fabric.

Across all platforms, the default value for the next-hops table is 8K, and the default interface-num value is 4K, which can get exhausted.

Customers can validate the currently allocated and available next-hops and interface-num on each platform with the following commands:

show pfe vxlan nh-usage
show pfe vxlan intf-usage

It is highly recommended, however, that customers increase these values when creating an EVPN VXLAN Fabric with any of the QFX5110, QFX5120, and EX4400 device models to avoid any connectivity issues.

The next-hop parameter indicates the number of next-hops to be used in the VXLAN OVERLAY and UNDERLAY networks. The maximum number of entries is 48K (49152), shared between both OVERLAY and UNDERLAY.

The following functionalities occupy a next-hop table and can be affected when the next-hop table is exhausted.

1. ARPs resolved via local IRBs (VXLAN enabled VLANs) and ARPs resolved over remote Leafs

2. ARPs of remote Leaf IRBs

3. Active TYPE-5 tunnel next-hops. This is equal to the number of TYPE-5 tunnels toward other Leafs and Spines from each routing instance.

interface-num indicates the number of local L3 interfaces that are part of the OVERLAY and UNDERLAY networks. The maximum count here is 12K (12288), which is again shared between the OVERLAY and UNDERLAY networks. When interface-num is exhausted, no new IRB interfaces and no new routing instances can be created.

WARNING: Changing the default number of next-hops OR physical interfaces that are reserved for use in an EVPN-VXLAN network causes the Packet Forwarding Engine (PFE) to restart. Restarting the PFE interrupts all forwarding operations. Therefore, we strongly recommend that you use this configuration statement before the EVPN-VXLAN network becomes operational OR during a Maintenance Window if the EVPN-VXLAN network is operational.

To change the default next-hop and interface-num values, users can create a configlet and assign to all the devices in the Fabric. The following values are set with the below configlet and users can change these values based on the recommendation made by Support.

Perform the following steps:

1. Create the configlet as shown here:

• OS Style: Junos

• Section: System

• Template Text            

{%if ('Juniper_QFX5110' in model or 'Juniper_QFX5120' in model or 'Juniper_EX4400' in model) %}
forwarding-options {
vxlan-routing {
{% if 'Juniper_QFX5110' in model %}
next-hop 32768;
interface-num 8192;
overlay-ecmp;
{% endif %}
{% if 'Juniper_QFX5120' in model %}
next-hop 45056;
interface-num 8192;
overlay-ecmp;
{% endif %}
{% if 'Juniper_EX4400' in model %}
next-hop 16384;
interface-num 6144;
{% endif %}
}
}
{% endif %}

2. Import the configlet. From your Blueprint, import this configlet and apply it to all required leaf-pairs. The configlet will set the value for the next-hops and physical interfaces that will be reserved for the EVPN-VXLAN network. This can be removed later by removing the configlet from the Blueprint.

Note that you will need to re-import the configlet only when you add new switches to the Fabric.

Configlets are not automatically applied to the newly added switches in Juniper Apstra. While adding any new Leaf switches to the Fabric, users must manually delete and re-import the configlet.

2022-07-14: Initial version

2023-03-24: Minor modifications