New routes are not getting installed and the following error messages are seen on the device:
%PFE-3: fpc0 BRCM_NH-,brcm_nh_mpls_vc_and_swap_install(),2297:BCM L3 Egress create object failed - NH 9495(-6:Table full), BCM NH Params: PFE:0, Port:1, VLAN:4095, L3_INTF:108, Flags: 0x200, mpls_qos_map_id:0 type 2
and
%PFE-3: fpc0 BRCM_NH-,brcm_nh_l3_hw_install(),988:bcm l3 egress create object failed for nh 8140 (-6:Table full) pfe:0, port:13, vlan:4, l3-intf:0 flags: 0 unit 0
This issue is due to the route scale exceeding one of the allowed route types, which causes exhaustion of the maximum L3 egress objects in the forwarding engine.
Verify the unified forwarding table configured on the device and check the number of routes installed in routing table:
root@QFX5200> show chassis forwarding-options localre: -------------------------------------------------------------------------- UFT Configuration: lpm-profile. prefix-65-127 = enable --------------Bank details for various types of entries------------------ Entry type Dedicated Bank Size(K) Shared Bank Size(K) L2(mac) 8 32 * num shared banks L3(unicast & multicast) 8 32 * num shared banks Exact Match 0 16 * num shared banks Longest Prefix Match(lpm) 16 32 * num shared banks root@QFX5200> show route summary inet.0: 28847 destinations, 163377 routes (28846 active, 0 holddown, 1 hidden) Limit/Threshold: 100000/85000 destinations Direct: 12 routes, 11 active Local: 10 routes, 10 active OSPF: 7650 routes, 7650 active BGP: 155679 routes, 21150 active Static: 5 routes, 4 active LDP: 1 routes, 1 active Access-internal: 20 routes, 20 active
lpm-profile.
Verify the number of routes supported in lpm-profile. Refer to the technical documentation on Understanding the Unified Forwarding Table . According to this technical documentation, when lpm-profile is used, it allows 8K unicast host-routes. Verify the number of /32 routes (host routes) in the routing table using 'show route' .
'show route'
Example:
labroot@QFX5200> show route | grep /32 1.1.1.0/32 *[Local/0] 1d 00:01:30 2.2.2.0/32 *[Local/0] 1d 00:01:30 10.102.162.204/32 *[Local/0] 1d 00:21:44 66.129.255.62/32 *[Static/5] 1d 00:21:44 101.101.101.101/32 *[OSPF/150] 21:33:12, metric 1, tag 0 128.102.162.204/32 *[Direct/0] 1d 00:21:44 ... ... <<Output ommitted for brevity>> labroot@QFX5200> show route | grep /32 | count Count: 9057 lines
If the number of host routes in the routing table exceeds this 8K limit, it will exhaust the l3 table on the PFE and lead to no other next-hop entries being able to be added. When this happens, the following error message will occur. In this case MPLS next-hop is not getting installed:
To resolve the issue, scale down the number of routes as per the configured UFT or configure suitable UFT profile under forwarding options using the command below as per the number of routes of the particular type (LPM/L3-Host or MACs) required. For example, we may choose l3-profile if more host-routes are required. That will increase the limit of host routes to 72K as shown below:
root@QFX5200# set chassis forwarding-options ? Possible completions: + apply-groups Groups from which to inherit configuration data + apply-groups-except Don't inherit configuration data from these groups > custom-profile (Partition the shared banks, will automatically restart the PFE when configured) > l2-profile-one MAC: 136K L3-host: 8K LPM: 16K. (restarts PFE automatically once the UFT profile is set) > l2-profile-three MAC: 72K L3-host: 72K LPM: 16K. (restarts PFE automatically once the UFT profile is set) > l2-profile-two MAC: 104K L3-host: 40K LPM: 16K. (restarts PFE automatically once the UFT profile is set) > l3-profile MAC: 40K L3-host: 104K LPM: 16K. (restarts PFE automatically once the UFT profile is set) > lpm-profile MAC: 8K L3-host: 8K LPM: 128K. (restarts PFE automatically once the UFT profile is set)