Description

This article explains on how the hardware limit is calculated in EX4300 for IPv4 and IPv6 LPM routes.

Symptoms

This article can be used to calculate the route scale on an EX4300 device when below issues are observed.
  • Connectivity issue. Route entry is present in the routing table but still unable to reach the destination.
  • Following table full messages are reported
[Wed Mar 13 03:53:36.401] [291186106] pfe_bcm_rt_ip_uc_lpm_install:295 PFE_BCM_RT-(pfe_bcm_rt_ip_uc_lpm_install:LPM route change failed) Reason : Table full
[Wed Mar 13 03:53:36.401] [291186107] pfe_bcm_rt_ip_uc_entry_install:1044 PFE_BCM_RT-pfe_bcm_rt_ip_uc_entry_install vrf: 0 rt: 2607:f098:50fe:0:500:100:0:da/127, action: 1
[Wed Mar 13 03:53:36.401] [291186108] pfe_bcm_rt_ip_entry_change:1508 PFE_BCM_RT-Route Change RTT idx:0 rt:2607:f098:50fe:0:500:100:0:da/127, status:0
[Wed Mar 13 03:53:36.401] [291186109] pfe_bcm_rt_entry_change:2110 PFE_BCM_RT-vrf:0 rt:2607:f098:50fe:0:500:100:0:dc/127
[Wed Mar 13 16:41:08.390] [77196] pfe_bcm_rt_ip_uc_lpm_install:295 PFE_BCM_RT-(pfe_bcm_rt_ip_uc_lpm_install:LPM route add failed) Reason : Table full
[Wed Mar 13 16:41:08.390] [77197] pfe_bcm_rt_ip_uc_entry_install:1040 PFE_BCM_RT-pfe_bcm_rt_ip_uc_entry_install vrf: 0 ipaddr: 71.41.34.0, ip mask: fffffff8, rt: 71.41.34.0/29 action: 0 class_id 0
 
  • The "LPM route add failed) Reason: Table full" errors are not reported in the Junos Messages file or PFE Syslog, irrespective of the Syslog file severity settings configuration. It's only reported in the RT-Halp Ukern_traces.

user@lab> start shell
% vty fpc0

(vty)# sh ukern_trace handles
Ukernel Trace Info:
ID     Name             Level     Printf Logging Size   Wrap  Filter-id
-----  ---------------  ---------  -----  -----  -----  ----- ---------

30     RT-HALP          terse      Off    On     1000000      4     - >>>>>>>>>> get the ID for RT-HALP (here it is 30)

(vty)# show ukern_trace 30 >>>>> this will dump all the logs.

Please note:
sh ukern_trace output will dump lot of entries, so make sure you have the session saved as it may get overwritten. Also the buffer size of ukern_trace is very low, so it is recommended to dump the output during the problem state as the logs are rolled quickly.
  • You can confirm the route utilization using below cli commands

user@lab> show route summary
user@lab> show pfe route summary hw   
Slot 0
Type            Max       Used      Free      % free
----------------------------------------------------
IPv4 Host       86016     4032      79944     92.94
IPv4 LPM        16384     507       15663     95.60
IPv4 Mcast      43008     0         39972     92.94
 
IPv6 Host       43008     1016      39972     92.94
IPv6 LPM(< 64)  8192      107       7832      95.61
IPv6 LPM(> 64)  4096      3860      236       5.76
IPv6 Mcast      21504     2         19986     92.94
 

Solution

On EX4300 the Hardware LPM table size supports up to 8K indexes/entries.
This is the hardware limitation beyond which it cannot scale. Hence any excess LPM route (either individually by one of these route categories or collectively by these LPM route categories) will result in LPM Table full scenario. This LPM memory table is specific for handling all categories of LPM routes in the hardware. For Host Routes ( /32 and /128) and Mcast Routes a different memory table is used.

(vty)# set ex bc "listmem L3_DEFIP"
Memory: L3_DEFIP.ipipe0 address 0x20000000
Flags: valid cachable(on)
Blocks:  ipipe0 (1 copy)
Entries: 8192 with indices 0-8191>>>>>>>>> Max entries supported.
 
By default,  each IPv4 route will take half ( ½) index whereas each (IPv6 LPM > 64 ) route will use 2 indexes and each (IPv6 LPM < 64 ) will use 1 index.
  • One L3_DEFIP  entry can contain  ~2 IPv4 LPM routes
  • One L3_DEFIP  entry can contain ~1 IPv6 < 64 LPM route.
  • Two  L3_DEFIP  entry can contain the 1 IPv6 > 64 LPM  route

Based on this calculation,
  • The maximum number of IPv4 LPM routes supported =  8K x 2 = 16K.
  • The maximum number of IPv6 < 64 LPM routes supported =  8K x 1 = 8K.
  • The maximum number of IPv6 > 64 LPM routes supported = 8K / 2 = 4K
 
Note that these max ranges are not collective instead it is mutually exclusive i.e. either Max of IPv6 or IPv4 entries can be achieved.
To further simply this calculation :

Scenario 1 :
Total number of L3 ipv4 LPM routes = 2000.
Total number of L3 IPv6 > 64 LPMroutes = 3500.

Per calculation:
L3 ipv4 LPM routes will consume 2000/2 = 1000 entries
L3 IPv6 > 64 LPMroutes will consume 3500 x 2 = 7000 entries.

In total = 1000 + 7000  = 8000 entries  (close to max scale limit of 8192 entries). LPM Table full issue *will not be seen* in this scenario as the Max limit is still not breached though the utilization is quite close to 100%.
 
 
Scenario 2 :
Total number of L3 ipv4 LPM routes = 2000
Total number of L3 IPv6 > 64 LPMroutes = 4000

Per calculations:
L3 ipv4 LPM routes will consume 2000/2 = 1000 entries.
L3 IPv6 > 64 LPMroutes will consume 4000 x 2 = 8000 entries.

In total = 1000 +  8000 = 9000 entries  ( exceeding the max scale limit of 8192 entries). LPM Table full issue *will  be seen* in this scenario as the Max limit is not breached.

Below data collection commands can be used for examining the LPM utilization (Please verify these commands in the lab EX4300 running customer release before running it on the customer device)
 
CLI :
show route summary
show route forwarding-table summary 
show pfe route summary hw
show route forwarding-table family inet6
show route forwarding-table family inet 

PFE :
cprod -A fpc0 -c "show route ip table"  
cprod -A fpc0 -c "show route inet6"
cprod -A fpc0 -c "show route inet6 hw lpm" " | wc -l
cprod -A fpc0 -c "show route inet6 hw lpm" 
cprod -A fpc0 -c "show route ip hw lpm" | wc -l
cprod -A fpc0 -c "show route ip hw lpm"
cprod -A fpc0 -c "show route ip"

BROADCOM:
cprod -A fpc0 -c ‘set ex bc "set dcbcm getconfig "’
cprod -A fpc0 -c ‘set ex bc "listmem L3_DEFIP”’
cprod -A fpc0 -c 'set ex bc " set ex d chg L3_DEFIP"'
cprod -A fpc0 -c ‘set ex bc "l3 ip6route show"’
cprod -A fpc0 -c ‘set ex bc "l3 l3table show"’
cprod -A fpc0 -c 'set ex bc "l3 defip show"'
 

Modification History

2024-03-14 : Article Created