Description

on ACX710 and ACX5448 , Non-Cascaded FECs consists of "ECMP Shadow-FECs","Protection Pair FECs" , "Single FECs" and "Reserved FECs" . Non-Cascaded FECs space is limited to 36864 on ACX710 and 77824 on ACX5448. Non cascaded FECs are assigned to NHs. once the FEC space get exhausted, PFE errors for installing NHs will be seen in the PFE logs which may lead to out of sync between control and forwarding plane and hence service impact.

Jun 26 00:50:12  ACX5440 fpc0 dnx_nh_unilist_install_multipath: Failed to allocate ECMP chunk 
Jun 26 00:50:12  ACX5440 fpc0 dnx_nh_unilist_install_multipath: Failed to allocate ECMP chunk
​​​​Jun 26 00:50:13 ACX5440 fpc0 DNX_NH::dnx_nh_alloc_fec_id(),181: System reached max FEC space 77824
Jun 26 00:50:13 mACX5440 fpc0 DNX_NH::dnx_nh_alloc_fec_id(),181: System reached max FEC space 77824
Jun 26 00:50:13 ACX5440 fpc0 dnx_nh_ucast_vpn_hw_create: Failed to allocate FEC id for NH 20842 proto IPv4->MPLS
Jun 26 00:50:13 ACX5440 fpc0 dnx_nh_ucast_vpn_hw_create: Failed to allocate FEC id for NH 20843 proto IPv4->MPLS

Symptoms

Depletion of Non-Cascaded FECs space may be due to high usage of "ECMP Shadow-FECs" which is related to the number of unlist NHs that is related to labeled ECMP routes which has 4k (77824/18) ECMP resource limit for ACX5448 (without considering MBB)

By default, each unilist NH has 16 ECMP members.Here ECMP chunk allocation is happened for all 16 ECMP members for every unilist NH. Hence, this variable get incremented (ECMP Shadow-FECs) by 16 during FEC allocation and same will be decremented by 16 in dealloc.

to check FEC usage you can use the below PFE command
 

LNX-FPC0(rosemary vty)# show pfe-hw fec-usage
Entity                                 Total        Allocated
------------------------------------   ----------  ----------
Non-Cascaded FECs                           77824        24578
      ECMP Shadow-FECs                                 16380       >>>>>>>>related to ECMP paths
      Protection Pair FECs                                 0
      Single FECs                                       8194
      Reserved FECs                                        4
Cascaded FECs                               32768            0
Lo0 FECs                                    16384            2
ECMP FECs                                    4095         4095	
 

Solution

customer need to avoid having more that 4K ECMP label routes on the PFE of ACX5448. this will lead to the depletion of "Non-Cascaded FECs" and will lead to out of sync between control and forwarding plane. to recover from this state rebooting the box is the only way for recovering

starting from 24.1 , you can control the number of assigned FECs per ECMP using the below knob

# set system packet-forwarding-options ecmp-profile <4/8/12>

 

Modification History

2024-07-31 : Article Created