on ACX710 and ACX5448 , Non-Cascaded FECs consists of "ECMP Shadow-FECs","Protection Pair FECs" , "Single FECs" and "Reserved FECs" . Non-Cascaded FECs space is limited to 36864 on ACX710 and 77824 on ACX5448. Non cascaded FECs are assigned to NHs. once the FEC space get exhausted, PFE errors for installing NHs will be seen in the PFE logs which may lead to out of sync between control and forwarding plane and hence service impact.
Jun 26 00:50:12 ACX5440 fpc0 dnx_nh_unilist_install_multipath: Failed to allocate ECMP chunk Jun 26 00:50:12 ACX5440 fpc0 dnx_nh_unilist_install_multipath: Failed to allocate ECMP chunk Jun 26 00:50:13 ACX5440 fpc0 DNX_NH::dnx_nh_alloc_fec_id(),181: System reached max FEC space 77824 Jun 26 00:50:13 mACX5440 fpc0 DNX_NH::dnx_nh_alloc_fec_id(),181: System reached max FEC space 77824 Jun 26 00:50:13 ACX5440 fpc0 dnx_nh_ucast_vpn_hw_create: Failed to allocate FEC id for NH 20842 proto IPv4->MPLS Jun 26 00:50:13 ACX5440 fpc0 dnx_nh_ucast_vpn_hw_create: Failed to allocate FEC id for NH 20843 proto IPv4->MPLS
Depletion of Non-Cascaded FECs space may be due to high usage of "ECMP Shadow-FECs" which is related to the number of unlist NHs that is related to labeled ECMP routes which has 4k (77824/18) ECMP resource limit for ACX5448 (without considering MBB)
By default, each unilist NH has 16 ECMP members.Here ECMP chunk allocation is happened for all 16 ECMP members for every unilist NH. Hence, this variable get incremented (ECMP Shadow-FECs) by 16 during FEC allocation and same will be decremented by 16 in dealloc.
to check FEC usage you can use the below PFE command
LNX-FPC0(rosemary vty)# show pfe-hw fec-usage Entity Total Allocated ------------------------------------ ---------- ---------- Non-Cascaded FECs 77824 24578 ECMP Shadow-FECs 16380 >>>>>>>>related to ECMP paths Protection Pair FECs 0 Single FECs 8194 Reserved FECs 4 Cascaded FECs 32768 0 Lo0 FECs 16384 2 ECMP FECs 4095 4095
customer need to avoid having more that 4K ECMP label routes on the PFE of ACX5448. this will lead to the depletion of "Non-Cascaded FECs" and will lead to out of sync between control and forwarding plane. to recover from this state rebooting the box is the only way for recovering
starting from 24.1 , you can control the number of assigned FECs per ECMP using the below knob
# set system packet-forwarding-options ecmp-profile <4/8/12>