Traffic backhole is observed when shifting the transport path from BGP-LU to SR-TE on ACX7024 running 25.2R1-EVO

1.) SR-TE head-end device is ACX7024 running 25.2R1-EVO. But the problem was also seen in 24.4R2 and 24.2R2

2.) During initial state, the router was configured to take BGP-LU as transport.

3.) The traffic-blackhole started being observed as soon as the configuration update was made to shift transport path from BGP-LU to SR-TE.

4.) It was observed that the Label stack was getting replaced / reprogrammed in place instead of a destroy/create during the transport change and a new NH wasn't getting created. 

Below Logs are seen during the issue:

Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] brcm-sdk : dnx_mpls_tunnel_initiator_create_verify: Error 'Invalid parameter' indicated, replacing entry with logical phase(2) with new value(0) - not supported, logical phase must not change in REPLACE
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] brcm-sdk : bcm_dnx_mpls_tunnel_initiator_create: Error indicated (Invalid parameter) on VERIFY ;
Jun 22 15:43:42 an3 evo-pfemand[8585]: EVO_PFEMAND_ASIC_PROGRAM_FAILED: Failed to program the hardware with an error - invalid parameters specified
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] Broadcom : BrcmErrorMsg: Failed to program the hardware with an error - = invalid parameters specified, during = jbcm_mpls_tunnel_initiator_create
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] NH : BrcmPlusNhMplsVpn: status_t BrcmPlusNhMplsVpnHwList::createEntryInHw(uint32_t, BrcmPlusNhEncapAccessList&) = mpls encap create failed, unit = 0, Error = Invalid parameter
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] NH : BrcmPlusNhMpls: status_t BrcmPlusNhMplsVpnHwList::create() = mpls encap create hw basic
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] NH : BrcmPlusNhMplsVpn: mpls vpn failed to program the hardware = { tnlIntfIdx 0 : 1074790218}, ]
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] NH : BrcmPlusNhMplsVpn: status_t BrcmPlusNhMplsVpn::vpnHwCreate(BrcmPlusNhMplsVpnDataShrPtr&) = vpnJ2EncapHwCreate failed, nhIndex = 8315
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] NH : BrcmPlusNhMplsVpn: static status_t BrcmPlusNhMplsVpn::create(BrcmPlusNhMplsVpn_t&, BrcmPlusNhMplsVpnDataShrPtr&) = vpnHwCreate FAILED, guid = M-^P, nhIndex = 8315, fecId = 536933358, proto = unkwn->mpls
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] BrcmPfe : BrcmPlusNhCompChain: vpnHwCreate failed for NH-index 8315 tnl fec id =537002128 proto = 19
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] compName = "NH", ddsType = "CompChain" tpName = "BrcmPlusNhCompChain", guid = "781684049296" msg = " nhIndex = 8315, nhToken = 536933358, proto = unkwn->mpls, tnl fec id = 537002128"
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] compName = "NH", ddsType = "CompChain" tpName = "BrcmPlusNhCompChain", guid = "781684049296" msg = " nhIndex = 8315, nhToken = 536933358, proto = unkwn->mpls, NH-Token = = 536933358, Op = Add Composite Chain"
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] BrcmPfe : BrcmPlusNhCompChain: compChainHwInstall failed for NH-index 8315 proto = 19
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] NH : BrcmNhGeneric: static status_t BrcmNhGeneric<BrcmParamsType, BrcmNhType>::create(const BrcmParamsType&) [with BrcmParamsType = std::shared_ptr<BrcmParamsNhCompChain>; BrcmNhType = BrcmNhCompChain; status_t = int] = Failed to create entries in HW for NH Modify, nhIndex = 8315, nhType = 19
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] compName = "NH", ddsType = "Nexthop" tpName = "BrcmNhCompChain", guid = "781684049296" msg = " nhIndex = 8315, proto = unkwn->mpls, Failed to get Plus Ptr for service = TunnelService"
Jun 22 15:43:42 an3 evo-pfemand[8585]: EVO_PFEMAND_NEXTHOP_PROCESSING_FAILED: Failed to process modification for the next-hop with index 8315 of type composite
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] brcm-sdk : dnx_mpls_tunnel_initiator_create_verify: Error 'Invalid parameter' indicated, replacing entry with logical phase(2) with new value(0) - not supported, logical phase must not change in REPLACE
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] brcm-sdk : bcm_dnx_mpls_tunnel_initiator_create: Error indicated (Invalid parameter) on VERIFY ;
Jun 22 15:43:42 an3 evo-pfemand[8585]: EVO_PFEMAND_ASIC_PROGRAM_FAILED: Failed to program the hardware with an error - invalid parameters specified
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] Broadcom : BrcmErrorMsg: Failed to program the hardware with an error - = invalid parameters specified, during = jbcm_mpls_tunnel_initiator_create
Jun 22 15:43:42 an3 evo-pfemand[8585]: [t:8585] [Error] NH : BrcmPlusNhMplsVpn: status_t BrcmPlusNhMplsVpnHwList::createEntryInHw(uint32_t, BrcmPlusNhEncapAccessList&) = mpls encap create failed, unit = 0, Error = Invalid parameter

This issue has been fixed via PR1890392.

The PR is fixed in the following releases:

evo:25.2R1-S1-EVO

evo:25.2R2-EVO

evo:25.4R1-EVO