On Junos Evolved ACX platforms, if the ECMP paths tied to a route change due to network churn, the Packet Forwarding Engine (PFE) may fail to properly update the hierarchy change in the hardware. Due to this failed update, traffic drops may be seen for the affected flows.
When the problem is hit, the following evo-pfemand logs will be reported in messages:
evo-pfemand[13548]: [t:13548] [Error] compName = "RT", tpName = "BrcmRtV4", msg = " int BrcmRt::handleEalOp(BrcmRt::rtOper) = Failed to perform operation for route on BCM, operation = Add, prefix = 172.16.1/24, prefixlen = 24, NH type = 4"
evo-pfemand[13548]: [t:13548] [Error] compName = "RT", tpName = "BrcmRt", msg = " int BrcmRt::addImpl() = Handling route add failed, nhIndex = 17732, prefix = 172.16.1/24, prefixlen = 24, proto = ^B"
evo-pfemand[13548]: EVO_PFEMAND_ROUTE_PROCESSING_FAILED: Failed to process addition for the ipv4 route proto: ipv4 prefix: 172.16.1/24 with vrf id 0 reachable via next hop with index 17732
evo-pfemand[13548]: EVO_PFEMAND_ASIC_PROGRAM_FAILED: Failed to program the hardware with an error - entry doesn't exist in hardware
evo-pfemand[13548]: [t:20136] [Error] compName = "Broadcom", tpName = "BrcmErrorMsg", msg = " Failed to program the hardware with an error - = entry doesn't exist in hardware, during = jbcm_l3_route_delete"
evo-pfemand[13548]: [t:20136] [Error] compName = "BrcmPlus", tpName = "func:", msg = " int brcmL3Route::brcm_qdl_entry_process() = brcm route delete failed, error = Entry not found"
evo-pfemand[13548]: [t:13548] [Error] compName = "NH", tpName = "BrcmPlusNh", msg = " static uint32_t BrcmPlusNhApi::getNhFecIndex(uint32_t, uint32_t, bool) = Failed to retrieve FEC Id, NH Index = 17732, NH type = 4, fwdRslt = 1"
evo-pfemand[13548]: [t:13548] [Error] compName = "RT", tpName = "BrcmPlusRtV4", msg = " static status_t BrcmRtV4Entry::updateHwLpmTblDispatch(const BrcmParamsRtV4Ptr&, Op) [with DispatchType = brcmL3Route; status_t = int; BrcmParamsRtV4Ptr = std::shared_ptr; Op = BrcmParams::brcmOper] = Failed to get the HW nexthop index, nhType = 4, Eal NH Index = 17732"
Resolved in PR1810357