The user upgraded from 20.3X75-D21.7 to 22.4R3.25. After upgrade user saw dlu.ucode.not_routable trapcode increasing on fpc3.pfe0 and pfe1.

The user was seeing service impact.

Trapcode: dlu.ucode.not_routable trapcode increasing during problem state.

Router> request pfe execute target fpc3 command "show pechip
0 (  4) dlu.ucode.discard               2          0  DLU UCODE discard
0 (  8) dlu.ucode.not_routable         247942466        130719  DLU UCODE not routable
0 ( 12) dlu.ucode.pkt_to_cpu           100490          4  DLU UCODE packet to cpu
1 (  8) dlu.ucode.not_routable         247849057        126566  DLU UCODE not routable
1 ( 12) dlu.ucode.pkt_to_cpu           100263          4  DLU UCODE packet to cpu
2 ( 12) dlu.ucode.pkt_to_cpu            40178          1  DLU UCODE packet to cpu
3 ( 12) dlu.ucode.pkt_to_cpu            40085          1  DLU UCODE packet to cpu
4 ( 12) dlu.ucode.pkt_to_cpu            19380          0  DLU UCODE packet to cpu

Issue

   During the software upgrade, user's MOP had non-compliant interwoven steps of doing an ungraceful switchover. This led to a situation where the critical hardware programming of the SLU block (PE Asic) was wiped out as part of IFD/IFL Add message from the kernel. During switchover, FPCs are expected to reconnect to the new Master at a specific offset in the kernel ifstate chain. They enact this by embedding reconnect creds (Commit Marker Sequence number) in the reconnect message to the kernel. However, the kernel as no way of determining the extent of states post this marker, which could have already been shipped to the FPCs. So, on reconnecting, it tries to replay some of the previously known states. This is by design since the Junos kernel does not elicit acks from FPCs for every ifstate it sends down-stream. This is known as "Compressed Acks".

   Now as a virtue of state replay, FPC was overwriting the SLU l2d config in the PE Asic as it absolutely cannot differentiate between the same message arriving multiple times for the given AE child link. Under correct/expected (graceful switchover & system is GRES ready) circumstance, this wouldn't happen as FPCs would have synced their state with the current Master completely & only then be allowed to switchover.

Consequence of above

   In the aftermath of the above, the SLU l2d config loses the l3VpnId's for both IPv4 & IPv6 IFFs configured on the AE parent IFL for the AE child IFL. This child IFL will now start dropping both IPv4 & IPv6 traffic with "dlu-ucode-not-routable" trap-code.

Proposed Fix

   In the proposed fix under validation at CFTS, we're first reading the existing SLU config, populating a template, making point changes as part of the above message processing, and only then, overwriting the hardware config. This way, we ensure that sane values are retained across multiple instances of the same state replay at FPC & that across each rewrite, SLU is rendered in a sane state.

Pointers to useful documents

Internal page on GRES: https://coreos.juniper.net/junos-kernel/gres

Recommended upgrade procedure with gres interplay : https://www.juniper.net/documentation/us/en/software/junos/release-notes/21.2/junos-release-notes-21.2r1/topics/upgrade-downgrade/ptx-upgrade-downgrade.html

Please refer PR for more details.