On QFX5220 devices, when MPLS FRR (Fast Re-Route) is enabled, it takes ae child interfaces as ECMP members and incorrectly programs the hardware which eventually causing connectivity issues.
Programming before FRR:
[vrf:none] root@user:~# jbcmcmd.py "d chg MPLS_ENTRY_SINGLE" | grep 0x21
MPLS_ENTRY_SINGLE.ipipe0[1932]: <V6_ENABLE=1,
V4_ENABLE=1,
TABLE_FIELDS=0x047006946003c00c00004200,
NEXT_HOP_INDEX=0xf,
MPLS_LABEL=0x21, <--- Label 33
MPLS_ACTION_IF_NOT_BOS=2,
MPLS_ACTION_IF_BOS=4, <--- 4=L3_VPN and NHI,
MPLS:V6_ENABLE=1,
MPLS:V4_ENABLE=1,
MPLS:NEXT_HOP_INDEX=0xf, <--- NHI is 0xf (15)
MPLS:MPLS_LABEL=0x21,
MPLS:MPLS_ACTION_IF_NOT_BOS=2,
MPLS:MPLS_ACTION_IF_BOS=4,
MPLS:L3_IIF=0xf,
MPLS:KEY=0x10800,
MPLS:HASH_LSB=0x21,
MPLS:FLEX_CTR_POOL_NUMBER=3,
MPLS:FLEX_CTR_OFFSET_MODE=2,
MPLS:FLEX_CTR_BASE_COUNTER_IDX=0xd,
MPLS:ECMP_PTR=0xf,
MPLS:DO_NOT_CHANGE_PAYLOAD_DSCP=1,
MPLS:DECAP_USE_EXP_FOR_PRI=1,
MPLS:DECAP_USE_EXP_FOR_INNER=1,
MPLS:DATA_0=0x047006946003c00c,
L3_IIF=0xf,
KEY=0x10800,
HASH_LSB=0x21,
FLEX_CTR_POOL_NUMBER=3,
FLEX_CTR_OFFSET_MODE=2,
FLEX_CTR_BASE_COUNTER_IDX=0xd,
ECMP_PTR=0xf,
DO_NOT_CHANGE_PAYLOAD_DSCP=1,
DECAP_USE_EXP_FOR_PRI=1,
DECAP_USE_EXP_FOR_INNER=1,
DATA_0=0x047006946003c00c,
BASE_VALID=1>
Programming after FRR:
TABLE_FIELDS=0x067006886000c00e00004200,
NEXT_HOP_INDEX=3,
MPLS_ACTION_IF_NOT_BOS=4,
MPLS_ACTION_IF_BOS=6, <--- Value changed to 6 which is incorrect. It should be either 4 or 5.
MPLS:NEXT_HOP_INDEX=3, <--- ECMP NH
MPLS:MPLS_ACTION_IF_NOT_BOS=4,
MPLS:MPLS_ACTION_IF_BOS=6,
MPLS:L3_IIF=3,
MPLS:ECMP_PTR=3,
MPLS:DECAP_USE_TTL=1,
MPLS:DATA_0=0x067006886000c00e,
L3_IIF=3,
ECMP_PTR=3,
DECAP_USE_TTL=1,
DATA_0=0x067006886000c00e,
This issue is due to a software bug and resolved in evo:22.2R3-S4-EVO evo:23.4R2-EVO evo:24.2R1-EVO evo:24.3R1-EVO through PR1797319.