Description

The unilist selector is stuck with a weight of 65535, and the session ID associated with protocol next-hop (PNH) remains down following rapid flaps of the primary interface event (DOWN-UP-DOWN) .

Symptoms

The rapid flapping of the primary interface (DOWN-UP-DOWN) caused traffic to blackhole. In this scenario, the session ID associated with the protocol next-hop (PNH) remains down, and the unilist selector becomes stuck with a weight of 65535, leading to traffic disruption.

In the topology described, xe-0/0/2 serves as the primary link, xe-0/0/0 as the backup, and xe-0/0/1 as the tertiary link. When the primary link (xe-0/0/2) experienced rapid flapping, we observed that traffic did not route through the backup link. Instead, traffic blackholing occurred on the DUT.

The route of interest is 2600:387:c:7400::8, received from two Route Reflectors (RRs) with two protocol next-hops (PNHs): ::ffff:32.128.6.130 and ::ffff:32.128.6.140. These PNHs are resolved via all links, with xe-0/0/2 as primary, xe-0/0/0 as backup, and xe-0/0/1 as tertiary. During normal operation, traffic from the southbound direction of DUT should only traverse the primary link. When the primary link fails, traffic should automatically switch to the backup link (xe-0/0/0). However, during the issue, this expected behavior wasn't observed. Instead, traffic was blackholed on the DUT and didn't route over the backup link as intended.

Working state:(primary link is up)

labroot@router-RE0> show route forwarding-table destination 2600:387:c:7400::9 table GN
Routing table: GN.inet6
Internet6:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
2600:387:c:7400::8/126
                   user     0                    ulst 1069710     2
                                                 comp 1069708     2
                                                 comp 1069706     2

labroot@router-RE0> request pfe execute command "show nhdb id 1069710 recursive" target fpc0
SENT: Ukern command: show nhdb id 1069710 recursive

1069710(Unilist, IPv4, ifl:0:-, pfe-id:0)
    1069708(Label, IPv6->MPLS, ifl:0:-, pfe-id:0)
        1069704(Indirect, IPv6, ifl:324:ae0.102, pfe-id:0, i-ifl:0:-)
            1093926(Unilist, IPv4, ifl:0:-, pfe-id:0)
                40751(Unicast, IPv4->MPLS, ifl:324:ae0.102, pfe-id:0)
                1093458(Indirect, IPv4, ifl:327:ae1.102, pfe-id:0, i-ifl:0:-)
                    44214(Unicast, IPv4->MPLS, ifl:327:ae1.102, pfe-id:0)
    1069706(Label, IPv6->MPLS, ifl:0:-, pfe-id:0)
        1069062(Indirect, IPv6, ifl:324:ae0.102, pfe-id:0, i-ifl:0:-)
            1069056(Unilist, IPv4, ifl:0:-, pfe-id:0)
                55034(Unicast, IPv4->MPLS, ifl:324:ae0.102, pfe-id:0)
                1069054(Indirect, IPv4, ifl:327:ae1.102, pfe-id:0, i-ifl:0:-)
                    27860(Unicast, IPv4->MPLS, ifl:327:ae1.102, pfe-id:0)

labroot@router-RE0> request pfe execute command "show nhdb id 1069710 extensive" target fpc0
SENT: Ukern command: show nhdb id 1069710 extensive

   ID      Type      Interface    Next Hop Addr    Protocol       Encap     MTU               Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- ------------------ ------------------
1069710   Unilist -              -                      IPv4             -     0 0x0000000000000000 0x0000000000000000

Selector (Second):
ID:11(1), Ref:2, Type:1 (Compact), subtype:0, Symmetric-LB: Off, Target_id:0
   Key:FRR:Y, Balances:N, Locality:N/unicast, Type:Unicast, Size:2, flags:0x14, dist-mode-default

Weight Info (Selector's view): Current Weight = 1, Target_id = 0   >>>>CURRENT WEIGHT IS 1
Idx Balance    Weight   Orig-Weight    Ifl     Session   Install
----- -------   -------   -----------   ------   -------   -------
    0       **         1             1       **       336      Yes
    1       **     16384         16384       **       335      No

Unilist: 1
Unilist Table (2 entries): List flags 0x0000000000000000

Unilist core-facing-index: 343
Underlying ifl-index : 0

HFRR force ifl selector : NO
ECMP : NO

1069708     Label -              -                IPv6->MPLS             -     0 0x0000000000000001 0x0000000000000000
1069706     Label -              -                IPv6->MPLS             -     0 0x0000000000000001 0x0000000000000000

Weight Info: Current Weight = 1
   ID Balance Orig-Balance Weight Orig-Weight State     Install      Flags
----- ------- --------- ------ ----------- -------- ----------- -----
1069708        0        0       1            1    Active    Installed 0x00
1069706        0        0   16384        16384   Standby    Installed 0x00

Routing-table id: 0

labroot@router-RE0> start shell pfe network fpc0
NPC platform (1067Mhz MPC 8548 processor, 2048MB memory, 512KB flash)

NPC0(COLNY401MOX-RE0 vty)# show pfe bfdsession id 336
-------------------
SESSION ID: 336
-------------------
Session Status : UP              <<<< SESSION IS UP
Session Version : 0x0
Number of NHs        : 1

NPC0(router-RE0 vty)# show pfe bfdsession id 335

-------------------
SESSION ID: 335
-------------------
Session Status : UP      <<<< SESSION IS UP
Session Version : 0x0
Number of NHs        : 1

Non-working state:(primary link is down)

labroot@router-RE0> show route forwarding-table destination 2600:387:c:7400::9 table GN
Routing table: GN.inet6
Internet6:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
2600:387:c:7400::8/126
                   user     0                    ulst 1087598     2
                                                 comp 1087597     2
                                                 comp 1126659     2

labroot@router-RE0> request pfe execute command "show nhdb id 1087598 recursive" target fpc0
SENT: Ukern command: show nhdb id 1087598 recursive

1087598(Unilist, IPv4, ifl:0:-, pfe-id:0)
    1087597(Label, IPv6->MPLS, ifl:0:-, pfe-id:0)
        1124025(Indirect, IPv6, ifl:0:-, pfe-id:0, i-ifl:0:-)
            1118498(Unilist, IPv4, ifl:0:-, pfe-id:0)
                1122182(Indirect, IPv4, ifl:0:-, pfe-id:0, i-ifl:0:-)
                    18683(Compst, IPv4->MPLS, ifl:0:-, pfe-id:0, comp-fn:Chain)
                        835(Unicast, IPv4, ifl:327:ae1.102, pfe-id:0)
                1118497(Indirect, IPv4, ifl:0:-, pfe-id:0, i-ifl:0:-)
                    64157(Compst, IPv4->MPLS, ifl:0:-, pfe-id:0, comp-fn:Chain)
                        2065(Unicast, IPv4, ifl:330:ae2.11, pfe-id:0)
    1126659(Label, IPv6->MPLS, ifl:0:-, pfe-id:0)
        1126658(Indirect, IPv6, ifl:0:-, pfe-id:0, i-ifl:0:-)
            1118500(Unilist, IPv4, ifl:0:-, pfe-id:0)
                1125995(Indirect, IPv6, ifl:0:-, pfe-id:0, i-ifl:0:-)
                    18689(Compst, IPv4->MPLS, ifl:0:-, pfe-id:0, comp-fn:Chain)
                        835(Unicast, IPv4, ifl:327:ae1.102, pfe-id:0)
                1118499(Indirect, IPv4, ifl:0:-, pfe-id:0, i-ifl:0:-)
                    65560(Compst, IPv4->MPLS, ifl:0:-, pfe-id:0, comp-fn:Chain)
                        2065(Unicast, IPv4, ifl:330:ae2.11, pfe-id:0)

labroot@router-RE0> request pfe execute command "show nhdb id 1087598 extensive" target fpc0
SENT: Ukern command: show nhdb id 1087598 extensive

   ID      Type      Interface    Next Hop Addr    Protocol       Encap     MTU               Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- ------------------ ------------------
1087598   Unilist -              -                      IPv4             -     0 0x0000000000000000 0x0000000000000000

Selector (Second):
ID:11(6), Ref:2, Type:1 (Compact), subtype:0, Symmetric-LB: Off, Target_id:0
   Key:FRR:Y, Balances:N, Locality:N/unicast, Type:Unicast, Size:2, flags:0x14, dist-mode-default

Weight Info (Selector's view): Current Weight = 65535, Target_id = 0   <<< CURRENT WEIGHT IS 65535
Idx Balance    Weight   Orig-Weight    Ifl     Session   Install
----- -------   -------   -----------   ------   -------   -------
    0       **     65535             1       **       336      Yes
    1       **     65535         16384       **       335      Yes

Unilist: 1
Unilist Table (2 entries): List flags 0x0000000000000000

Unilist core-facing-index: 343
Underlying ifl-index : 0

HFRR force ifl selector : NO
ECMP : NO

1087597     Label -              -                IPv6->MPLS            -     0 0x0000000000000001 0x0000000000000000
1126659     Label -              -                IPv6->MPLS             -     0 0x0000000000000001 0x0000000000000000

Weight Info: Current Weight = 1
   ID Balance Orig-Balance Weight Orig-Weight State     Install      Flags
----- ------- --------- ------ ----------- -------- ----------- -----
1087597        0        0       1            1    Active    Installed 0x00
1126659        0        0   16384        16384   Standby    Installed 0x00
Routing-table id: 0

labroot@router-RE0> start shell pfe network fpc0
NPC platform (1067Mhz MPC 8548 processor, 2048MB memory, 512KB flash)
NPC0(COLNY401MOX-RE0 vty)# show pfe bfd id 336
-------------------
SESSION ID: 336
-------------------
Session Status : DOWN    <<< SESSION DOWN
Session Version : 0x0
Number of NHs        : 1

NPC0(router-RE0 vty)# show pfe bfd id 335
-------------------
SESSION ID: 335
-------------------
Session Status : DOWN   <<< SESSION DOWN
Session Version : 0x0
Number of NHs        : 1

The following sequence illustrates the issue that causes the session IDs for Protocol Next Hops (PNHs) to go down and the unilist (ECMP)path get stuck causing traffic impact.

- BGP route with PNH1 is resolving over active route Rt1.
- Session-id 'S' is allocated for PNH is associated with Rt1 for IGP-FRR.
- Preserve nexthop hierarchy is enabled and FRR-Indirect nexthops exist for same PNH.
- When FRR-Indirect nexthop gets created, it is sent with '0' version-id. So in PFE version-id is 0 for most of the session-ids.
- At some point Rt1 becomes deleted, Rt2 takes over as the best route. Resolve re-resolves the PNH over Rt2. Rt2 gets session-id "S" and notifies session-UP.
- Rt1 gets readded again and becomes active. Session-id S gets associated Rt1. Rt1 sends session-id UP message.
- Due to Quick link flap Rt1 gets deleted again. Rt1 sends session-id Down. At this time, Rt2 is still not fully disassoicated with session-id S. So after re-resolution, resolve see 'S' is already associated with Rt2 and no further action takes place.

-- Another symptom of this issue is traffic being dropped, with the "egress PFE unspecified errors" counter increasing under PFE exceptions

NPC0(router-RE0 vty)# show jnh 0 exceptions terse
Reason                             Type           Packets      Bytes
==================================================================
PFE State Invalid
----------------------
unknown iif                        DISC( 1)            4        208
egress pfe unspecified             DISC( 19)     50839079 75140158762     <<<< COUNTER IS INCREASING RAPIDLY

Solution

PR1831421 has resolved this issue. The fix is not present in the given release. After implementing the fix, we did not observe the issue on the private image. The fix involved adding a new flag to the existing RPD infrastructure to detect these cases and trigger a forcible session-id UP message. Please note that from 21.4 (RLI 48120) onwards, a major redesign was implemented for this code area, eliminating the need for PR1831421.

Modification History

2024-09-19 : Article Created