Customer had a previous fiber event where they had to cost traffic away to mitigate a fiber cut and when the fiber event was resolved, customer costed traffic back to normal status. However, traffic was not passing through and causing an outage.
Data error counter incrementing at about 1 million iterations per second.
labroot@labrouter> show pfe statistics traffic | match "Data error :"
Data error : 904168977333
Data error : 904169139345
Data error : 904170028723
IRP.CFG_ERROR incrementing on PFEs across multiple FPCs.
labroot@labrouter> request pfe execute command "show pechip trapstats" target fpc0 | match irp.cfg_err
0 ( 168) irp.cfg_err 27463937 5
1 ( 168) irp.cfg_err 22661732 0
2 ( 168) irp.cfg_err 98921614906 87311
3 ( 168) irp.cfg_err 8511147705 7050
4 ( 168) irp.cfg_err 13811727052 8001
5 ( 168) irp.cfg_err 5269675527 3022
labroot@labrouter> request pfe execute command "show pechip trapstats" target fpc1 | match irp.cfg_err
0 ( 168) irp.cfg_err 26823460 3
1 ( 168) irp.cfg_err 22026223 0
2 ( 168) irp.cfg_err 98397099919 87880
3 ( 168) irp.cfg_err 8581295832 7102
4 ( 168) irp.cfg_err 13819674681 8335
5 ( 168) irp.cfg_err 5460958175 3289
Below logs indicate AE0 and AE302 UP/DOWN event due to fiber-cut events. The issue can come even when peer device reboots, or due to manual link flaps, etc.
Apr 18 05:54:22 2023 labrouter mib2d[20210]: SNMP_TRAP_LINK_DOWN: ifIndex 1357, ifAdminStatus up(1), ifOperStatus down(2), ifName ae302
Apr 18 05:54:22 2023 labrouter mib2d[20210]: SNMP_TRAP_LINK_DOWN: ifIndex 589, ifAdminStatus up(1), ifOperStatus down(2), ifName ae0
Apr 18 06:05:39 2023 labrouter mib2d[20210]: SNMP_TRAP_LINK_UP: ifIndex 589, ifAdminStatus up(1), ifOperStatus up(1), ifName ae0
Apr 18 06:07:25 2023 labrouter mib2d[20210]: SNMP_TRAP_LINK_UP: ifIndex 1357, ifAdminStatus up(1), ifOperStatus up(1), ifName ae302
Apr 18 06:54:27 2023 labrouter mib2d[20210]: SNMP_TRAP_LINK_DOWN: ifIndex 1357, ifAdminStatus up(1), ifOperStatus down(2), ifName ae302
Apr 18 06:54:28 2023 labrouter mib2d[20210]: SNMP_TRAP_LINK_DOWN: ifIndex 589, ifAdminStatus up(1), ifOperStatus down(2), ifName ae0
Apr 18 08:36:50 2023 labrouter mib2d[20210]: SNMP_TRAP_LINK_UP: ifIndex 1357, ifAdminStatus up(1), ifOperStatus up(1), ifName ae302
Apr 18 08:39:27 2023 labrouter mib2d[20210]: SNMP_TRAP_LINK_UP: ifIndex 589, ifAdminStatus up(1), ifOperStatus up(1), ifName ae0
...
Unilist mis-programming error on PFE
labroot@labrouter> show route 2002:1998:0:4::205
inet6.0: 181988 destinations, 646734 routes (180493 active, 1 holddown, 5187 hidden)
+ = Active Route, - = Last Active, * = Both
2002:1998:0:4::200/125
*[BGP/170] 1w6d 09:38:25, localpref 100, from 10.1.1.1
AS path: 11427 I, validation-state: unverified
> to fe80::36ed:1bff:fe1c:80a0 via ae0.0
to fe80::36ed:1bff:fe1c:80a7 via ae302.0
labroot@labrouter> show route forwarding-table destination 2002:1998:0:4::200
Routing table: default.inet6
Internet6:
Enabled protocols: Bridging,
Destination Type RtRef Next hop Type Index NhRef Netif
2002:1998:0:4::200/125user 0 indr 2097416 1086
ulst 2097213 629 >>> mis-programmed unilist next-hop
fe80::36ed:1bff:fe1c:80a0
ucst 1634 3 ae0.0
fe80::36ed:1bff:fe1c:80a7
ucst 1648 3 ae302.0
FPC0(labrouter vty)# show nhdb id 2097213 extensive
ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- ------------------ ------------------
2097213 Unilist ae0.0 - IPv4 Ethernet 0 0x0000000000000000 0x0000000000000000
BFD Session Id: 0
Platform Information:
=====================
Number of unilist cnhs: 2
Resilient routes refcount: 0
Unilist Sort Enabled: 0x1
Unilist Sorted: FALSE Unilist Reinstall: FALSE
List of member next-hops:
Index NHID
------- ----------
0 1634
1 1648
<Snipped>
Selector:
=========
ID:127883, Ref:1
Key:FRR/size = 2 agg_type: Unicast
Weight Info (Selector's view): Current Weight = 65535 >>> mis-programmed weight
Idx Balance Weight Orig-Weight Ifl Session Install
----- ------- ------- ----------- ------ ----- -------
0 ** 65535 1 69 329 Yes
1 ** 65535 1 154 385 Yes
Unilist Table (2 entries): List flags 0x0000000000000000
Unilist core-facing-index: 0
Underlying ifl-index : 0
HFRR force ifl selector : NO
ECMP : YES
1634 Unicast ae0.0 fe80::36ed:1bff:fe1c:80a0
IPv6 Ethernet 0 0x0000000000000000 0x0000002000000000
1648 Unicast ae302.0 fe80::36ed:1bff:fe1c:80a7
Weight Info: Current Weight = 65535
ID Balance Orig-Balance Weight Orig-Weight State Install Flags
----- ------- --------- ------ ----------- -------- ----------- -----
1634 0 0 65535 1 Replaced Rerouted 0x00 >> In a working scenario, “Replaced, Rerouted” should reflect as “Active Installed”
1648 0 0 65535 1 Replaced Rerouted 0x00 >> In a working scenario, “Replaced, Rerouted” should reflect as “Active Installed”
The Packet Forwarding Engine (PFE) maintains a list of unicast next-hops that are resolved over the logical interfaces, which are used in the creation of unilists. Whenever there is a change in the RIB (Routing Information Base), RPD (Routing Protocol Daemon) will keep track of all the logical interfaces. If there is a change in the state of any of the logical interfaces (e.g. a link going down), RPD creates a new unilist reflecting the change. Normal operations dictate that the new unilist will be used for forwarding and the old unilist will get deleted and not be used for forwarding anymore. In the problem state, the old unilist does not get deleted, and instead it gets re-used when the old unilist becomes viable again (i.e. the interface that went down changes state to UP). Due to the old unilist not getting deleted, PFE re-uses the stale unilist, where it should be using a new unilist.
Issue can be resolve by performing bouncing of impacted AE bundles locally or from remote side.