Description

Customer was performing changes to establish an ELINE (l2circuit) connection over logical tunnel interfaces towards Remote PE's . But, right after the changes, they noticed that their protocol adjacencies are flapping. This article explains the customer issue and changes performed to resolve the issue

 

Customer topology

 

 

Symptoms

Message Logs

 

May 16 07:23:16 2024 re0.labrouter rpd[33259]: RPD_RSVP_NBRDOWN: RSVP neighbor 10.10.10.110 down on interface None nbr-type Node, neighbor timeout

May 16 07:23:16 2024 re0.labrouter rpd[33259]: RPD_RSVP_NBRDOWN: RSVP neighbor 10.10.10.106 down on interface None nbr-type Node, neighbor timeout

...

May 16 07:46:58 2024 re0.labrouter rpd[33259]: RPD_RSVP_NBRDOWN: RSVP neighbor 10.10.10.106 down on interface None nbr-type Node, neighbor timeout

May 16 07:47:52 2024 re0.labrouter rpd[33259]: RPD_RSVP_NBRDOWN: RSVP neighbor 10.10.10.110 down on interface None nbr-type Node, neighbor timeout

May 16 07:47:52 2024 re0.labrouter rpd[33259]: RPD_RSVP_NBRDOWN: RSVP neighbor 10.10.10.106 down on interface None nbr-type Node, neighbor timeout

 

 

May 16 07:23:17 2024 re0.labrouter rpd[33259]: RPD_ISIS_ADJDOWN: IS-IS lost L2 adjacency to re0.labrouter on lt-1/0/0.7, reason: 3-Way handshake failed

May 16 07:23:20 2024 re0.labrouter rpd[33259]: RPD_ISIS_ADJDOWN: IS-IS lost L1 adjacency to re0.labrouter on lt-1/0/0.5, reason: Interface Down

May 16 07:23:20 2024 re0.labrouter rpd[33259]: RPD_ISIS_ADJDOWN: IS-IS lost L1 adjacency to re0.labrouter on lt-1/0/0.3, reason: Interface Down

...

May 16 07:50:54 2024 re0.labrouter rpd[33259]: RPD_ISIS_LDP_SYNC: IS-IS interface lt-1/0/0.7 advertised with infinite metric for 180 seconds already due to loss of synchronization with LDP

May 16 08:00:17 2024 re0.labrouter rpd[33259]: RPD_ISIS_ADJDOWN: IS-IS lost L1 adjacency to re0.labrouter on lt-1/0/0.3, reason: Interface Down

May 16 08:00:17 2024 re0.labrouter rpd[33259]: RPD_ISIS_ADJDOWN: IS-IS lost L2 adjacency to re0.labrouter on lt-1/0/0.7, reason: Interface Down

May 16 08:03:17 2024 re0.labrouter rpd[33259]: RPD_ISIS_LDP_SYNC: IS-IS interface lt-1/0/0.3 advertised with infinite metric for 180 seconds already due to loss of synchronization with LDP

 

 

May 16 07:23:18 2024 re0.labrouter rpd[33259]: RPD_LDP_SESSIONDOWN: LDP session 10.10.10.141 is down, reason: received notification from peer

May 16 07:23:18 2024 re0.labrouter rpd[33259]: RPD_LDP_SESSIONDOWN: LDP session 10.10.10.110 is down, reason: received notification from peer

May 16 07:23:18 2024 re0.labrouter rpd[33259]: RPD_LDP_SESSIONDOWN: LDP session 10.10.10.140 is down, reason: received notification from peer

May 16 07:23:18 2024 re0.labrouter rpd[33259]: RPD_LDP_SESSIONDOWN: LDP session 10.10.10.106 is down, reason: received notification from peer

May 16 07:23:18 2024 re0.labrouter rpd[33259]: RPD_LDP_SESSIONDOWN: LDP session 10.10.10.167 is down, reason: received notification from peer

May 16 07:23:18 2024 re0.labrouter rpd[33259]: RPD_LDP_SESSIONDOWN: LDP session 10.10.10.42 is down, reason: received notification from peer

....

May 16 07:47:56 2024 re0.labrouter rpd[33259]: RPD_LDP_NBRDOWN: LDP neighbor 10.10.10.87 (lt-1/0/0.3) is down

May 16 07:47:56 2024 re0.labrouter rpd[33259]: RPD_LDP_NBRDOWN: LDP neighbor 10.10.10.167 (lt-1/0/0.7) is down

May 16 07:49:05 2024 re0.labrouter rpd[33259]: RPD_LDP_NBRDOWN: LDP neighbor 10.10.10.42 (lo0.0) is down

May 16 08:00:17 2024 re0.labrouter rpd[33259]: RPD_LDP_NBRDOWN: LDP neighbor 10.10.10.87 (lt-1/0/0.3) is down

May 16 08:00:17 2024 re0.labrouter rpd[33259]: RPD_LDP_NBRDOWN: LDP neighbor 10.10.10.167 (lt-1/0/0.7) is down

May 16 08:00:41 2024 re0.labrouter rpd[33259]: RPD_LDP_SESSIONDOWN: LDP session 10.10.10.202 is down, reason: hold time expired

 

 

May 16 07:23:19 2024 re0.labrouter rpd[33259]: RPD_LAYER2_VC_DOWN: State of Layer 2 VC (Neighbor : 10.10.10.141, VC-ID : 414538) changed from UP to DELETED

May 16 07:23:19 2024 re0.labrouter rpd[33259]: RPD_LAYER2_VC_DOWN: State of Layer 2 VC (Neighbor : 10.10.10.140, VC-ID : 412314) changed from UP to DELETED

May 16 07:23:19 2024 re0.labrouter rpd[33259]: RPD_LAYER2_VC_DOWN: State of Layer 2 VC (Neighbor : 10.10.10.167, VC-ID : 416734) changed from UP to DELETED

...

May 16 07:47:55 2024 re0.labrouter rpd[33259]: RPD_LAYER2_VC_DOWN: State of Layer 2 VC (Neighbor : 10.10.10.140, VC-ID : 412314) changed from UP to DELETED

May 16 07:47:55 2024 re0.labrouter rpd[33259]: RPD_LAYER2_VC_DOWN: State of Layer 2 VC (Neighbor : 10.10.10.167, VC-ID : 416734) changed from UP to DELETED

May 16 08:00:17 2024 re0.labrouter rpd[33259]: RPD_LAYER2_VC_DOWN: State of Layer 2 VC (Neighbor : 10.10.10.140, VC-ID : 412314) changed from UP to Dn

May 16 08:00:17 2024 re0.labrouter rpd[33259]: RPD_LAYER2_VC_DOWN: State of Layer 2 VC (Neighbor : 10.10.10.167, VC-ID : 416734) changed from UP to Dn

 

 

May 16 07:23:20 2024 re0.labrouter mib2d[37318]: SNMP_TRAP_LINK_DOWN: ifIndex 608, ifAdminStatus up(1), ifOperStatus down(2), ifName lt-1/0/0.4

May 16 07:23:20 2024 re0.labrouter mib2d[37318]: SNMP_TRAP_LINK_DOWN: ifIndex 610, ifAdminStatus up(1), ifOperStatus down(2), ifName lt-1/0/0.5

May 16 07:23:20 2024 re0.labrouter mib2d[37318]: SNMP_TRAP_LINK_DOWN: ifIndex 609, ifAdminStatus up(1), ifOperStatus down(2), ifName lt-1/0/0.2

May 16 07:23:20 2024 re0.labrouter mib2d[37318]: SNMP_TRAP_LINK_DOWN: ifIndex 594, ifAdminStatus up(1), ifOperStatus down(2), ifName lt-1/0/0.3

May 16 07:23:20 2024 re0.labrouter mib2d[37318]: SNMP_TRAP_LINK_DOWN: ifIndex 611, ifAdminStatus up(1), ifOperStatus down(2), ifName lt-1/0/0.6

May 16 07:23:20 2024 re0.labrouter mib2d[37318]: SNMP_TRAP_LINK_DOWN: ifIndex 612, ifAdminStatus up(1), ifOperStatus down(2), ifName lt-1/0/0.7

 

 

May 16 07:23:20 2024 re0.labrouter kernel: RLT:lt_ifl_peer_update:lt-1/0/0.4 lt-1/0/0.5 op 2

May 16 07:23:20 2024 re0.labrouter kernel: RLT:lt_ifl_peer_update:lt-1/0/0.2 lt-1/0/0.3 op 2

May 16 07:23:20 2024 re0.labrouter kernel: RLT:lt_ifl_peer_update:lt-1/0/0.6 lt-1/0/0.7 op 2

 

May 16 07:46:12 2024 re0.labrouter fpc1 MQSS(0): LI-2: Received a parcel with more than 512B accompanying data 

May 16 07:46:12 2024 re0.labrouter fpc1 Captured cookie (LI-2): C0: de 28 51 c0 00 08 06 14, SOP: 1, CTL: START C1: 05 b2 50 4b 40 28 04 0a, SOP: 0, CTL: MIDDLE C2: 38 27 00 40 03 f1 00 00, SOP: 0, CTL: MIDDLE C3: 9e 2a 00 f8 00 10 f2 18, SOP: 0, CTL: MIDDLE  

May 16 07:46:13 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 645 

May 16 07:46:13 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/645, Data = d02850200000000000000000000000000000 

May 16 07:46:14 2024 re0.labrouter fpc1 MQSS(0): LI-0: Received a parcel with more than 512B accompanying data 

May 16 07:46:14 2024 re0.labrouter fpc1 Captured cookie (LI-0): C0: de 41 c1 c0 00 08 06 14, SOP: 1, CTL: START C1: 05 b2 00 0c 00 28 04 0a, SOP: 0, CTL: MIDDLE C2: 38 22 00 02 20 18 04 00, SOP: 0, CTL: MIDDLE C3: 91 10 00 90 00 11 52 ca, SOP: 0, CTL: MIDDLE  

May 16 07:46:14 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1052 

May 16 07:46:14 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1052, Data = d041c0200000000000000000000000000000 

May 16 07:46:15 2024 re0.labrouter fpc1 MQSS(0): LI-0: Received a parcel with more than 512B accompanying data 

May 16 07:46:15 2024 re0.labrouter fpc1 Captured cookie (LI-0): C0: de 4b 31 c0 00 08 06 14, SOP: 1, CTL: START C1: 05 b0 40 15 00 28 04 0a, SOP: 0, CTL: MIDDLE C2: 38 20 40 00 0b 78 00 00, SOP: 0, CTL: MIDDLE C3: 85 a1 00 30 00 0d a6 ab, SOP: 0, CTL: MIDDLE  

May 16 07:46:15 2024 re0.labrouter fpc1 MQSS(0): LI-1: Received a parcel with more than 512B accompanying data 

May 16 07:46:15 2024 re0.labrouter fpc1 Captured cookie (LI-1): C0: de 46 21 c0 00 08 06 14, SOP: 1, CTL: START C1: 05 b5 08 02 00 28 04 0a, SOP: 0, CTL: MIDDLE C2: 38 20 70 40 03 f1 00 00, SOP: 0, CTL: MIDDLE C3: 9d da 00 f0 00 01 cd 63, SOP: 0, CTL: MIDDLE  

May 16 07:46:15 2024 re0.labrouter fpc1 MQSS(0): LI-2: Received a parcel with more than 512B accompanying data 

May 16 07:46:15 2024 re0.labrouter fpc1 Captured cookie (LI-2): C0: de 50 e1 c0 00 08 06 14, SOP: 1, CTL: START C1: 05 b5 50 0b 00 28 04 0a, SOP: 0, CTL: MIDDLE C2: 38 25 50 0b 00 18 04 00, SOP: 0, CTL: MIDDLE C3: 85 0b 00 30 00 11 45 46, SOP: 0, CTL: MIDDLE  

May 16 07:46:15 2024 re0.labrouter fpc1 MQSS(0): LI-3: Received a parcel with more than 512B accompanying data 

May 16 07:46:15 2024 re0.labrouter fpc1 Captured cookie (LI-3): C0: de 52 21 c0 00 08 06 14, SOP: 1, CTL: START C1: 05 b5 98 02 00 28 04 0a, SOP: 0, CTL: MIDDLE C2: 38 20 50 00 28 01 27 00, SOP: 0, CTL: MIDDLE C3: 91 78 00 90 00 20 4c eb, SOP: 0, CTL: MIDDLE  

May 16 07:46:15 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1294 

May 16 07:46:15 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1294, Data = d050e0200000000000000000000000000000 

May 16 07:46:17 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1203 

May 16 07:46:17 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1203, Data = d04b30200000000000000000000000000000 

May 16 07:46:18 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1351 

May 16 07:46:18 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1351, Data = d05470200000000000000000000000000000 

May 16 07:46:19 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1122 

May 16 07:46:19 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1122, Data = d04620200000000000000000000000000000 

May 16 07:46:20 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1112 

May 16 07:46:20 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1112, Data = d04580200000000000000000000000000000 

May 16 07:46:21 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1396 

May 16 07:46:21 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1396, Data = d05740200000000000000000000000000000 

May 16 07:46:22 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1314 

May 16 07:46:22 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1314, Data = d05220200000000000000000000000000000 

May 16 07:46:23 2024 re0.labrouter fpc1 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1253 

May 16 07:46:23 2024 re0.labrouter fpc1 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1253, Data = d04e50200000000000000000000000000000 

Solution

Issue was due to incorrect metric configured on Core facing interface. We corrected metric on Core facing interfaces from 1234567 to 65000.

 

  • Core interfaces (AE200/AE201) was configured with 1234567 and lt-* interfaces were configured with 520000. We suspected that the router receiving routes with better preference/metric via lt-* interface IGP session when l2circuit is established over them.
  •  
  • Due to the metric, whenever IGP adjacencies established over lt-* interface, route convergence happens and routes prefers exit via lt-* interfaces. This was causing LDP/RSVP keepalive loss on AE200/AE201 and sessions gets disconnected.  
  •  
  • LDP session down on core interfaces (AE200/AE201) cause l2circuit down event for lt-* interface and bring down ISIS adjacency built over lt-* interfaces. Since no other better route exists, router prefer path via AE200/AE201 when its loses ISIS adjacency over lt-* interfaces.
  •  
  • Then, Router will form l2circuit towards remote neighbors via lt-* interface, form ISIS adjacencies and repeat the above causing protocols flaps on lt-* interfaces

Modification History

2024-05-30 : Article Created