[MX/QFX/PTX] [ALL Junos] MPLS LSP self ping time outs

Description

LSP self ping timeout on QFX

Feb 29 08:39:46 rpd[8320]: RPD_MPLS_LSP_SELFPING_TIMEOUT: MPLS LSP Self-ping timeout for LSP-02, duration: 1800 s
Feb 29 08:40:09 rpd[8320]: RPD_MPLS_LSP_SELFPING_TIMEOUT: MPLS LSP Self-ping timeout for LSP-04, duration: 1800 s
Feb 29 08:40:43 rpd[8320]: RPD_MPLS_LSP_SELFPING_TIMEOUT: MPLS LSP Self-ping timeout for LSP-01, duration: 1800 s
Feb 29 08:40:51 rpd[8320]: RPD_MPLS_LSP_SELFPING_TIMEOUT: MPLS LSP Self-ping timeout for LSP-03, duration: 1800 s

Symptoms

Feb 29 08:39:46 rpd[8320]: RPD_MPLS_LSP_SELFPING_TIMEOUT: MPLS LSP Self-ping timeout for LSP-02, duration: 1800 s
Feb 29 08:40:09 rpd[8320]: RPD_MPLS_LSP_SELFPING_TIMEOUT: MPLS LSP Self-ping timeout for LSP-04, duration: 1800 s
Feb 29 08:40:43 rpd[8320]: RPD_MPLS_LSP_SELFPING_TIMEOUT: MPLS LSP Self-ping timeout for LSP-01, duration: 1800 s
Feb 29 08:40:51 rpd[8320]: RPD_MPLS_LSP_SELFPING_TIMEOUT: MPLS LSP Self-ping timeout for LSP-03, duration: 1800 s

Solution

1. During investigation, I saw firewall filter on lo0 in RSI. Filter was looking for a match with a prefix-list, protocol and port number. If the received packet matches all three criteria, then it will be accepted.

lo0 {
unit 0 {
family inet {
filter {
input PROTECT-ROUTING-ENGINE_v4;
}
address 95.100.192.5/32;
}
family iso {
address 49.6963.1002.0951.0019.2005.00;
}
family inet6 {
filter {
input PROTECT-ROUTING-ENGINE_v6;
}
address 2600:1488:5fc1::5/128;
}
}
}

   firewall {
       family inet {
           filter PROTECT-ROUTING-ENGINE_v4 {
               term ALLOW-LSP-SELF-PING {
                   from {
                       source-prefix-list {
                           ICN-PREFIXES;
                       }
                       protocol udp;
                       destination-port 8503;
                   }
                   then accept;
               }

       prefix-list ICN-PREFIXES {
           23.32.62.0/23;
           23.193.112.0/23;
           23.197.64.0/20;
           23.214.115.0/24;
           23.215.54.0/24;
           95.100.192.0/24;
           104.119.40.0/21;
           104.119.40.0/24;
       }

2. After removing the filter, we could see LSP self ping working fine. This helps us understand why the packet received on QFX is getting dropped as it is not matching the filter.

3. We replicated the issue in JTAC LAB with above topology. Configured both routers with RSI. During the problem state, I ran pcaps and could see as below on ae1 at QFX side.
4. As per RFC, LSP self ping packet will be a UDP packet with a source and destination port of 8503 and a source and destination IP as the router ID.
5. When EVO PTX is headend for LSP from PTX ----> to QFX, LSP self-ping packet is received as expected, with the source IP: QFX router ID and the destination IP: EVO PTX router ID. Below is the pcap from EVO PTX.

6. When QFX is headend for LSP from QFX----> to PTX, LSP self ping packet is received, and it should have a source IP: PTX router ID and destination IP: QFX router ID but instead source IP was filled with its local ae1 interface address: 100.1.1.2. Please find below pcap snapshot.

start shell user root
tcpdump -i ae1 -s 65535 -w QFX.pcap

7. Prefix-list: ICN-PREFIXES contains all the router IDs, not the interface IP. The source IP field had a interface IP and not a router ID; hence, this packet was getting dropped, and LSP self ping was not successful.
8. After adding local interface IP: 100.1.1.2/32 in the prefix-list ICN-PREFIXES, we could see LSP self ping successful.

QFX# run show mpls lsp name LSP-TO-PTX extensive

Ingress LSP: 5 sessions

2.1.1.1
From: 2.1.1.2, State: Up, ActiveRoute: 0, LSPname: LSP-TO-PTX, LSPid: 17
ActivePath: (primary)
Link protection desired
LSPtype: Static Configured, Penultimate hop popping
LoadBalance: Least-fill
Follow destination IGP metric
Autobandwidth
MinBW: 200kbps
AdjustTimer: 300 secs AdjustThreshold: 1%
Max AvgBW util: 0bps, Bandwidth Adjustment in 281 second(s).
Overflow limit: 0, Overflow sample count: 0
Underflow limit: 0, Underflow sample count: 0, Underflow Max AvgBW: 0bps
Encoding type: Packet, Switching type: Packet, GPID: IPv4
LSP Self-ping Status : Enabled
*Primary State: Up
Priorities: 7 0
Bandwidth: 200kbps
OptimizeTimer: 1800
SmartOptimizeTimer: 180
Flap Count: 4
MBB Count: 0
Reoptimization in 1752 second(s).
Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 10)
100.1.1.1 S
Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt 20=Node-ID):
2.1.1.1(flag=0x20) 100.1.1.1(Label=3)
54 Mar 8 12:23:49.147 Selected as active path
53 Mar 8 12:23:49.146 Up
52 Mar 8 12:23:49.145 Self-ping started --------------------------> Self-ping started but was not successful due to filter dropping the UDP packet
51 Mar 8 12:23:49.145 Self-ping enqueued

QFX# set policy-options prefix-list ICN-PREFIXES 100.1.1.2/32 -----------------> Adding the local interface IP to prefix-list on QFX

QFX# commit

QFX# run show mpls lsp name LSP-TO-PTX extensive

Ingress LSP: 5 sessions

2.1.1.1
From: 2.1.1.2, State: Up, ActiveRoute: 0, LSPname: LSP-TO-PTX, LSPid: 17
ActivePath: (primary)
Link protection desired
LSPtype: Static Configured, Penultimate hop popping
LoadBalance: Least-fill
Follow destination IGP metric
Autobandwidth
MinBW: 200kbps
AdjustTimer: 300 secs AdjustThreshold: 1%
Max AvgBW util: 0bps, Bandwidth Adjustment in 257 second(s).
Overflow limit: 0, Overflow sample count: 0
Underflow limit: 0, Underflow sample count: 0, Underflow Max AvgBW: 0bps
Encoding type: Packet, Switching type: Packet, GPID: IPv4
LSP Self-ping Status : Enabled
*Primary State: Up
Priorities: 7 0
Bandwidth: 200kbps
OptimizeTimer: 1800
SmartOptimizeTimer: 180
Flap Count: 4
MBB Count: 0
Reoptimization in 1729 second(s).
Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 10)
100.1.1.1 S
Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt 20=Node-ID):
2.1.1.1(flag=0x20) 100.1.1.1(Label=3)
55 Mar 8 12:24:29.853 Self-ping ended successfully -------------------------> Self-ping is successful after adding prefix
54 Mar 8 12:23:49.147 Selected as active path
53 Mar 8 12:23:49.146 Up
52 Mar 8 12:23:49.145 Self-ping started
51 Mar 8 12:23:49.145 Self-ping enqueued
50 Mar 8 12:23:49.145 Record Route: 2.1.1.1(flag=0x20) 100.1.1.1(Label=3)
49 Mar 8 12:23:49.140 LSP-ID: 5 created

9. I tested MX router and found the same behaviour as QFX.

Root-Cause:-

RPD sends ?tunnel destination address (remote node?s router id)? as source address to kernel for both JUNOS and EVO. In JUNOS, cmsg type used for source address is IP_SENDSRCADDR. JUNOS kernel validates the source address received from application for cmsg type IP_SENDSRCADDR and if source address doesn?t belongs to local node?s address then JUNOS kernel ignores it and uses the packet?s outgoing interface address as source address. Hence issue is seen in case of JUNOS.

Workaround:-

If firewall filter is configured and applied on the loopback with prefix list to allow only certain prefixes, then add the interface address of the local router to the prefix-list.

Modification History

2024-03-28 : Article Created