This article explains the scenario where an active bypass LSP can prevent the BFD session to establish. The BFD session will stay down unless the ingress router re-routes the tunnel via alternate path.
This article describes an example using "optimize-hold-dead-delay" to induce delay in ingress router and keep bypass LSP active for longer time.
Topology:
2.2.2.2 3.3.3.3 ge-0/0/1 +------+ge-0/0/3 ge-0/0/1 +------+ge-0/0/3 +------+ | R2 +---------------------+ R3 +----------------+ R4 | 4.4.4.4 +------+.1 .2 +------+.21 .22+------+ .13|ge-0/0/8 ge-0/0/9 |.18 | | | | WAN - 10.1.1/ | ge-0/0/8 ge-0/0/9 | | +-------+ | +-------+ R7 +--------+ .14+-------+.17 7.7.7.7
We have established an LSP successfully between R2 and R4. This LSP has node-link protection enabled and has only one feasible path. There is a BFD session between R3 and R4. All interfaces have default 1g bandwidth. Interfaces between R3 and R4 have a BFD session up and running.
Relevant Config Snippet:
R2:
set protocols mpls label-switched-path test to 4.4.4.4 set protocols mpls label-switched-path test optimize-hold-dead-delay 65535 set protocols mpls label-switched-path test bandwidth 600m set protocols mpls label-switched-path test admin-group exclude deny set protocols mpls label-switched-path test node-link-protection
root@R2_re> show mpls lsp extensive Ingress LSP: 1 sessions 4.4.4.4 From: 2.2.2.2, State: Up, ActiveRoute: 0, LSPname: test ActivePath: (primary) Node/Link protection desired LSPtype: Static Configured, Penultimate hop popping LoadBalance: Random Encoding type: Packet, Switching type: Packet, GPID: IPv4 *Primary State: Up Priorities: 7 0 Bandwidth: 600Mbps OptimizeTimer: 900 SmartOptimizeTimer: 180 Exclude: deny Reoptimization in 887 second(s). Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 20) 10.1.1.2 S 10.1.1.22 S Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt 20=Node-ID): 3.3.3.3(flag=0x21) 10.1.1.2(flag=1 Label=299792) 4.4.4.4(flag=0x20) 10.1.1.22(Label=3) 10 Jan 19 16:39:24.199 Self-ping ended successfully 9 Jan 19 16:39:23.899 Record Route: 3.3.3.3(flag=0x21) 10.1.1.2(flag=1 Label=299792) 4.4.4.4(flag=0x20) 10.1.1.22(Label=3) 8 Jan 19 16:39:23.152 Selected as active path 7 Jan 19 16:39:23.152 Record Route: 3.3.3.3(flag=0x20) 10.1.1.2(Label=299792) 4.4.4.4(flag=0x20) 10.1.1.22(Label=3) 6 Jan 19 16:39:23.151 Up 5 Jan 19 16:39:23.151 Self-ping started 4 Jan 19 16:39:23.151 Self-ping enqueued 3 Jan 19 16:39:23.114 Originate Call 2 Jan 19 16:39:23.114 CSPF: computation result accepted 10.1.1.2 10.1.1.22 1 Jan 19 16:38:53.912 CSPF failed: no route toward 4.4.4.4 Created: Fri Jan 19 16:38:25 2018
R3:
At R3, bypass LSP is in UP state but not in use. There is a hidden route for the /32 WAN IP pointing to bypass LSP.
root@R3_re> show mpls lsp Ingress LSP: 0 sessions Total 0 displayed, Up 0, Down 0 Egress LSP: 1 sessions To From State Rt Style Labelin Labelout LSPname 3.3.3.3 2.2.2.2 Up 0 1 SE 3 - Bypass->10.1.1.2 Total 1 displayed, Up 1, Down 0 Transit LSP: 1 sessions To From State Rt Style Labelin Labelout LSPname 4.4.4.4 2.2.2.2 Up 0 1 SE 299792 3 test Total 1 displayed, Up 1, Down 0 root@R3_re> show mpls lsp bypass Ingress LSP: 1 sessions To From State Rt Style Labelin Labelout LSPname 4.4.4.4 3.3.3.3 Up 0 1 SE - 299776 Bypass->10.1.1.22 Total 1 displayed, Up 1, Down 0 Egress LSP: 0 sessions Total 0 displayed, Up 0, Down 0 Transit LSP: 0 sessions Total 0 displayed, Up 0, Down 0 root@R3_re> show bfd session Detect Transmit Address State Interface Time Interval Multiplier 10.1.1.22 Up ge-0/0/3.0 1.500 0.500 3 1 sessions, 1 clients Cumulative transmit rate 2.0 pps, cumulative receive rate 2.0 pps root@R3_re> show route 10.1.1.22 hidden inet.0: 21 destinations, 21 routes (20 active, 0 holddown, 1 hidden) + = Active Route, - = Last Active, * = Both 10.1.1.22/32 [RSVP] 00:00:37, metric 1 > to 10.1.1.14 via ge-0/0/8.0, label-switched-path Bypass->10.1.1.22 root@R3_re> show route 10.1.1.22 inet.0: 21 destinations, 21 routes (20 active, 0 holddown, 1 hidden) + = Active Route, - = Last Active, * = Both 10.1.1.20/30 *[Direct/0] 22:59:18 > via ge-0/0/3.0
Now, the link between R3 and R4 is shut (ge-0/0/3 interface at R3) to kick in bypass LSP. Bypass LSP becomes active and the hidden /32 route now becomes active. WAN IP route now is pointing to bypass LSP. This is "/32" route which is more specific than "/30" or "/31" of WAN subnet.
root@R3_re> show mpls lsp bypass Ingress LSP: 1 sessions To From State Rt Style Labelin Labelout LSPname 4.4.4.4 3.3.3.3 BackupActive 0 1 SE - 299776 Bypass->10.1.1.22 Total 1 displayed, Up 1, Down 0 Egress LSP: 0 sessions Total 0 displayed, Up 0, Down 0 Transit LSP: 0 sessions Total 0 displayed, Up 0, Down 0 root@R3_re> show route 10.1.1.22 inet.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.1.1.22/32 *[RSVP/7/1] 00:00:32, metric 1 > to 10.1.1.14 via ge-0/0/8.0, label-switched-path Bypass->10.1.1.22
Ingress router could not re-route the LSP due to CSPF failure.
root@R2_re> show mpls lsp extensive Ingress LSP: 1 sessions 4.4.4.4 From: 2.2.2.2, State: Up, ActiveRoute: 0, LSPname: test ActivePath: (primary) Node/Link protection desired LSPtype: Static Configured, Penultimate hop popping LoadBalance: Random Encoding type: Packet, Switching type: Packet, GPID: IPv4 *Primary State: Up Priorities: 7 0 Bandwidth: 600Mbps OptimizeTimer: 900 SmartOptimizeTimer: 180 Exclude: deny Reoptimization in 142 second(s). Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 20) 10.1.1.2 S 10.1.1.22 S Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt 20=Node-ID): 3.3.3.3(flag=0x23) 10.1.1.2(flag=3 Label=299792) 10.1.1.18(Label=3) 25 Jan 19 16:43:02.492 Link-protection Up 24 Jan 19 16:43:01.992 10.1.1.18: Session preempted 23 Jan 19 16:43:01.968 CSPF failed: no route toward 4.4.4.4 22 Jan 19 16:43:01.968 10.1.1.2: Tunnel local repaired 21 Jan 19 16:43:01.967 Record Route: 3.3.3.3(flag=0x23) 10.1.1.2(flag=3 Label=299792) 10.1.1.18(Label=3) 20 Jan 19 16:43:01.947 Link-protection Down 19 Jan 19 16:43:01.904 CSPF failed: no route toward 4.4.4.4 18 Jan 19 16:43:01.904 10.1.1.2: Tunnel local repaired[2 times] 17 Jan 19 16:42:56.811 CSPF failed: no route toward 4.4.4.4
Now, restore the failed link. Ingress router waits for optimize hold delay before tearing the bypass LSP, hence bypass LSP is still active.
root@R2_re> show mpls lsp extensive Ingress LSP: 1 sessions 4.4.4.4 From: 2.2.2.2, State: Up, ActiveRoute: 0, LSPname: test ActivePath: (primary) Node/Link protection desired LSPtype: Static Configured, Penultimate hop popping LoadBalance: Random Encoding type: Packet, Switching type: Packet, GPID: IPv4 *Primary State: Up Priorities: 7 0 Bandwidth: 600Mbps OptimizeTimer: 900 SmartOptimizeTimer: 180 Exclude: deny Reoptimization in 787 second(s). Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 20) 10.1.1.2 S 10.1.1.22 S Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt 20=Node-ID): 3.3.3.3(flag=0x21) 10.1.1.2(flag=1 Label=299808) 4.4.4.4(flag=0x20) 10.1.1.22(Label=3) 46 Jan 19 16:46:12.112 Pending old path instance deletion 45 Jan 19 16:45:04.487 Link-protection Up 44 Jan 19 16:45:03.691 Link-protection Down 43 Jan 19 16:45:01.490 Link-protection Up 42 Jan 19 16:45:00.193 Make-before-break: Switched to new instance 41 Jan 19 16:45:00.193 Link-protection Down 40 Jan 19 16:45:00.192 Self-ping ended successfully 39 Jan 19 16:44:59.913 Record Route: 3.3.3.3(flag=0x21) 10.1.1.2(flag=1 Label=299808) 4.4.4.4(flag=0x20) 10.1.1.22(Label=3) 38 Jan 19 16:44:59.691 Record Route: 3.3.3.3(flag=0x20) 10.1.1.2(Label=299808) 4.4.4.4(flag=0x20) 10.1.1.22(Label=3) 37 Jan 19 16:44:59.691 Up 36 Jan 19 16:44:59.690 Self-ping started 35 Jan 19 16:44:59.690 Self-ping enqueued 34 Jan 19 16:44:59.659 Originate make-before-break call 33 Jan 19 16:44:59.659 CSPF: computation result accepted 10.1.1.2 10.1.1.22 32 Jan 19 16:44:59.659 10.1.1.2: Tunnel local repaired 31 Jan 19 16:44:55.536 CSPF failed: no route toward 4.4.4.4
[edit] root@R3_re# show |compare [edit interfaces ge-0/0/3] - disable;
The failed link ge-0/0/3 is restored, but the LSP "test" is still riding on the bypass LSP. We can see that WAN route is still pointing to bypass LSP and BFD stays down in "Init" state.
root@R3_re> show mpls lsp bypass Ingress LSP: 1 sessions To From State Rt Style Labelin Labelout LSPname 4.4.4.4 3.3.3.3 BackupActive 0 1 SE - 299776 Bypass->10.1.1.22 Total 1 displayed, Up 1, Down 0 Egress LSP: 0 sessions Total 0 displayed, Up 0, Down 0 Transit LSP: 0 sessions Total 0 displayed, Up 0, Down 0 root@R3_re> show route 10.1.1.22 inet.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.1.1.22/32 *[RSVP/7/1] 00:00:32, metric 1 > to 10.1.1.14 via ge-0/0/8.0, label-switched-path Bypass->10.1.1.22 root@R3_re> show bfd session Detect Transmit Address State Interface Time Interval Multiplier 10.1.1.22 Init ge-0/0/3.0 6.000 2.000 3 1 sessions, 1 clients Cumulative transmit rate 0.5 pps, cumulative receive rate 0.5 pps root@R3_re>
BFD packets generated by R3 follows the forwarding table and will reach R4 via bypass LSP. It will be rejected by R4 due to TTL.
root@R4_re> show log bfdd |match ttl |last 3 Jan 19 16:46:24 PPM Trace: BFD packet ignored: invalid TTL 254 (expected 255 min. for this single-hop session) Jan 19 16:46:26 PPM Trace: BFD packet ignored: invalid TTL 254 (expected 255 min. for this single-hop session) Jan 19 16:46:28 PPM Trace: BFD packet ignored: invalid TTL 254 (expected 255 min. for this single-hop session)
Conclusion:
This is expected behavior in the event of Ingress router waiting for optimize-hold-dead-delay or unable to find a feasible path to re-route the LSP post link restoration. Bypass LSP being in active state post link down is an interim state and should go away once the ingress router finds an alternate path. If the Bypass LSP stays active even after link restoration, then it has to be troubleshooted further from the ingress node for the failure.