Description

When customer is running BGP over IGP for example OSPF then sometimes there is delay of 60 sec when connected link flaps.

Symptoms

customer is running BGP over IGP i.e OSPF then sometimes there is delay of 60 sec when connected link flaps.

Model affected :- ACX7024
VERSION:- 22.4-R2-EVO
  
USER1@R2# set interfaces et-0/0/0 disable
Sep 30 07:05:35

[edit]
USER1@R2# commit and-quit
Sep 30 07:05:37
commit complete
Exiting configuration mode

USER1@R2> show bgp summary | match 10.228.0.1 | refresh 20
Sep 30 07:05:44
---(refreshed at 2023-09-30 07:05:44 CEST)---
10.228.0.1            64700         12         10       0       4           2 Active
---(*more 100%)---[abort]

USER1@R2> configure
Sep 30 07:05:49
Entering configuration mode

[edit]
USER1@R2# rollback 1
Sep 30 07:05:52
load complete

[edit]
USER1@R2# commit and-quit
Sep 30 07:05:54
commit complete
Exiting configuration mode

USER1@R2> show bgp summary | match 10.228.0.1 | refresh 20
Sep 30 07:06:02
---(refreshed at 2023-09-30 07:06:02 CEST)---
10.228.0.1            64700          0          0       0       4          20 Connect
---(refreshed at 2023-09-30 07:06:22 CEST)---
10.228.0.1            64700          0          0       0       4          40 Connect
---(refreshed at 2023-09-30 07:06:42 CEST)---
10.228.0.1            64700          0          0       0       4        1:00 Connect
---(refreshed at 2023-09-30 07:07:02 CEST)---
10.228.0.1            64700          0          0       0       4        1:20 Connect
---(refreshed at 2023-09-30 07:07:22 CEST)---
10.228.0.1            64700          0          0       0       4        1:40 Connect
---(refreshed at 2023-09-30 07:07:42 CEST)---
10.228.0.1            64700          0          0       0       4        2:00 Connect
---(refreshed at 2023-09-30 07:08:02 CEST)---
10.228.0.1            64700         11         10       0       4           8 Establ
---(refreshed at 2023-09-30 07:08:22 CEST)---
10.228.0.1            64700         11         10       0       4          28 Establ


 

Solution

Issue is seen because of TCP backoff of underlying Linux kernel of EVO platform and this is expected behaviour in customer design and config.

Behaviour explanation:-

  • We observed that BGP peer IP is reachable though OSPF.

  • when link flaps it took 40 sec for OSPF to get into full.during this time BGP TCP tried to connect the peer with retry interval exponentially increasing.

In evo device, it exponential backop i.e 2^X (X=0,1,2,3,4,5,6) and then reset. this behaviour  is according to RFC RFC1122 section 4.2.3.1

notes:
This implementation also MUST include "exponential backoff"
for successive RTO values for the same segment.

  • OSPF was finally full and connectivity to peer BGP was also UP however, BGP TCP retry timer was 64 sec so it attempted the connection after ~ 60 sec which cause delay of BGP establishment of ~ 120 sec ( 40 sec ospf up time + 64 sec delay of BGP retry timer. )


Exponential backoff
 
root@ultron-pfe-p1b-12> monitor traffic interface re0:mgmt-0.0 matching "port 179"
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
02:56:03.869803 IP 10.228.0.2.35370 > 10.228.0.1.bgp: Flags [P.], seq 967427883:967427902, ack 1294458597, win 26888, options [nop,n
op,TS val 366301034 ecr 1198083828], length 19: BGP
02:56:07.197844 IP 10.228.0.2.35370 > 10.228.0.1.bgp: Flags [P.], seq 0:19, ack 1, win 26888, options [nop,nop,TS val 366304363 ecr
1198083828], length 19: BGP
02:56:13.854807 IP 10.228.0.2.35370 > 10.228.0.1.bgp: Flags [P.], seq 0:19, ack 1, win 26888, options [nop,nop,TS val 366311019 ecr
1198083828], length 19: BGP
02:56:27.165830 IP 10.228.0.2.35370 > 10.228.0.1.bgp: Flags [P.], seq 0:19, ack 1, win 26888, options [nop,nop,TS val 366324330 ecr
1198083828], length 19: BGP
02:56:53.789801 IP 10.228.0.2.35370 > 10.228.0.1.bgp: Flags [P.], seq 0:19, ack 1, win 26888, options [nop,nop,TS val 366350954 ecr 1198083828], length 19: BGP
02:57:03.950599 IP 10.228.0.2.35370 > 10.228.0.1.bgp: Flags [FP.], seq 19:78, ack 1, win 26888, options [nop,nop,TS val 366361115 ecr 1198083828], length 59: BGP
02:57:04.953511 IP 10.228.0.2.46226 > 10.228.0.1.bgp: Flags [S], seq 2570739081, win 32120, options [mss 1460,sackOK,TS val 366362118 ecr 0,nop,wscale 0], length 0
02:57:06.013799 IP 10.228.0.2.46226 > 10.228.0.1.bgp: Flags [S], seq 2570739081, win 32120, options [mss 1460,sackOK,TS val 366363178 ecr 0,nop,wscale 0], length 0
02:57:08.061832 IP 10.228.0.2.46226 > 10.228.0.1.bgp: Flags [S], seq 2570739081, win 32120, options [mss 1460,sackOK,TS val 366365227 ecr 0,nop,wscale 0], length 0
02:57:12.093837 IP 10.228.0.2.46226 > 10.228.0.1.bgp: Flags [S], seq 2570739081, win 32120, options [mss 1460,sackOK,TS val 366369259 ecr 0,nop,wscale 0], length 0
02:57:20.413817 IP 10.228.0.2.46226 > 10.228.0.1.bgp: Flags [S], seq 2570739081, win 32120, options [mss 1460,sackOK,TS val 366377579 ecr 0,nop,wscale 0], length 0
02:57:36.797798 IP 10.228.0.2.46226 > 10.228.0.1.bgp: Flags [S], seq 2570739081, win 32120, options [mss 1460,sackOK,TS val 366393962 ecr 0,nop,wscale 0], length 0
02:58:09.053853 IP 10.228.0.2.46226 > 10.228.0.1.bgp: Flags [S], seq 2570739081, win 32120, options [mss 1460,sackOK,TS val 366426219 ecr 0,nop,wscale 0], length 0
02:59:14.079615 IP 10.228.0.2.54512 > 10.228.0.1.bgp: Flags [S], seq 536567235, win 32120, options [mss 1460,sackOK,TS val 366491244 ecr 0,nop,wscale 0], length 0
02:59:15.101798 IP 10.228.0.2.54512 > 10.228.0.1.bgp: Flags [S], seq 536567235, win 32120, options [mss 1460,sackOK,TS val 366492266 ecr 0,nop,wscale 0], length 0

Please note that Junos platform implementation of TCP backoff is different.
  
retry timer is (3 sec,3,3,3,3,6,12,24,16) then same cycle repeat

 

Modification History

2023-11-22 : Article Created