ACX7024X IBGP Flapping - Hold time expired
IBGP peers flapping over directly connected links.
Simplified topology
---ixia—ebgp(zayo-v4)------et-0/0/4—ACX7024x-1-et-0/0/3—mtu 9996-------ibgp-----et-0/0/3—mtu 9996—ACX7024x-2
It is seen that while ACX1 is redistributing 1.3m prefixes that the ibgp connection will flap due to bgp holddown timer expire using default values 90.
ACX1:et-0/0/3
set interfaces et-0/0/3 description "165Halsey-rt801b: et-0/0/3"
set interfaces et-0/0/3 mtu 9996
set interfaces et-0/0/3 unit 0 family inet address 10.60.4.18/31
set interfaces et-0/0/3 unit 0 family iso
ACX2: et-0/0/3
set interfaces et-0/0/3 description "1: et-0/0/3"
set interfaces et-0/0/3 unit 0 family inet address 10.60.4.19/31
Example
ACX1:
labroot@g22-46# run show bgp summary group IBGP-ISP-PEER
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 7 Peers: 8 Down peers: 6
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1300006 1300000 0 0 0 0
bgp.l2vpn.0
0 0 0 0 0 0
bgp.evpn.0
lsdist.0
inet.2
inet6.0
inet6flow.0
inet6.2
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.60.0.11 399589 14 36 1230622 13 1:01 Establ
inet.0: 0/6/5/0
bgp.l2vpn.0: 0/0/0/0
bgp.evpn.0: 0/0/0/0
lsdist.0: 0/0/0/0
inet.2: 0/0/0/0
inet6.0: 0/0/0/0
inet6.2: 0/0/0/0
[edit]
10.60.0.11 399589 12 29 904200 14 16 Establ
ACX2:
labroot@g22-47# run show bgp summary group IBGP-ISP-PEER
Groups: 2 Peers: 3 Down peers: 2
15762 15756 0 0 0 0
10.60.0.10 399589 22 11 0 9 1:28 Establ
inet.0: 15756/15762/15761/0
Groups: 2 Peers: 3 Down peers: 3
10.60.0.10 399589 22 11 0 10 1 Active
ACX2 syslog:
Nov 7 06:40:21 g22-47 rpd[13113]: bgp_io_mgmt_cb:3105: NOTIFICATION sent to 10.60.0.10 (Internal AS 399589): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 10.60.0.10 (Internal AS 399589), socket buffer sndcc: 0 rcvcc: 0, hold timer 90s, hold timer remain 0s, last sent 7s, TCP port (local 179, remote 39638)
On the ACX7xxxx platforms, MTU of 10k is now supported, giving the option to choose 9996 MTU under the IFD configuration stanza. However, this is only for data traffic (transient) and not for control traffic. When ACX1 is sending large BGP updates, they will fail to send causing congestion in the BGP out Queue resulting in keep alive drops causing the peer to time out.
Fix:
Use MTU values of 9198 or less on interfaces that require control traffic such as BGP/OSFP peers or use TCP MSS for BGP sessions.