Customer reported that IPv6 IBGP neighbors flapping with holdtime expired error despite igp route to neighbor being stable.
IPv6 IBGP neighbors flapping due to holdtime expired error. We also observed logs related to NSR being reported
Oct 2 09:35:45.103 2024 router rpd[40432]: %DAEMON-4: bgp_handle_notify:5240: NOTIFICATION received from 2400:9800:7008:a425:: (Internal AS 24208): code 4 (Hold Timer Expired Error), socket buffer sndacc: 0 rcvacc: 0 , socket buffer sndccc: 0 rcvccc: 0 TCP state: 5, snd_una: 723445224 snd_nxt: 723445224 snd_wnd: 30561 rcv_nxt: 187912421 rcv_adv: 187929468, hold timer 90s, hold timer remain 0s, last sent 90s, TCP port (local 50145, remote 179)
<<SNIP>>
Oct 2 09:35:51.847081 BGP SEND message type 4 (KeepAlive) length 19
Oct 2 09:35:51.886686 BGP RECV message type 4 (KeepAlive) length 19
Oct 2 09:36:28.685625 BGP RECV message type 4 (KeepAlive) length 19
Oct 2 09:36:58.675052 BGP RECV message type 4 (KeepAlive) length 19
Oct 2 09:37:58.674973 BGP RECV message type 4 (KeepAlive) length 19
Oct 2 09:38:27.296982 BGP RECV message type 3 (Notification) length 21
Oct 2 09:38:27.296990 BGP RECV Notification code 4 (Hold Timer Expired Error) subcode 0 (unused)
Oct 2 09:38:27.325 2024 router rpd[40432]: %DAEMON-4: bgp_handle_notify:5240: NOTIFICATION received from 1234:5678:1234:5678:: (Internal AS 12345): code 4 (Hold Timer Expired Error), socket buffer sndacc: 0 rcvacc: 0 , socket buffer sndccc: 0 rcvccc: 0 TCP state: 5, snd_una: 2754915788 snd_nxt: 2754915788 snd_wnd: 30637 rcv_nxt: 3039075112 rcv_adv: 3039092180, hold timer 90s, hold timer remain 0s, last sent 90s, TCP port (local 53072, remote 179)
Oct 2 12:09:23.756 2024 router kernel: %KERN-3: jsr_unreplicate: INFO: unreplicating handle 0x100ec1d0000001e, laddr 2400:9800:7000:309f::, lport 55264, faddr 1234:5678:1234:5678::, fport 179, rtb_idx 0, due to error 0, msg Application asked for unreplication
1) From the bgp replication logs, we noticed that the rsync session between the RE's was flapped by the backup RE
root@router> show bgp replication
Oct 02 11:13:05
Synchronization master:
Precision timer registration: NotRegistered
Session state: Up, Since: 16
Flaps: 1879, Last flap reason: Backup closed connection
Protocol state: RIBSync, Since: 16, Flags: WriteSet
Synchronization state: InProgress
Number of peers waiting: AckWait: 0, SoWait: 0, Scheduled: 23
Send buffer: 2505
rsync walk at: bgp.evpn.0 2:24208:1289::2386::6c:d7:19:db:d2:41/304, queued 1000
Messages sent: Open 1, Establish 28, GrHelper 0, Update 309, GrStaleLabel 0 Error 0, Complete 0
Messages received: Open 1, Request 1 wildcard 0 targeted, EstablishAck 28, GrHelperAck 0, CompleteAck 0
2) Deactivating NSR stopped the BGP flapping
3) Enabled the below BGP Traceoptions under BGP and for the specific neighbor which was flapping.
set protocols bgp traceoptions file bgp.log size 1g set protocols bgp traceoptions flag nsr-synchronization detail set protocols bgp traceoptions flag state set protocols bgp traceoptions flag task set protocols bgp traceoptions flag timer 4) Collected RPD live core dump from both Master and Backup RE 5) Filed PR1840929 to engage engineering 6) Engineering identified that the DUT is receiving an ipv6 unicast route from its neighbor (with SRv6 TLVs encoded) and is advertising the same route to another router through family inet6-labeled unicast(which is currently not supported). 7)When standby RE is parsing rsync RIB-OUT update messages of NLRI type inet/inet6 labeled-unicast containing SRv6 TLVs encoded and if it is not able to find the accept-srv6-service knob configured and the label index TLV, then we are treating the update as malformed. In older releases where error tolerance behaviour is not present by default and if the error tolerance knob is not configured, then we are treating it as fatal and flapping the rsync session. This keeps happening repeatedly causing the I/O of the peers in master RE to get disabled/re-enabled continuously thereby missing out on sending keepalives in regular intervals.
8) Customer configured bgp-error-tolerance, post which the BGP flapping issue with NSR configured was resolved
Document reg BGP error tolerance:
https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/bgp-error-tolerance.html