BGP peers seem to randomly flap with the log entry below. An existing peer that has been up for over a year with no issues. No config changes have taken place on either side. After it flaps it stays up for 2-3 hours or less and then flaps again.
Apr 26 15:40:07 10.2.0.x rpd[18765]: %DAEMON-4-BGP_IO_ERROR_CLOSE_SESSION: BGP peer 10.3.64.x (External AS 6500x): Error event Operation timed out(60) for I/O session - closing it (instance 00811Vxxx)
Apr 26 15:33:25 routerMX rpd[18765]: %DAEMON-4: bgp_io_mgmt_cb:2361: NOTIFICATION sent to 10.3.114.x (External AS 6500x): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 10.3.114.x (External AS 6500x), socket buffer sndacc: 57 rcvacc: 0 , socket buffer sndccc: 57 rcvccc: 0 TCP state: 4, snd_una: 3472585405 snd_nxt: 3472585462 snd_wnd: 16384 rcv_nxt: 1739929910 rcv_adv: 1739946294, hold timer 90s, hold timer remain 0s, last sent 12s, TCP port (local 52072, remote 179), JSR handle (primary 17077649787980087297, secondary 17077649790094082049)
As per logs it indicates that neighboring devices stop responding at certain times and reason why holdtimer expired is recorded. ie:
BGP Neighbor - 10.3.64.xxx
Apr 26 15:40:07 routerMX rpd[18765]: %DAEMON-4-BGP_IO_ERROR_CLOSE_SESSION: BGP peer 10.3.64.xxx (External AS 6500x): Error event Operation timed out(60) for I/O session - closing it (instance 00811Vxxx)
Apr 26 15:40:07 routerMX rpd[18765]: %DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 10.3.64.xxx (External AS 6500x) changed state from Established to Idle (event HoldTime) (instance 00811Vxxx)
Apr 26 15:40:07 routerMX rpd[18765]: %DAEMON-4: bgp_io_mgmt_cb:2361: NOTIFICATION sent to 10.3.64.xxx (External AS 6500x): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 10.3.64.xxx (External AS 6500x), socket buffer sndacc: 76 rcvacc: 0 , socket buffer sndccc: 76 rcvccc: 0 TCP state: 4, snd_una: 3085568562 snd_nxt: 3085568619 snd_wnd: 16384 rcv_nxt: 2749456934 rcv_adv: 2749473318, hold timer 90s, hold timer remain 0s, last sent 2s, TCP port (local 60016, remote 179), JSR handle (primary 17726168134438879233, secondary 17726168136552873985)
Apr 26 15:40:26 routerMX rpd[18765]: %DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 10.3.64.xxx (External AS 6500x) changed state from EstabSync to Established (event RsyncAck) (instance 00811Vxxx)
BGP Neighbor - 10.3.114.xxx
Apr 26 15:33:25 routerMX rpd[18765]: %DAEMON-4-BGP_IO_ERROR_CLOSE_SESSION: BGP peer 10.3.114.xxx (External AS 65008): Error event Operation timed out(60) for I/O session - closing it (instance 00817Vxxx)
Apr 26 15:33:25 routerMX rpd[18765]: %DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 10.3.114.xxx (External AS 6500x) changed state from Established to Idle (event HoldTime) (instance 00817Vxxx)
Apr 26 15:33:25 routerMX rpd[18765]: %DAEMON-4: bgp_io_mgmt_cb:2361: NOTIFICATION sent to 10.3.114.xxx (External AS 6500x): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 10.3.114.xxx (External AS 6500x), socket buffer sndacc: 57 rcvacc: 0 , socket buffer sndccc: 57 rcvccc: 0 TCP state: 4, snd_una: 3472585405 snd_nxt: 3472585462 snd_wnd: 16384 rcv_nxt: 1739929910 rcv_adv: 1739946294, hold timer 90s, hold timer remain 0s, last sent 12s, TCP port (local 52072, remote 179), JSR handle (primary 17077649787980087297, secondary 17077649790094082049)
Apr 26 15:34:40 routerMX rpd[18765]: %DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 10.3.114.xxx (External AS 6500x) changed state from EstabSync to Established (event RsyncAck) (instance 00817Vxxx)
routerMX already sent 3x Keepalive packet to peer, but did not received TCP ack packet, so the 3x Keepalive packet still buffered in the TCP socket buffer, it is waiting for tcp/ack from peer for clearing the buffer.
Apr 26 15:33:25 routerMX rpd[18765]: %DAEMON-4: bgp_io_mgmt_cb:2361: NOTIFICATION sent to 10.3.114.xxx (External AS 65008): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 10.3.114.xxx (External AS 6500x), socket buffer sndacc: 57 <<<<<
(1x Keepalive packet is 19 byte x3=57byte)
Apr 26 15:40:07 routerMX rpd[18765]: %DAEMON-4: bgp_io_mgmt_cb:2361: NOTIFICATION sent to 10.3.64.xxx (External AS 65008): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 10.3.64.xxx (External AS 6500x), socket buffer sndacc: 76 <<
(1x Keepalive packet is 19 byte x4=76byte)
So as conclusion,
root# set interface ge-x/x/x disable
root# delete interface ge-x/x/x disable
If performing BGP interface bounce the issue persists it is strongly recommended to check if there are connectivity issue on the link, in between the link or BGP neighbor. From the MX logs perspective the EBGP peer is not replying to the TCP keepalive Ack packets when these logs are occurring. on this example the peer side was causing it’s ARP table to constantly be flushed causing BGP to flap so the issue was not this routerMX.