-Customer made catastrophic BGP changes on back-to-back connected PTX (PTX-1 and PTX-2) devices. (IBGP running between the PTX)

-Catastrophic changes made by the customer here: Adding BGP add-path and BGP PIC (Prefix Independent Convergence configuration). This will terminate the existing BGP session and establish the new structure.

-During such a change, the expected behaviour is a BGP Flap between DUT (PTX-1) and remote peer (PTX-2)

-But the issue is, the customer observed multiple BGP flaps on the remote peer.

Changes made by customer:

set protocols bgp group <bgp_ipv4_group> family inet unicast add-path receive

set protocols bgp group <bgp_ipv4_group> family inet unicast add-path send path-count 2

set protocols bgp group <bgp_ipv4_group> family inet labeled-unicast add-path receive

set protocols bgp group <bgp_ipv4_group> family inet labeled-unicast add-path send path-count 2

set protocols bgp group <bgp_ipv6_group> family inet6 unicast add-path receive

set protocols bgp group <bgp_ipv6_group> family inet6 unicast add-path send path-count 2

set routing-options protect core

set routing-options rib inet.3 protect core

set routing-options rib inet6.0 protect core

After the changes were pushed below observation was seen on the production routers:

Flapping neighbors are seen on the PTX-2 remote peer. (removed all the other outputs):

root@PTX-2-re0> show bgp summary | no-more

Peer                 AS      InPkt     OutPkt    OutQ   Flaps    Last Up/Dwn   State|#Active/Received/Accepted/Damped...

x-x-x-x             xxxxx    198208    263145    0      23      45:27:00       Establ =====>FLAPS seens before establishing the IBGP session with the PTX-1 DUT

y-y-y-y(ipv6)      xxxxx     114150    107696   0       1      51:05:00        Establ

Not seen in PTX-1 DUT(removed all the other outputs):

root@PTX-1> show bgp summary | no-more

Peer                  AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn   State|#Active/Received/Accepted/Damped...

x-x-x-x               xxxxx    263148    198210      0       0       45:30   Establ

u.u.u.u (ipv6)       xxxxx    107726    114180      0       0       51:09   Establ

BGP logs:

Log messages DUT:

Jul 15 06:50:34.512 PTX-1-re0-DUT rpd[30507]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event PeerConfigDeletion) (instance master)

Jul 15 06:50:34.516 PTX-1-re1-DUT rpd[3074]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event Closed) (instance master)

Jul 15 06:50:34.541 PTX-1-re0-DUT rpd[30507]: bgp_peer_delete:13390: NOTIFICATION sent to x.x.x.x (Internal AS xxxxx): code 6 (Cease) subcode 9 (Hard Reset) [code 6 (Cease) subcode 3 (Peer Unconfigured)], Reason: Peer Deletion

Jul 15 06:50:37.797 PTX-1-re0-DUT rpd[30507]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:50:45.410 PTX-1-re0-DUT rpd[30507]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:50:48.966 PTX-1-re0-DUT rpd[30507]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:50:51.921 PTX-1-re0-DUT rpd[30507]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:00.986 PTX-1-re0-DUT rpd[30507]: bgp_peer_established:11776: NOTIFICATION sent to x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:10.151 PTX-1-re0-DUT rpd[30507]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:23.973 PTX-1-re0-DUT rpd[30507]: bgp_peer_established:11776: NOTIFICATION sent to x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:33.188 PTX-1-re0-DUT rpd[30507]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:35.441 PTX-1-re0-DUT rpd[30507]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+44486 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:41.752 PTX-1-re0-DUT rpd[30507]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+37920 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:43.955 PTX-1-re0-DUT rpd[30507]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+37936 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:45.909 PTX-1-re0-DUT rpd[30507]: bgp_peer_established:11776: NOTIFICATION sent to x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:51.718 PTX-1-re0-DUT rpd[30507]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+38336 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:53.771 PTX-1-re0-DUT rpd[30507]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+38350 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:55.874 PTX-1-re0-DUT rpd[30507]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+38352 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:58.637 PTX-1-re1-DUT rpd[3074]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Idle to Established (event RecvEstab) (instance master)

Jul 15 06:51:59.003 PTX-1-re0-DUT rpd[30507]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from EstabSync to Established (event RsyncAck) (instance master)

Logs messages Remote-Peer (flaps seen):

Jul 15 06:50:34.561 PTX-2-re1-remote-peer rpd[32423]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event RecvNotify) (instance master)

Jul 15 06:50:34.569 PTX-2-re0-remote-peer rpd[1040]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event RecvNotify) (instance master)

Jul 15 06:50:34.603 PTX-2-re1-remote-peer rpd[32423]: bgp_standby_handle_pdu:4747: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 6 (Cease) subcode 9 (Hard Reset) [code 6 (Cease) subcode 3 (Peer Unconfigured)]

Jul 15 06:50:34.609 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 6 (Cease) subcode 9 (Hard Reset) [code 6 (Cease) subcode 3 (Peer Unconfigured)]

Jul 15 06:50:36.618 PTX-2-re0-remote-peer rpd[1040]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+53772 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:50:43.941 PTX-2-re0-remote-peer rpd[1040]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+53776 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:50:47.571 PTX-2-re0-remote-peer rpd[1040]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+47876 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:50:51.145 PTX-2-re0-remote-peer rpd[1040]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+47878 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:00.673 PTX-2-re1-remote-peer rpd[32423]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Idle to Established (event RecvEstab) (instance master)

Jul 15 06:51:00.996 PTX-2-re0-remote-peer rpd[1040]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from EstabSync to Established (event RsyncAck) (instance master)

Jul 15 06:51:00.996 PTX-2-re1-remote-peer rpd[32423]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event RecvNotify) (instance master)

Jul 15 06:51:01.046 PTX-2-re1-remote-peer rpd[32423]: bgp_standby_handle_pdu:4747: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:01.444 PTX-2-re0-remote-peer rpd[1040]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event RecvNotify) (instance master)

Jul 15 06:51:01.458 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:09.303 PTX-2-re0-remote-peer rpd[1040]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+42702 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:22.736 PTX-2-re1-remote-peer rpd[32423]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Idle to Established (event RecvEstab) (instance master)

Jul 15 06:51:23.088 PTX-2-re0-remote-peer rpd[1040]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from EstabSync to Established (event RsyncAck) (instance master)

Jul 15 06:51:23.942 PTX-2-re1-remote-peer rpd[32423]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event RecvNotify) (instance master)

Jul 15 06:51:24.146 PTX-2-re0-remote-peer rpd[1040]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event RecvNotify) (instance master)

Jul 15 06:51:23.980 PTX-2-re1-remote-peer rpd[32423]: bgp_standby_handle_pdu:4747: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:24.174 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:32.079 PTX-2-re0-remote-peer rpd[1040]: bgp_pp_recv:5857: NOTIFICATION sent to x.x.x.x+49748 (proto): code 2 (Open Message Error), Reason: pending close

Jul 15 06:51:35.561 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:41.867 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:43.971 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:45.762 PTX-2-re1-remote-peer rpd[32423]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Idle to Established (event RecvEstab) (instance master)

Jul 15 06:51:45.762 PTX-2-re0-remote-peer rpd[1040]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from EstabSync to Established (event RsyncAck) (instance master)

Jul 15 06:51:45.884 PTX-2-re1-remote-peer rpd[32423]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event RecvNotify) (instance master)

Jul 15 06:51:45.885 PTX-2-re0-remote-peer rpd[1040]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Established to Idle (event RecvNotify) (instance master)

Jul 15 06:51:45.923 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:45.914 PTX-2-re1-remote-peer rpd[32423]: bgp_standby_handle_pdu:4747: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:51.733 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:53.736 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:55.839 PTX-2-re0-remote-peer rpd[1040]: bgp_handle_notify:5295: NOTIFICATION received from x.x.x.x (Internal AS xxxxx): code 2 (Open Message Error)

Jul 15 06:51:58.158 PTX-2-re1-remote-peer rpd[32423]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from Idle to Established (event RecvEstab) (instance master)

Jul 15 06:51:58.159 PTX-2-re0-remote-peer rpd[1040]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer x.x.x.x (Internal AS xxxxx) changed state from EstabSync to Established (event RsyncAck) (instance master)

This issue is fixed in future target releases below mentioned:
evo:22.2R3-S7-EVO evo:22.3X80-D47-EVO evo:22.3X80-D49-EVO evo:23.2R2-S4-EVO evo:23.4R2-S4-EVO evo:24.2R2-EVO evo:24.4R1-EVO evo:25.1R1-EVO
junos:20.3X75-D441 junos:20.3X75-D52 junos:22.2R3-S7 junos:22.4R3-S7-J1 junos:23.2R2-S4 junos:23.4R2-S4 junos:24.2R2 junos:24.4R1 junos:25.1R1 

Based on the analysis conducted by JTAC and the engineering team, we were able to root-cause the BGP Session Flaps. On Junos platforms, when a catastrophic Border Gateway Protocol (BGP) configuration change occurs, creating a new peer structure, the BGP state transitions from open-sent to established multiple times until the local device finishes cleaning the old BGP session, due to which we see the BGP sessions flapping.  This is a transient issue and will eventually get cleared once the clean-up process has completed. This issue is considered a race condition in the code and may not be observed in every instance.