After rebooting a QFX5000 device, it is unable to send BGP notification messages to the peer device.
Topology:
QFX5100-r019 (xe-0/1/2, 192.168.20.1/30) --- (xe-0/0/0:0, 192.168.20.2/30 ) QFX5100-r030
Setting eBGP and iBGP between two QFX5100s, eBGP uses physical interfaces and iBGP uses logical interfaces (Loopback interface). Both lo0 routes will be exported to each other via eBGP.
root@QFX5100-r019# show protocols bgp | display set set protocols bgp traceoptions file bgp.txt set protocols bgp traceoptions flag all set protocols bgp group ibgp type internal set protocols bgp group ibgp local-address 192.168.10.11 set protocols bgp group ibgp family evpn signaling set protocols bgp group ibgp multipath set protocols bgp group ibgp neighbor 192.168.10.1 peer-as 64512 set protocols bgp group ebgp type external set protocols bgp group ebgp family inet unicast set protocols bgp group ebgp export export-bgp set protocols bgp group ebgp local-as 65101 set protocols bgp group ebgp multipath multiple-as set protocols bgp group ebgp neighbor 192.168.20.2 peer-as 65201 root@QFX5100-r019# show interfaces lo0 | display set set interfaces lo0 unit 0 family inet address 192.168.10.11/32 [master:0][edit] root@QFX5100-r019# show policy-options | display set set policy-options policy-statement export-bgp term 1 from protocol direct set policy-options policy-statement export-bgp term 1 from protocol static set policy-options policy-statement export-bgp term 1 from protocol local set policy-options policy-statement export-bgp term 1 from interface lo0.0 set policy-options policy-statement export-bgp term 1 then accept ************** root@QFX5100-r030# show protocols bgp | display set set protocols bgp traceoptions file bgp.txt set protocols bgp traceoptions flag all set protocols bgp group ibgp type internal set protocols bgp group ibgp local-address 192.168.10.1 set protocols bgp group ibgp family evpn signaling set protocols bgp group ibgp multipath set protocols bgp group ibgp neighbor 192.168.10.11 peer-as 64512 set protocols bgp group ebgp type external set protocols bgp group ebgp family inet unicast set protocols bgp group ebgp export export-bgp set protocols bgp group ebgp local-as 65201 set protocols bgp group ebgp multipath multiple-as set protocols bgp group ebgp neighbor 192.168.20.1 peer-as 65101 root@QFX5100-r030# show interfaces lo0 | display set set interfaces lo0 unit 0 family inet address 192.168.10.1/32 [master:0][edit] root@QFX5100-r030# show policy-options | display set set policy-options policy-statement export-bgp term 1 from protocol direct set policy-options policy-statement export-bgp term 1 from protocol local set policy-options policy-statement export-bgp term 1 from interface lo0.0 set policy-options policy-statement export-bgp term 1 then accept
When rebooting r019 via the command 'request system reboot' , then checking the BGP status on r030, IBGP (192.168.10.11) will not change to down immediately.
'request system reboot'
root@QFX5100-r030# run show bgp summary Groups: 2 Peers: 2 Down peers: 1 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 0 0 0 0 0 0 bgp.evpn.0 0 0 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 192.168.10.11 64512 6 6 0 0 2:57 Establ bgp.evpn.0: 0/0/0/0 192.168.20.1 65101 18 17 0 2 1:07 Idle [master:0][edit] root@QFX5100-r030# run show bgp summary Groups: 2 Peers: 2 Down peers: 2 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 0 0 0 0 0 0 bgp.evpn.0 0 0 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 192.168.10.11 64512 0 0 0 1 49 Connect 192.168.20.1 65101 18 17 0 2 2:14 Idle [master:0][edit] root@QFX5100-r030# run show bgp neighbor Peer: 192.168.10.11+179 AS 64512 Local: 192.168.10.1 AS 64512 Group: ibgp Routing-Instance: master Forwarding routing-instance: master Type: Internal State: Connect Flags: <> Last State: Active Last Event: ConnectRetry Last Error: Hold Timer Expired Error Options: <Preference LocalAddress AddressFamily PeerAS Multipath Rib-group Refresh> Address families configured: evpn Local Address: 192.168.10.1 Holdtime: 90 Preference: 170 Number of flaps: 1 Last flap event: HoldTime <-- it is not RecvNotify Error: 'Hold Timer Expired Error' Sent: 1 Recv: 0 Trace options: all Trace file: /var/log/bgp.txt size 131072 files 10 Peer: 192.168.20.1 AS 65101 Local: 192.168.20.2 AS 65201 Group: ebgp Routing-Instance: master Forwarding routing-instance: master Type: External State: Idle Flags: <PeerInterfaceError> Last State: Established Last Event: Stop Last Error: Cease Export: [ export-bgp ] Options: <Preference AddressFamily PeerAS Multipath LocalAS Refresh> Options: <MultipathAs> Address families configured: inet-unicast Holdtime: 90 Preference: 170 Local AS: 65201 Local System AS: 64512 Number of flaps: 2 Last flap event: Stop <-- it is not RecvNotify Error: 'Cease' Sent: 2 Recv: 0 Trace options: all Trace file: /var/log/bgp.txt size 131072 files 10
This is a limitation because there is no hypervisor on QFX5000 PFE, so PFE will go down immediately.
From the following messages, the physical interface is down before sending notifications. Thus, notification messages will not be sent to the peer.
Mar 1 06:23:16 QFX5100-r019 fpc0 fxpc_check_signal: Going to shutdown network ports Mar 1 06:23:16 QFX5100-r019 fpc0 ifp xe-0/1/2 ifd_mdown Mar 1 06:23:21 QFX5100-r019 rpd[2053]: bgp_peer_delete:9410: NOTIFICATION sent to 192.168.10.1 (Internal AS 64512): code 6 (Cease) subcode 3 (Peer Unconfigured), Reason: Peer Deletion Mar 1 06:23:21 QFX5100-r019 rpd[2053]: bgp_peer_delete:9410: NOTIFICATION sent to 192.168.20.2 (External AS 65201): code 6 (Cease) subcode 3 (Peer Unconfigured), Reason: Peer Deletion
The Bidirectional Forwarding Detection (BFD) protocol is a simple hello mechanism that detects failures in a network。
Please add BFD, which will let all BGP sessions go down immediately when the peer device reboots.