[MX/PTX] NSR is not supported with RE reboot CLI on NG-RE

Description

On MX/PTX platforms with Next-Generation Routing Engine (NG-RE) installed and graceful Routing Engine switchover (GRES)/nonstop active routing (NSR) enabled, NSR works with Routing Engine (RE) switchover, but is not supported in the case of RE reboot.

The Border Gateway Protocol (BGP) neighbor might flap during primary RE reboot, even though the Junos OS version has the fix of PR1417966, PR1415077, and PR1371045.

This is a product limitation discussed in PR1524791.

Symptoms

On MX/PTX with NG-RE, GRES/NSR is supported with RE switchover operation.

request chassis routing-engine master switch

However, it is not supported in the case of RE reboot operation.

```
request vmhost power-off 
```
```
request vmhost halt 
```
```
request vmhost reboot 
```
```
request system reboot 
```

The reboot operation is treated as connection close requested to TCP. TCP sends FIN to the peer, hence the TCP-based BGP session can flap.

This is a product limitation. In the reboot scenario, BGP flapping or not flapping depends on whether the FIN is sent out before the interface shuts down during reboot.

########## BGP is not affected by GRES/NSR RE switchover##########

 {MASTER}
User@mx2010-re0> show system information
Model: mx2010
Family: junos
Junos: 20.3R1.8  <---- This version has software fix of PR 1371045/1415077/1417966 
Hostname: mx2010-re0

{BACKUP.EN_US}
User@mx2010-re1> show system information
Model: mx2010
Family: junos
Junos: 20.3R1.8  <---- This version has software fix of PR 1371045/1415077/1417966 
Hostname: mx2010-re1

{MASTER}
User@mx2010-re0> show configuration chassis redundancy
graceful-switchover;

{MASTER}
User@mx2010-re0> show configuration routing-options
nonstop-routing;

{MASTER}
User@mx2010-re0> show task replication
        Stateful Replication: Enabled
        RE mode: Master 

    Protocol                Synchronization Status
    OSPF                    Complete
    OSPF3                   Complete
    BGP                     Complete 

{MASTER}
User@mx2010-re0> show database-replication summary

General:
    Graceful Restart           Enabled
    Mastership                 Master
    Connection                 Up
    Database                   Synchronized
    Message Queue              Ready 

{BACKUP.EN_US}
User@mx2010-re1> show system switchover
Graceful switchover: On
Configuration database: Ready
Kernel database: Ready
Switchover Status: Ready 

{MASTER}
User@mx2010-re0> request chassis routing-engine master switch check
Switchover Ready

{MASTER}
User@mx2010-re0> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
bgp.l3vpn.0
                       0          0          0          0          0          0
inet6.0
                       0          0          0          0          0          0
bgp.l3vpn-inet6.0
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.1.1.2              65001        139        138       0       0     1:39:37 Establ
  inet.0: 1/1/1/0
72.72.72.72           65000        141        139       0       0 4d 20:37:17 Establ
  inet.0: 1/1/1/0
  bgp.l3vpn.0: 0/0/0/0
72:72::72             65000        141        138       0       0 4d 20:35:29 Establ
  inet6.0: 0/0/0/0
  bgp.l3vpn-inet6.0: 0/0/0/0 

{MASTER}
User@mx2010-re0>  request chassis routing-engine master switch 
Toggle mastership between routing engines ? [yes,no] (no) yes

{MASTER}
User@mx2010-re1> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
bgp.l3vpn.0
                       0          0          0          0          0          0
inet6.0
                       0          0          0          0          0          0
bgp.l3vpn-inet6.0
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.1.1.2              65001        150          6       0       0       1:44:54   Establ 
  inet.0: 1/1/1/0
72.72.72.72           65000        151          7       0       0  4d 20:42:34   Establ 
  inet.0: 1/1/1/0
  bgp.l3vpn.0: 0/0/0/0
72:72::72             65000        151          6       0       0  4d 20:40:46 Establ 
  inet6.0: 0/0/0/0
  bgp.l3vpn-inet6.0: 0/0/0/0 


########## BGP gets flapped by RE reboot ##########

{MASTER}
User@mx2010-re0> show task replication
        Stateful Replication: Enabled
        RE mode: Master

    Protocol                Synchronization Status
    OSPF                    Complete
    OSPF3                   Complete
    BGP                     Complete 

{MASTER}
User@mx2010-re0> show database-replication summary

General:
    Graceful Restart           Enabled
    Mastership                 Master
    Connection                 Up
    Database                   Synchronized
    Message Queue              Ready 

{BACKUP.EN_US}
User@mx2010-re1> show system switchover
Graceful switchover: On
Configuration database: Ready
Kernel database: Ready
Switchover Status: Ready 

{MASTER}
User@mx2010-re0> request chassis routing-engine master switch check
Switchover Ready

{MASTER}
User@mx2010-re0> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
bgp.l3vpn.0
                       0          0          0          0          0          0
inet6.0
                       0          0          0          0          0          0
bgp.l3vpn-inet6.0
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.1.1.2              65001         52         27       0       0     2:11:39 Establ
  inet.0: 1/1/1/0
72.72.72.72           65000         53         27       0       0 4d 21:09:19 Establ
  inet.0: 1/1/1/0
  bgp.l3vpn.0: 0/0/0/0
72:72::72             65000         53         27       0       0 4d 21:07:31 Establ
  inet6.0: 0/0/0/0
  bgp.l3vpn-inet6.0: 0/0/0/0 

{MASTER}
User@mx2010-re0>  request vmhost halt 
Halt the vmhost ? [yes,no] (no) yes

Initiating vmhost halt...
warning: Initiating Junos shutdown...
shutdown: [pid 35688]

*** FINAL System shutdown message from User@mx2010-re0 ***

System going down IMMEDIATELY

Shutdown NOW!

{MASTER}
labroot@mx2010-re1> show bgp summary 
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          2          0          0          0          0
bgp.l3vpn.0
                       0          0          0          0          0          0
inet6.0
                       0          0          0          0          0          0
bgp.l3vpn-inet6.0
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.1.1.2              65001          5          4       0       1           42  Establ <---- flap
  inet.0: 1/1/1/0
72.72.72.72           65000          6          4       0       1           42  Establ <---- flap
  inet.0: 1/1/1/0
  bgp.l3vpn.0: 0/0/0/0
72:72::72             65000          3          3       0       1           22  Establ <---- flap
  inet6.0: 0/0/0/0
  bgp.l3vpn-inet6.0: 0/0/0/0

Solution

On MX/PTX platforms with NG-RE installed and GRES/NSR enabled, NSR works with RE switchover, but is not supported in the case of RE reboot.

The BGP neighbor might flap during primary RE reboot, even though the Junos OS version has the fix of PR1417966, PR1415077, and PR1371045.

In the reboot case, BGP flapping or not flapping depends on whether the FIN is sent out before the interface shuts down during reboot. This is a product limitation discussed in PR1524791.

PR1524791 - BGP flap during Master RE power-off
PR1417966 - BGP peers flap/restart during the RE mastership switchover.
PR1415077 - RPD protocol down/up with vmhost RE switchover on NG-RE
PR1371045 - MX10008 sent TCP FIN packet to remote session(s) during RE switchover (NSR enabled)

Modification History

2023-03-24 : PTX is added as an affected platform