Description

During Junos software upgrade of an SRX Multi-Node High-Availability (MNHA) cluster, one node at a time, there is normally no impact to the traffic flow. One reason for the MNHA upgrade being practically hitless is that the flow sessions can be synchronized between the nodes during the upgrade, even if they are temporarily running different Junos releases.

However in some rare cases there may be an incompatibility between the two Junos releases, due to a known software issue or due to a new feature introduced in the target release.

In the exceptional case where the two Junos releases are not compatible, the Junos upgrade needs to be performed in such a way that the nodes will not be in contact with eachother through the Inter Chassis Link (ICL) during the upgrade.

There are several options to achieve a Junos upgrade without the MNHA nodes getting into contact with each other.

1. Upgrade both nodes and reboot them simultaneously.

2. Upgrade and reboot one node at a time while isolating each node during the upgrade.

This article provides a procedure to perform the second option, which results in less downtime than the first option.


 

Solution


##### Preparing for the Upgrade #####

- Backup the configuration, certificates, license-keys and any other important files from both the nodes.
- Copy the new Junos OS image to both devices in the /var/tmp/ directory location.
- Ensure that you have a console connection to both nodes. In addition, ensure that the out-of-band management (fxp0) interface is reachable through SSH (recommended due to the faster speed than console).
- Deactivate SRG (Services Redundancy Group) preemption, if configured.
  
user@srx# delete chassis high-availability services-redundancy-group <SRG_number> preemption
user@srx# commit


- Disable SYN bit checking and sequence number checking, allowing for TCP traffic to rebuild sessions after failover to the upgraded device without requiring a 3-way TCP handshake. 
user@srx# set security flow tcp-session no-syn-check
user@srx# set security flow tcp-session no-sequence-check


##### Upgrade Procedure #####

In this example we consider a MNHA cluster with a node 1 (srx-01) and a node 2 (srx-02). In case of an Active/Backup cluster, the node 2 is the backup node.


1. Configure node 2 to be in software-upgrade mode and commit, to force all traffic to node 1 (optional if all traffic and all SRG are already active on node 1). 
user@srx-02# set chassis high-availability software-upgrade
user@srx-02# commit

Make sure that all traffic is passing as expected through node 1 now and no traffic is passing through node 2.


2. On node 2 (the node without traffic), disable all physical traffic interfaces, including the interface through which the ICL is connected. The fxp0 management interface can stay up. 
user@srx-02# set interfaces xx-x/x/x disable
user@srx-02# set interfaces xx-x/x/x disable
user@srx-02# set interfaces xx-x/x/x disable
...
user@srx-02# commit

This step fully isolates the node 2 from the network and from the MNHA cluster peer device.


3. delete the software-upgrade mode again on node 2 (if configured) 
user@srx-02# delete chassis high-availability software-upgrade
user@srx-02# commit


4. Upgrade node 2 and reboot it. 
user@srx-02> request system software add /var/tmp/<file_name.tgz> reboot


5. Wait until node 2 is fully booted up.
Check the status using "show chassis high-availability information"

Node Status will be "ONLINE"
SRG0 status will be "Current State: ISOLATED [ CS Pending post reboot ]"
SRG1+ status will be "HOLD"


6. If required, update the configuration to match new requirements in the installed Junos OS release. For example, if you have updated to the Junos OS Release 22.4 from Junos 22.3 or below, you must do additional configuration for the IPSec VPN feature. See IPsec VPN Support in Multinode High Availability.


7. On node 1, disable all traffic interfaces, including the interface through which the ICL is connected.
NOTE: Traffic interruption will start at this point. 
user@srx-01# set interfaces xx-x/x/x disable
user@srx-01# set interfaces xx-x/x/x disable
user@srx-01# set interfaces xx-x/x/x disable
...
user@srx-01# commit

Node 1 should now be fully isolated from the network and from the MNHA cluster peer device.


8. On node 2, enable all traffic interfaces again.
NOTE: Traffic interruption will end at this point. 
user@srx-02# delete interfaces xx-x/x/x disable
user@srx-02# delete interfaces xx-x/x/x disable
user@srx-02# delete interfaces xx-x/x/x disable
...
user@srx-02# commit


Node 2 should now move to Active state for SRG1+ (if SRG1+ are configured). SRG0 will stay in state "ISOLATED [ CS Pending post reboot ]", but that will not stop the traffic from flowing through the device.

It can take some time for dynamic routing protocols to establish and converge again before traffic starts flowing through node 2.


9. Confirm that the traffic flows as expected on node 2 now.

In case of any issues, you can move the traffic flow back to node 1 by reverting the previous two steps. If required you can use the "request system software rollback" command to rollback the software to the previous Junos release.


10. Configure node 1 to be in software-upgrade mode and commit. 
user@srx-01# set chassis high-availability software-upgrade
user@srx-01# commit


11. Upgrade node 1 and reboot it. 
user@srx-01> request system software add /var/tmp/<file_name.tgz> reboot


12. If required, update the configuration to match new requirements in the installed Junos OS release. For example, if you have updated to the Junos OS Release 22.4 from Junos 22.3 or below, you must do additional configuration for the IPSec VPN feature. See IPsec VPN Support in Multinode High Availability.


13. On node 1, enable all traffic interfaces, including the interface through which the ICL is connected. This will make the node 1 join the MNHA cluster, while it is still in software-upgrade mode (Node Status: OFFLINE [ SU ]). 
user@srx-01# delete interfaces xx-x/x/x disable
user@srx-01# delete interfaces xx-x/x/x disable
user@srx-01# delete interfaces xx-x/x/x disable
...
user@srx-01# commit


14. Confirm that the ICL is up and the MNHA cluster is getting joined as expected. 
user@srx> show chassis high-availability information
(check for "Conn State: UP" and wait for "Cold Sync Status: COMPLETE")


15. Take node 1 out of software-upgrade mode. 
user@srx-01# delete chassis high-availability software-upgrade
user@srx-01# commit


16. Confirm that the MNHA cluster is fully operational as expected.



##### Cleanup #####


- if applicable: activate SRG preemption again 
user@srx# set chassis high-availability services-redundancy-group <SRG_number> preemption
user@srx# commit


- if applicable: enable SYN bit checking and sequence number checking again. 
user@srx# delete security flow tcp-session no-syn-check
user@srx# delete security flow tcp-session no-sequence-check
user@srx# commit



 

Modification History

2023-05-26: First publication