Points to remember for EX 8200 stand-alone switch upgrade to Resilient Dual Root Partition

Description

This article provides information about the procedure to upgrade Junos OS and the Loader software on stand-alone EX8200 switches to the resilient dual root file system. Starting with Junos 10.4R3 and 11.1R1, the EX switch file system will be formatted from a 3 slice partition to a 4 slice partitioning scheme.

The resilient dual-root partitioning provides additional resiliency to switches in the following ways:

It allows the switch to boot transparently from the second root partition if the system fails to boot from the primary root partition.
It provides separation of the root Junos OS file system from the /var file system. If corruption occurs in the /var file system (a higher probability than in the root file system, due to the greater frequency of reads and writes in /var), the root file system is insulated from the corruption.

Symptoms

The goal of this article is to discuss in detail about the upgrade requirements, points to remember before performing the upgrade, and the step by step upgrade procedure.

Solution

Number of Reboot required for upgrade :

To upgrade Junos OS, one reboot per routing-engine is required.
To upgrade the jloader software image, two reboots per routing-engine are required. The loader software for an EX8200 Routing Engine resides in two flash memory banks; so you must perform the jloader upgrade twice - one for each memory bank. Each upgrade requires a reboot of the Routing Engine.
A total of three reboots per routing-engine is required to ensure that both Junos software and jloader firmware is installed on both of the memory banks of a routing-engine.

Planning for EX 8200 Upgrade :

As an EX 8200 upgrade (Junos software and jloader image) requires multiple reboots per routing-engine, it is recommended to plan for a maintenance window, before performing the upgrade on EX 8200 devices. In a production environment, it is recommended to open a proactive case with JTAC, well in advance, to intimate JTAC about the maintenance window that is scheduled for the EX 8200 upgrade.

Also, when planning Junos software and jloader image upgrade, carefully select the window time to appropriately accommodate the required downtime during the multiple reboots that are required for each routing-engine.

Approximate time for the Junos OS software upgrade : The disk is automatically reformatted from three partitions to four partitions during the reboot of the switch that completes the Junos OS upgrade. The reformat increases the reboot time for EX8200 switches by 10 to 25 minutes per Routing Engine. The total time for the Junos software upgrade will be Reformat time + Software installation time .
Approximate Time for jloader Software Upgrade on a single routing engine : A reboot after the jloader upgrade package is installed will take only a couple minutes more than the normal reboot time. Usually, it will be complete within 5 minutes per reboot after jloader is installed.

Note :

On EX8200 switches only, you must upgrade Junos OS, before upgrading the loader software. The loader software for an EX8200 Routing Engine resides in two flash memory banks. At any time, one bank acts as the primary bank and the Routing Engine boots from it. The other bank is the backup bank; if the Routing Engine cannot boot from the primary bank, it boots from the backup bank.

When you upgrade the loader software, the upgraded software is installed in the backup bank, which then becomes the new primary bank. So, the primary and backup banks alternate each time, the loader software is upgraded, with the primary bank containing the most recently installed version of the software and the backup bank containing the previous version.
Jloader software can be installed on a Routing-engine, only when its state is the current Primary. On EX 8200, every time RE0 is rebooted (Default Primary routing engine), RE1 (Default Backup routing engine) will become the current Primary RE.

On every RE, you need to install jloader software on two memory banks; so, after the first RE reboots, the second RE will become the current primary and you have to manually perform the routing-engine switchover to make the first routing engine as the current primary to proceed with the installation of jloader software on the second memory bank.
The jloader software image needs to be upgraded only for the two routing-engines and not for the line-cards.
The Junos OS software can be installed on the backup routing-engine, but the jloader software can be installed on a Routing-engine only when it is the Primary. This is by design.
If you upgrade the Junos software to the latest resilient dual root file system and do not upgrade the jloader firmware image, the switch will come up and function normally. However, on a particular routing-engine, if the switch cannot boot from the primary root partition due to file corruption, then it will not be able to automatically detect the corruption in the primary partition. Subsequently, it will not be able to transparently boot from the backup root partition. So, jloader upgrade is essential to serve the purpose of implementing the resilient dual root file system.

Tasks to perform before initiating the upgrade :

To proceed with the upgrade, the two files have to be on a formatted USB drive that has the FAT 32 file system.

For the Junos software code for EX 8200, refer to the following link: Software Downloads for EX8208

For the jloader image for EX 8200, refer to the following link: TSB15524 - Feature Release "Resilient Dual-Root Partitions" for EX Series Switches - (Junos OS Release 10.4R3 and later) [juniper.net]
The same files can be uploaded via FTP to the /var/tmp/ directory of the switch. However, the availability of a USB drive and console access during the time of upgrade is recommended to ensure that the fast recovery option is available, in case of any emergency.
It is recommended to have console access to both routing-engines; but if you want to remotely upgrade the device, then it is recommended to configure the management IP address on both of the routing-engines to ensure that when one of the REs is rebooted, you will have remote connectivity to the out-of-band management IP address of the other RE. For more information, refer to KB15687 - Configuring the EX 8200 Management Interfaces for Dual Routing Engines (RE's) with different IP addresses [juniper.net] .
The upgrade process to the dual root file system will ensure that the configuration file is not lost. However, it is strongly recommended to save configuration files on the EX device, before initiating the upgrade, as a power outage during upgrade may cause the loss of the configuration file. For more information, refer to KB20350 - [Upgrade EX Switch] Stage 2 - Backup configuration [juniper.net] .
Ensure that the console session is logged. This provides completely visibility to events that occur during the upgrade and helps to determine the root cause, in case of any issue during the upgrade.
The log files that are stored in the /var directory will not be saved and will be lost, after the upgrade. Take a backup of log files, if you want to preserve it for any future reference.

Phase 1 - Determining whether the loader software requires upgrading on EX8200 switches :

Before you begin the software upgrade on an EX8200 switch, check if the loader software needs to be upgraded. It is possible that a switch, which is running a Junos OS release that is earlier than Release 10.4R3, has a version of the loader software that supports resilient dual-root partitions.

For example, the switch might have been shipped from the factory with a Junos OS release that is earlier than Release 10.4R3; but with a version of the loader software that supports resilient dual-root partitions. Or the switch might have been downgraded from a Junos OS release that supports resilient dual-root partitions; but still retains a version of the loader software that supports resilient dual-root partitions.

user@switch> show chassis firmware Part Type Version FPC 6 U-Boot U-Boot 1.1.6 (Jan 13 2009 - 06:55:22) 2.3.0 loader FreeBSD/PowerPC U-Boot bootstrap loader 2.2 FPC 7 U-Boot U-Boot 1.1.6 (Jan 13 2009 - 06:55:22) 2.3.0 loader FreeBSD/PowerPC U-Boot bootstrap loader 2.2 Routing Engine 0 U-Boot U-Boot 1.1.6 (Mar 2 2011 - 04:29:01) 3.5.0 loader FreeBSD/PowerPC U-Boot bootstrap loader 2.3 Routing Engine 1 U-Boot U-Boot 1.1.6 (Mar 2 2011 - 04:29:01) 3.5.0 loader FreeBSD/PowerPC U-Boot bootstrap loader 2.3

The version of the loader software is displayed after the timestamp for U-Boot 1.1.6 . In the preceding example, the version is 3.5.0 (ignore the 1.1.6 version information in U-Boot 1.1.6 ; it does not indicate whether or not the version of the loader software supports resilient dual-root partitioning).

If the loader software version is 3.5.0 or later on EX8200 switches, the loader software does not have to be upgraded to support resilient dual-root partitioning. If the loader software version is earlier than 3.5.0, then the jloader software image has to be upgraded.

Note : The loader software version 3.5.0 is required only on the routing-engines. The loader software version on the linecards does not have to be upgraded.

Phase 2 - Enable the boot-sequencing function that is required by resilient dual-root partitions on the primary and backup routing engines :

To set the boot-sequencing environment variable, logon to the shell as the root user on the primary routing-engine:
```
% nvram setenv boot.btsq.disable 0
```
Perform the same task on the backup routing-engine.

Phase 3 - Install Junos OS 10.4R3 code on the EX 8200 primary and backup Routing-Engines with minimum down-time :

Logon to the primary Routing-Engine.
Deactivate the nonstop-routing routing option.
Deactivate the chassis redundancy graceful-switchover.
Commit synchronize.
user@switch> request routing-engine login other-routing-engine > logon to the backup Routing-Engine .
user@switch> request system software add /var/tmp/jinstall-ex-8200-10.4R3-domestic-signed.tgz
user@switch> request system reboot
user@switch> show version
Logon to the primary Routing Engine console port.
user@switch> request chassis routing-engine master switch > Transfer the console to the backup Routing Engine \

Verify if the default backup Routing Engine (shown as slot 1 in the command output) is now the primary Routing Engine:

user@switch> show chassis routing-engine


Routing Engine status:
Slot 0:
Current state Backup
Election priority Master (default)

Routing Engine status:
Slot 1:
Current state Master
Election priority Backup (default)

Install the new software on the current backup:

user@switch> request system software add /var/tmp/ jinstall-ex-8200-10.4R3-domestic-signed.tgz

Reboot the current backup Routing Engine:
```
user@switch> request system reboot
```

Note : Now both RE’s are upgraded with Junos 10.4R3/11.x. If you have determined that the jloader software image needs to be upgraded, as mentioned in Phase 1, continue with the procedure that is mentioned in Phase 4.If jloader does not have to be upgraded, then you can skip Phase 4 and proceed with the procedure that is mentioned in Phase 5.

Phase 4 - Upgrade the jloader software image :

The loader software for an EX8200 Routing Engine resides in two flash memory banks. At any time, one bank acts as the primary bank and the Routing Engine boots from it. The other bank is the backup bank; if the Routing Engine cannot boot from the primary bank, it boots from the backup bank.

When you upgrade the loader software, the upgraded software is installed in the backup bank, which then becomes the new primary bank. So, the primary and backup banks alternate each time the loader software is upgraded. The primary bank contains the most recently installed version of the software and the backup bank contains the previous version.

Jloader software can be installed on a Routing-Engine, only when it's state is the current Primary. On EX 8200, every time you reboot RE0 (Default Primary routing engine), RE1 (Default Backup routing engine) will become the current Primary RE. On every RE, you need to install the jloader software on two memory banks; so that after the first RE reboots, the second RE will become the current primary and you will have to manually perform the routing-engine switchover to make the first routing engine as the current primary to proceed with the installation of jloader software on the second memory bank.

Note :

: If you do not upgrade the loader software on both of the banks and the Routing Engine boots from the previous version of the loader software in the backup bank, the Routing Engine can no longer transparently boot from the alternate root partition, if it cannot boot from the primary root partition.
Jloader cannot be upgraded from 9.x code. It is essential to upgrade jloader from 10.x.
You can upgrade the loader software on a Routing Engine, only when it is the primary. Make sure that graceful Routing Engine switchover (GRES) and nonstop active routing (NSR) are disabled, before you begin the upgrade.
The jloader firmware image needs to be upgraded only for the two routing-engines and not for the line cards.

You will start at this state :

Slot 0: Current state Master Election priority Master (default) Routing Engine status: Slot 1: Current state Backup Election priority Backup (default)

Logon to the console of the current primary Routing Engine. Determine the primary bank and the version of the loader software on it:
```
% kenv | grep boot.primary.bank
boot.primary.bank="0"

% kenv | grep boot.ver
boot.ver="2.4.0"
```

Enter the CLI and install the loader package:

user@switch> request system software add /var/tmp/jloader-ex-8200-11.3I20110326_0802_hmerge-signed.tgz

user@switch> request system firmware upgrade scb

Firmware upgrade initiated....
Please wait for ~2mins for upgrade to complete....

After waiting for a couple of minutes, reboot the Routing Engine:

user@switch> request system reboot
Reboot the system ? [yes,no] (no) yes

When the reboot is complete, the chassis status will be:

Routing Engine status:
Slot 0:
Current state Backup
Election priority Master (default)

Routing Engine status:
Slot 1:
Current state Master
Election priority Backup (default)

You need to install jloader on the second memory bank of the RE in slot 0; so, you have to make it the primary by performing routing-engine switchover via the CLI and follow the same process again.
Enter the shell and verify if the previous backup bank is now the primary bank and that it contains the upgraded loader software:
```
% kenv | grep boot.primary.bank
boot.primary.bank="1"

% kenv | grep boot.ver
boot.ver="3.5.0"
```
To install the loader software in the current backup bank, repeat Steps 1-4. It will upgrade the loader software on Memory Bank 0 .

Verify if, after reboot, the jloader software is currently upgraded on Memory Bank 0 :

% kenv | grep boot.primary.bank
boot.primary.bank="0"

% kenv | grep boot.ver
boot.ver="3.5.0"

(Optional) The following message might be displayed, when a user logs on to the system:

 “ --- JUNOS 10.4R3.4 built 2011-03-19 22:06:32 UTC
At least one package installed on this device has limited support.
Run 'file show /etc/notices/unsupported.txt' for details….”

This message can be safely ignored. It is generated as a result of upgrading the loader software, after upgrading Junos OS. You can permanently remove this message by removing the loader software package and rebooting the routing-engine:

user@switch> request system software delete jloader-ex-8200

Unmounted /packages/mnt/jloader-ex-8200-11.3-20110310.0 ...

user@switch> request system reboot
Reboot the system ? [yes,no] (no) yes

To upgrade the backup Routing Engine on a switch with redundant Routing Engines, perform the following procedure:
1. Logon to the backup Routing Engine. It is recommended to use a console connection.
2. Perform the primary switchover, so that the backup Routing Engine becomes the primary:
```
user@switch> request chassis routing-engine master switch
```

Note : To upgrade RE1 (original backup RE), perform the same procedure to upgrade the loader software, which was used for RE0 (original primary Routing Engine; Steps 1 to 9). When you reboot the RE1, after upgrading the loader software in the first bank, the primary role returns to the RE0; which is the original primary Routing Engine. You will need to perform another primary switchover, before upgrading the loader software in the second bank of RE1.

Phase 5 - Enable GRES and NSR on the primary Routing-Engine and commit synchronize :

Logon to the current primary device.
user@switch# activate chassis redundancy graceful-switchover
user@switch# set routing-options nonstop-routing
user@switch# commit synchronize
Transfer the routing control back to the default primary routing engine:
```
user@switch> request chassis routing-engine master switch
```

Verify if the default primary Routing Engine (slot 0) is actually the primary Routing Engine:

user@switch> show chassis routing-engine

Routing Engine status:
Slot 0:
Current state Master
Election priority Master (default)

Routing Engine status:
Slot 1:
Current state Backup
Election priority Backup (default)

The upgrade process is complete now. You can review the Chassis status with the following commands:

user@switch >Show version
user@switch Show chassis firmware
user@switch Show chassis routing-engine
user@switch Show configuration chassis redundancy

Note :

If GRES is enabled and you try to add the jloader software, then it will not be able to extract the jloader software packet and the following error message will be generated:

warning: Disable GRES before software upgrade - state: 0

To fix this error message, you have to disable GRES and add the jloader software again.

Modification History

2021-03-19: Updated the article terminology to align with Juniper's Inclusion & Diversity initiatives.