Alert Type

PSN - Product Support Notification
Low/Notificationnotification
Low/Notificationnotification

Product Affected

PTX5K/PTX3K/MX240/MX480/MX960/MX2010/MX2020/MX2008/MX10016/MX10008/PTX10008/PTX10016 running with the following Routing Engine: RE-S-X6-64G RE-S-X6-64G-LT RE-PTX-X8-64G REMX2008-X8-64G REMX2K-X8-64G REMX2K-X8-64G-LT

Alert Description

The Linux kernel version WRL9 will no longer support the Intel i40e NVM firmware version prior to 6.01. In order to install WRL9 JUNOS vmhost image, an upgrade of i40e NVM firmware to version 6.01 is required.

Solution

In order to install JUNOS vmhost image based on Linux WRL9 on all VMHOST platforms, a firmware upgrade of Intel i40e NVM firmware is required on the Routing Engine if it is not running with an i40e NVM firmware version 6.01. For platforms PTX5K/PTX3K/MX240/MX480/MX960/MX2010/MX2020/MX2008/MX10016/MX10008/PTX10008/PTX10016, the JUNOS versions support i40e NVM firmware upgrade are listed in the following table. For the other platforms (PTX/QFX/EX/SRX), the JUNOS version which supports the i40e NVM firmware upgrade is 19.3R1 or later.

Note: i40e-NVM upgrade is NOT mandatory for the following platforms :

  • MX10003
  • MX10016/10008 with JNP10K-RE0 - shown in the cli output as "RE-MX-2X00x4"
  • PTX10016/10008 with JNP10K-RE0 - shown in the cli output as "RE-PTX-2X00x4"
  • PTX10002-XX
  • QFX1000, QFX10002
  • QFX5000
  • EX platforms

After the i40e NVM firmware is upgraded using the CLI command "request system firmware upgrade re i40nvm", power recycling of the Routing Engine (multiple times) is required and the upgrade status will be presented only on the console port. Hence, a console access is required to perform this upgrade procedure.

Once the i40e NVM firmware is upgraded, a downgrade action is NOT supported. If a user installs a JUNOS vmhost image lower than the one that supports the i40e NVM firmware version 6.01, the downgrade of the JUNOS vmhost image via JUNOS CLI will leave the system in an undetermined state and this is not recommended. If a user installs the old JUNOS vmhost image via USB, the Routing Engine will not be able to boot up properly. For recovery, use a USB to re-install the JUNOS vmhost image with a version that supports the new i40e NVM firmware.

Table 1. JUNOS versions that support i40e NVM firmware upgrade

Platform 15.x 16.x 17.x 18.x 19.x
PTX5000 15.1F6-S11 16.1R7 17.1R3 / 17.2R3 / 17.3R3 / 17.4R2 18.1R1 / 18.2R3 / 18.3R1 / 18.4R1 19.1R1 or later
PTX3000 N/A N/A N/A 18.2R3 / 18.3R3 / 18.4R2 19.1R2 / 19.2R1
MX240/480/960 15.1F6-S11 16.1R7 17.1R3 / 17.2R3 / 17.3R3 / 17.4R2 18.1R1 / 18.2R1 / 18.3R1 / 18.4R1 19.1R1 or later
MX2010/2020 15.1F6-S11 16.1R7 17.1R3 / 17.2R3 / 17.3R3 / 17.4R2 18.1R1 / 18.2R1 / 18.3R1 / 18.4R1 19.1R1 or later
MX2008 N/A N/A N/A 18.2R3 / 18.3R3 / 18.4R2 19.1R2 / 19.2R1
MX10016/10008
PTX10016/10008 		N/A N/A N/A 18.2R3 / 18.3R3 / 18.4R2 19.1R2 / 19.2R1

The jfirmware-vmhost package contains the i40e NVM firmware upgrade will be available under the JUNOS releases as listed from above table for various platforms, please select the jfirmware-vmhost package according to the running version of the system. Removing power / loss of power during upgrade will corrupt NVM and RMA might be required.

Note: Due to PR1446205, customer might encounter error when installing jfirmware-vmhost package on a MX system with Enhanced MX SCB 3 installed. To resolve the issue, please upgrade the JUNOS with a version supporting i40e NVM upgrade per the above table and containing the fix of PR1446205 before installing the jfirmware-vmhost package to upgrade the i40e NVM. To obtain the jifrmware-vmhost package for the targeted version, please contact JTAC.

The following upgrade procedures have to be performed from router console access and console access is required for all Routing Engines. Also, power recycling of the Routing Engine is required during the firmware upgrade process which implies that on-site manual operation will be required. However, if the platform has dual Routing Engines installed, Routing Engine power recycling could be done by power recycling the CB module from the primary Routing Engine.

1. Upgrade the system with the JUNOS image version, which supports i40e NVM firmware upgrade (listed above) and reboot the system to load the newly installed JUNOS image.

2. Upload the jfirmware-vmhost package from the corresponding JUNOS release to the /var/tmp/ directory of the system and install that to the primary Routing Engine of the system. 
user@mx> request vmhost software add /var/tmp/jfirmware-vmhost-x86-64-19.2R1.tgz

3. After the firmware installation, verify the new firmware is available under the system (available version 6.01) 
user@mx> show system firmware

Part             Type             Tag          Current     Available         Status
                                               version     version
Routing Engine 0  RE BIOS       0                0.53.1                           OK
Routing Engine 1 RE BIOS        0                0.43                             OK
Routing Engine 1 RE FPGA        1                28.0.0                           OK
Routing Engine 1 RE SSD1        3                0.0.0                            OK
Routing Engine 1 RE SSD2        3                0.0.0                            OK
Routing Engine 1 RE i40e-NVM    7                4.26         6.01                OK

4. Perform i40e NVM upgrade. Upgrade status could be seen with the "show system firmware" hidden command output. 
user@mx> request system firmware upgrade re i40nvm
Part               Type            Tag   Current   Available Status
                                         version   version
Routing Engine 1 RE i40e-NVM       7     4.26      6.01           OK

Perform indicated firmware upgrade ? [yes,no] (no) yes
Firmware upgrade initiated, use "show system firmware" after reboot to verify the firmware version
user@mx>

If upgrade initiated "show system firmware" command will display "PROGRAMMING   (0%)" as status. Actual firmware upgrade will happen only if user triggers a reboot using "request vmhost reboot". 
user@mx> show system firmware
Part             Type                            Tag          Current   Available Status
                                                          version   version
Routing Engine 0 RE BIOS           0             0.43                           OK
Routing Engine 1 RE FPGA          1             28.0.0                        OK
Routing Engine 1 RE SSD1          3              0.0.0                          OK
Routing Engine 1 RE SSD2          3              0.0.0                          OK
Routing Engine 1 RE i40e-NVM    7          4.26      6.01            PROGRAMMING   (0%)
user@mx>

5a. On a SINGLE Routing Engine system, after the above upgrade, system needs to be restarted with the command "request vmhost reboot". Watch the console output and follow the instructions in order to complete the upgrade. Power recycle of the Routing Engine will be required. (After reboot firmware upgrade will happen at bootup stage and customer will be requested to power cycle router in between multiple times , it’s a multistage upgrade). When requested to power cycle on console, use external power cycle for power cycling the RE. Followings are the sample messages on console which request customer to do power cycling.
"Reboot is required to complete the update process.
"Upgrade complete please power reboot"
"You may notify to  power reboot again after reboot if required”

5b. On a DUAL Routing Engines system, at this point, we need to switch Routing Engine mastership to the backup Routing Engine (eg. from RE1 to RE0). Once it's done, the old primary Routing Engine (eg. RE1) will need to be restarted using the CLI command "request vmhost reboot". When power recycling is required for the Routing Engine during the reboot process as we could see from the console output (RE1), from the new primary Routing Engine (ie. RE0), the CLI command "request chassis cb (offline | online) slot <slot> " (eg. slot-number = 1 for RE1) could initiate the power recycling (offline action followed by an online action) of the Routing Engine which is undergoing the upgrade process. Following is the sample message on console which request customer to do power cycling.
“Reboot is required to complete the update process.
Upgrade complete please power reboot
You may notify to  power reboot again after reboot if required”

6. After completing upgrade, router will boot up as usual,  use "show system firmware"  CLI to verify firmware version after login to RE1. When it's completed, the same upgrade process could be done on the other Routing Engine (ie. RE0). To power recycle the Routing Engine, the CLI command "request chassis cb (offline | online) slot <slot> " (eg. slot-number = 0 for RE0) could be issued from the primary Routing Engine (eg. RE1). Power recycle must be performed for each Routing Engine per upgrade instruction in order to complete the upgrade process. 
user@mx-re0> show system firmware
Part             Type              Tag Current   Available Status
                                       version   version
Routing Engine 0 RE BIOS           0   0.53.1                OK
Routing Engine 1 RE BIOS           0   0.43                   OK
Routing Engine 1 RE FPGA           1   28.0.0              OK
Routing Engine 1 RE SSD1           3   0.0.0                 OK
Routing Engine 1 RE SSD2           3   0.0.0                 OK
Routing Engine 1 RE i40e-NVM  7   6.1      6.01      OK   
user@mx-re0>
* Current version will display as ‘6.1’ for NVM version ‘6.01’ , which is known limitation.

7. This step is required only if the selected JUNOS version for upgrade does not contain the fix of PR 1529710. For each Routing Engine, download the package lldp-patch-for-i40e-upgrade.tgz attached in this PSN, upload it to the /var/tmp/ directory of the Routing Engine and install it using the "request system software add" command. The same package is used for both MX and PTX platforms. MX Platforms are not negative exposed to the impact of i40e-NVM driver sending periodic LLDP packets . This exposure is specific to PTX platform *only* but applying the LLDP patch does no harm. 
user@mx-re0> request system software add /var/tmp/lldp-patch-for-i40e-upgrade.tgz
Verified lldp-patch-for-i40e-upgrade signed by PackageDevelopmentEc_2018 method ECDSA256+SHA256
[ re_name = RE-PTX-2X00x8 ]
Pushing script(s) to host ...
Install the script(s) under host-os....
Script(s) copy done.

user@mx-re0> show version | match lldp
lldp-patch-for-i40e-upgrade

user@mx-re0>
 
Perform "request vmhost reboot" on the CLI to restart the Routing Engine after the above package is added successfully. When the Routing Engine comes back online, the upgrade process is completed. With this package installed, the following message will be shown when user logs in the system. This warning message can be ignored.

At least one package installed on this device has limited support.
Run 'file show /etc/notices/unsupported.txt' for details.

After an upgrade is done, user could install WRL9 JUNOS vmhost image into the Routing Engine. User could install older JUNOS image into the Routing Engine with an upgraded i40e NVM firmware as it supports i40e 1.1.23 and 2.4.3 driver versions in the older JUNOS versions as listed below:
 
Platform i40e-1.1.23/
NVM-4.26 Support 		i40e-2.4.3/
NVM-6.01 Support
PTX5000 15.1F3 15.1F6-S11 16.1R7 17.1R3 / 17.2R3 / 17.3R3 / 17.4R2 18.1R1 / 18.2R3 / 18.3R1 / 18.4R1 19.1R1 or later
PTX3000 16.1R4 N/A 16.1R7 17.3R3-S6 / 17.4R2-S7 18.2R3 / 18.2R3-S8 / 18.3R3 / 18.4R2 19.1R2/ 19.2R1
MX240/480/960 15.1F3 15.1F6-S11 16.1R7 17.1R3 / 17.2R3 / 17.3R3 / 17.4R2 18.1R1 / 18.2R1 / 18.3R1 / 18.4R1 19.1R1 or later
MX2010/2020 15.1F5-S1 15.1F6-S11 16.1R7 17.1R3 / 17.2R3 / 17.3R3 / 17.4R2 18.1R1 / 18.2R1 / 18.3R1 / 18.4R1 19.1R1 or later
MX2008 15.1F7 N/A N/A 17.2R3 / 17.3R3 / 17.4R2 18.1R1 / 18.2R1 / 18.3R1 / 18.4R1 19.1R1 or later
MX10016/10008
PTX10016/10008 		18.2R1 N/A N/A N/A 18.2R1 / 18.3R1 / 18.4R1 19.1R1 or later

Modification History

2021-12-03 - Update format for readability - no change to the content
2021-10-15 - Update format for readability - no change in the content
2020-12-10 - Update the content to include PR1446205
2020-06-16 - Remove PTX10008/16 and MX10016/100008 running JNP10K-RE0 as they don't need to upgrade.