This article describes the common guidelines to follow when taking MX Fabric Planes and MX SCB offline and then bringing them back online.
When troubleshooting Fabric Plane related issues, the offline and online of a Fabric Plane or SCB may be requested in the following articles:
KB23173 - Resolution Guide - MX - Troubleshoot Fabric Planes in Fault or Check State [juniper.net] KB23068 - MX SCB Troubleshooting Subroutine [juniper.net]
KB23173 - Resolution Guide - MX - Troubleshoot Fabric Planes in Fault or Check State [juniper.net]
KB23068 - MX SCB Troubleshooting Subroutine [juniper.net]
This article provides step-by-step instructions.
The following MX960 Switch Fabric diagram can be used for reference in this article. For more information about terminology and other platforms, refer to KB23065 - Understanding MX Fabric Planes [juniper.net] .
Note: It is strongly recommended that you perform the following procedures during a Maintenance Window .
Offlining a single plane should have no impact on services if redundancy is available. Redundancy can be verified with the command " show chassis fabric summary . " There should be at least one spare plane for redundancy.
show chassis fabric summary
Redundancy will not be available in MX routers that are using an SCB and at least one MPC line card.
Redundancy will not be available if a router is explicitly configured with " chassis fabric redundancy-mode increased-bandwidth ."
chassis fabric redundancy-mode increased-bandwidth
Also, these instructions assume that the MX router is configured with redundant Fabric capacity (MX960 3 SCBs and MX480/MX240 2 SCBs). If redundant Fabric Planes are not available, then the above guidelines are not recommended on production routers.
Click the following link to jump to the required instructions:
Offline/Online Fabric Plane
Offline/Online SCB (Switch Control Board)
The following steps provide an example for taking a plane in an MX960 router offline and then bringing it back online. The steps are also applicable to MX480 and MX240 routers.
Identify the status of each plane with the following command, and ensure that you have spare planes available for redundancy:
user@host> show chassis fabric summary For example: user@host> show chassis fabric summary Plane State Uptime 0 Online 3 hours, 7 minutes, 9 seconds 1 Check 3 hours, 7 minutes, 4 seconds 2 Fault 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Spare 3 hours, 7 minutes, 4 seconds 5 Spare 3 hours, 7 minutes, 4 seconds
user@host> show chassis fabric summary
For example:
user@host> show chassis fabric summary Plane State Uptime 0 Online 3 hours, 7 minutes, 9 seconds 1 Check 3 hours, 7 minutes, 4 seconds 2 Fault 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Spare 3 hours, 7 minutes, 4 seconds 5 Spare 3 hours, 7 minutes, 4 seconds
Offline the plane with the following command:
user@host> request chassis fabric plane offline <plane #> When the above command is executed, the corresponding plane will be taken offline and barred from switching data traffic. The next numerically lowest spare-plane is brought online and used for switching the packets. For example, in step 1, Plane 1 is in check state; it can be taken offline as follows: user@host> request chassis fabric plane offline 1 Offline initiated, use "show chassis fabric plane" to verify
user@host> request chassis fabric plane offline <plane #>
When the above command is executed, the corresponding plane will be taken offline and barred from switching data traffic. The next numerically lowest spare-plane is brought online and used for switching the packets.
For example, in step 1, Plane 1 is in check state; it can be taken offline as follows:
user@host> request chassis fabric plane offline 1 Offline initiated, use "show chassis fabric plane" to verify
Verify that the plane has been taken offline by running the show chassis fabric summary command again:
user@host> show chassis fabric summary Plane State Uptime 0 Online 3 hours, 7 minutes, 9 seconds 1 Offline 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Online 3 hours, 7 minutes, 4 seconds 5 Spare 3 hours, 7 minutes, 4 seconds Notice that Plane 1 is offline now and that Plane 4 has moved from Spare state to Online state. This Plane will now be used to forward traffic in place of Plane 1.
user@host> show chassis fabric summary Plane State Uptime 0 Online 3 hours, 7 minutes, 9 seconds 1 Offline 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Online 3 hours, 7 minutes, 4 seconds 5 Spare 3 hours, 7 minutes, 4 seconds
Notice that Plane 1 is offline now and that Plane 4 has moved from Spare state to Online state. This Plane will now be used to forward traffic in place of Plane 1.
Bring the plane online with the following command :
user@host> request chassis fabric plane online <plane #> When this command is executed, the corresponding plane will be brought online and placed in Spare state. This plane will be a spare until the next time the numerically higher planes encounter any issue or when they are manually taken offline. At that time, this plane will be brought online and will start switching traffic. For example, in Step 3, Plane 3 is in Offline state. It is brought online and moved to spare state as follows: user@host> request chassis fabric plane online 1 Online initiated, use "show chassis fabric plane" to verify
user@host> request chassis fabric plane online <plane #>
When this command is executed, the corresponding plane will be brought online and placed in Spare state. This plane will be a spare until the next time the numerically higher planes encounter any issue or when they are manually taken offline. At that time, this plane will be brought online and will start switching traffic.
For example, in Step 3, Plane 3 is in Offline state. It is brought online and moved to spare state as follows:
user@host> request chassis fabric plane online 1 Online initiated, use "show chassis fabric plane" to verify
Verify that the plane is back online by running the show chassis fabric summary command :
user@host> show chassis fabric summary Plane State Uptime 0 Online 3 hours, 7 minutes, 9 seconds 1 Spare 0 hours, 0 minutes, 4 seconds 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Online 3 hours, 7 minutes, 4 seconds 5 Spare 3 hours, 7 minutes, 4 seconds From the above output, clearing the Check state on Plane 1 was successful. Optionally, Plane 1 can be put into Online state again by offlining/onlining Plane 4.
user@host> show chassis fabric summary Plane State Uptime 0 Online 3 hours, 7 minutes, 9 seconds 1 Spare 0 hours, 0 minutes, 4 seconds 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Online 3 hours, 7 minutes, 4 seconds 5 Spare 3 hours, 7 minutes, 4 seconds
From the above output, clearing the Check state on Plane 1 was successful.
Optionally, Plane 1 can be put into Online state again by offlining/onlining Plane 4.
If offline/online of individual Fabric Planes does not resolve the issue, then the next step to offline/online the SCB is commonly performed.
The following steps provide an example for taking an SCB in an MX960 router offline and then bringing it back online. The steps are also applicable to MX480 and MX240 routers.
On MX960 routers, two SCBs provide full Fabric capacity for packet switching between the Packet Forwarding Engines (PFEs) (assuming only DPC cards) and the third SCB is a spare. On MX480 and MX240 routers, one SCB provides full Fabric capacity for packet switching and the second SCB is a spare.
Note : For MX960 routers, if the SCB in question is SCB2 (the spare SCB, which resides in slot8 and which does not have a Routing Engine), the SCB can be offlined/onlined directly without worrying about the RE's primary role.
Identify the status of each plane with the following command, and ensure that you have spare planes available for redundancy :
For example: > user@host> show chassis fabric summary Plane State Uptime 0 Online 3 hours, 7 minutes, 9 seconds 1 Check 3 hours, 7 minutes, 4 seconds 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Spare 3 hours, 7 minutes, 4 seconds 5 Spare 3 hours, 7 minutes, 4 seconds
> user@host> show chassis fabric summary Plane State Uptime 0 Online 3 hours, 7 minutes, 9 seconds 1 Check 3 hours, 7 minutes, 4 seconds 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Spare 3 hours, 7 minutes, 4 seconds 5 Spare 3 hours, 7 minutes, 4 seconds
Then identify the plane location of the Fabric Plane, which is in check or fault state by using the command: show chassis fabric plane-location . This command indicates which SCB must be taken offline.
show chassis fabric plane-location
For example: user@host> show chassis fabric plane-location ------------Fabric Plane Locations------------- Plane 0 Control Board 0 Plane 1 Control Board 0 Plane 2 Control Board 1 Plane 3 Control Board 1 Plane 4 Control Board 2 Plane 5 Control Board 2 In the above two example outputs, you can see that Plane1 is in check state and residing in Control Board 0.
user@host> show chassis fabric plane-location ------------Fabric Plane Locations------------- Plane 0 Control Board 0 Plane 1 Control Board 0 Plane 2 Control Board 1 Plane 3 Control Board 1 Plane 4 Control Board 2 Plane 5 Control Board 2
In the above two example outputs, you can see that Plane1 is in check state and residing in Control Board 0.
If the SCB to be taken offline is the one that holds the primary RE, then switch the RE primary role. In order to determine if the SCB holds the primary RE, first identify the RE with the primary role on the router by using the command: show chassis routing-engine.
show chassis routing-engine.
user@host> show chassis routing-engine Routing Engine status: Slot 0: Current state Master Election priority Master (default) Temperature 40 degrees C / 104 degrees F CPU temperature 44 degrees C / 111 degrees F DRAM 3584 MB Memory utilization 10 percent CPU utilization: User 0 percent Background 0 percent Kernel 3 percent Interrupt 1 percent Idle 96 percent Model RE-S-2000 Serial ID 9012001461 Start time 2021-03-31 18:47:07 PST Uptime 46 minutes, 37 seconds Last reboot reason could not be determined Load averages: 1 minute 5 minute 15 minute 0.03 0.02 0.00 Routing Engine status: Slot 1: Current state Backup Election priority Backup (default) Temperature 37 degrees C / 98 degrees F CPU temperature 42 degrees C / 107 degrees F DRAM 3584 MB Memory utilization 13 percent CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 99 percent Model RE-S-2000 Serial ID 9009037021 Start time 2021-03-31 19:17:16 PST Uptime 16 minutes, 30 seconds Last reboot reason 0x11:power cycle/failure misc hardware reason
In the above output, RE0 is primary and RE1 is backup. Control Board 0, the location for the SCB that is to be offlined, corresponds to Slot 0.
In the above output, RE0 is primary and RE1 is backup.
Control Board 0, the location for the SCB that is to be offlined, corresponds to Slot 0.
If the SCB to be taken offline is holding the primary RE, then perform a primary role switch with the command: request chassis routing-engine master switch . In the above output example, SCB0 is to be taken offline and is holding RE0, which is the primary. So you can perform an RE primary role switch: user@host> request chassis routing-engine master switch warning: Traffic will be interrupted while the PFE is re-initialized Toggle mastership between routing engines ? [yes,no] (no) yes Confirm that the primary role has been switched by using the show chassis routing-engine command.
If the SCB to be taken offline is holding the primary RE, then perform a primary role switch with the command: request chassis routing-engine master switch . In the above output example, SCB0 is to be taken offline and is holding RE0, which is the primary. So you can perform an RE primary role switch:
request chassis routing-engine master switch
user@host> request chassis routing-engine master switch warning: Traffic will be interrupted while the PFE is re-initialized Toggle mastership between routing engines ? [yes,no] (no) yes
Confirm that the primary role has been switched by using the show chassis routing-engine command.
show chassis routing-engine
Power-off the backup RE with the command: request system power-off other-routing-engine .
request system power-off other-routing-engine
For example, while you are logged in to RE1, which is the primary, the backup RE0 is powered off: user@host> request system power-off other-routing-engine
For example, while you are logged in to RE1, which is the primary, the backup RE0 is powered off:
user@host> request system power-off other-routing-engine
Verify that the RE is powered-off with the command: show chassis routing-engine . user@host> show chassis routing-engine Routing Engine status: Slot 0: Current state Present Slot 1: Current state Master Election priority Backup (default) Temperature 37 degrees C / 98 degrees F CPU temperature 42 degrees C / 107 degrees F DRAM 3584 MB Memory utilization 13 percent CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 99 percent Model RE-S-2000 Serial ID 9009037021 Start time 2021-03-31 19:17:16 PST Uptime 16 minutes, 30 seconds Last reboot reason 0x11:power cycle/failure misc hardware reason
Verify that the RE is powered-off with the command: show chassis routing-engine .
user@host> show chassis routing-engine Routing Engine status: Slot 0: Current state Present Slot 1: Current state Master Election priority Backup (default) Temperature 37 degrees C / 98 degrees F CPU temperature 42 degrees C / 107 degrees F DRAM 3584 MB Memory utilization 13 percent CPU utilization: User 0 percent Background 0 percent Kernel 0 percent Interrupt 0 percent Idle 99 percent Model RE-S-2000 Serial ID 9009037021 Start time 2021-03-31 19:17:16 PST Uptime 16 minutes, 30 seconds Last reboot reason 0x11:power cycle/failure misc hardware reason
Take the SCB offline with the command: request chassis cb slot offline <#>.
request chassis cb slot offline <#>.
In this example, CB is taken offline: user@host> request chassis cb offline slot 0 Offline initiated, use "show chassis environment cb" to verify Verify that CB has been taken offline: user@host> show chassis environment cb CB 0 status: State Offline Power 1 Disabled Power 1 Disabled CB 1 status: State Online Master Temperature 31 degrees C / 87 degrees F Power 1 1.2 V 1205 mV 1.25 V 1256 mV 1.5 V 1511 mV 1.8 V 1798 mV 2.5 V 2507 mV 3.3 V 3312 mV 3.3 V SM3 3293 mV 5.0 V 5033 mV 5.0 V RE 5085 mV 12.0 V 12084 mV 12.0 V RE 11930 mV Power 2 4.6 V bias MidPlane 4821 mV 11.3 V bias PEM 11330 mV 11.3 V bias FPD 11330 mV 11.3 V bias POE 0 11292 mV 11.3 V bias POE 1 11330 mV Bus Revision 42 FPGA Revision 1 The planes on the SCB have been taken offline and the planes in spare are switched to online state. Verify the same with the command: show chassis fabric summary . user@host> show chassis fabric summary Plane State Uptime 0 Offline 1 Offline 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Online 3 hours, 7 minutes, 4 seconds 5 Online 3 hours, 7 minutes, 4 seconds
In this example, CB is taken offline:
user@host> request chassis cb offline slot 0 Offline initiated, use "show chassis environment cb" to verify
Verify that CB has been taken offline:
user@host> show chassis environment cb CB 0 status: State Offline Power 1 Disabled Power 1 Disabled CB 1 status: State Online Master Temperature 31 degrees C / 87 degrees F Power 1 1.2 V 1205 mV 1.25 V 1256 mV 1.5 V 1511 mV 1.8 V 1798 mV 2.5 V 2507 mV 3.3 V 3312 mV 3.3 V SM3 3293 mV 5.0 V 5033 mV 5.0 V RE 5085 mV 12.0 V 12084 mV 12.0 V RE 11930 mV Power 2 4.6 V bias MidPlane 4821 mV 11.3 V bias PEM 11330 mV 11.3 V bias FPD 11330 mV 11.3 V bias POE 0 11292 mV 11.3 V bias POE 1 11330 mV Bus Revision 42 FPGA Revision 1
The planes on the SCB have been taken offline and the planes in spare are switched to online state. Verify the same with the command: show chassis fabric summary .
user@host> show chassis fabric summary Plane State Uptime 0 Offline 1 Offline 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Online 3 hours, 7 minutes, 4 seconds 5 Online 3 hours, 7 minutes, 4 seconds
Wait for a few seconds, and then bring the SCB online by using the command: request chassis online slot <#> .
request chassis online slot <#>
user@host> request chassis cb online slot 0 Online initiated, use "show chassis environment cb" to verify The planes on this SCB are brought online and placed in Spare state. Verify the same by using the show chassis fabric summary command: user@host> show chassis fabric summary Plane State Uptime 0 Spare 3 hours, 7 minutes, 9 seconds 1 Spare 3 hours, 7 minutes, 4 seconds 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Online 3 hours, 7 minutes, 4 seconds 5 Online 3 hours, 7 minutes, 4 seconds
user@host> request chassis cb online slot 0 Online initiated, use "show chassis environment cb" to verify
The planes on this SCB are brought online and placed in Spare state. Verify the same by using the show chassis fabric summary command:
user@host> show chassis fabric summary Plane State Uptime 0 Spare 3 hours, 7 minutes, 9 seconds 1 Spare 3 hours, 7 minutes, 4 seconds 2 Online 3 hours, 6 minutes, 59 seconds 3 Online 3 hours, 6 minutes, 54 seconds 4 Online 3 hours, 7 minutes, 4 seconds 5 Online 3 hours, 7 minutes, 4 seconds
Power on the RE, which was powered-off in Step 4 with the command: request system power-on other-routing-engine .
request system power-on other-routing-engine
While you are logged in to RE1, which is the primary, the other RE is powered on: user@host> request system power-on other-routing-engine Verify that the backup RE is online by using the command: show chassis routing-engine . If you want to make the current backup RE (in this example RE0) as the primary again, use the command request chassis routing-engine master acquire on the backup RE.
While you are logged in to RE1, which is the primary, the other RE is powered on:
user@host> request system power-on other-routing-engine
Verify that the backup RE is online by using the command: show chassis routing-engine .
If you want to make the current backup RE (in this example RE0) as the primary again, use the command request chassis routing-engine master acquire on the backup RE.
request chassis routing-engine master acquire
2022-03-09: Updated article to include verification of redundancy for offlining one plane
2021-03-19: Updated the article terminology to align with Juniper's Inclusion & Diversity initiatives