This article provides a procedure to follow when upgrading, using an In-Service Software Upgrade (ISSU) procedure, so as to avoid fabric mis-programming. It also provides steps to verify the fabric state.
After an ISSU upgrade, the fabric (plane) might get mis-programmed, thus preventing all the planes from being mapped to the Flexible PIC Concentrator (FPC), causing forwarding issues.
1. Impacted Software Releases 2. Summary Check List and Corrective Procedures 3. Clearing Selected Standard SCB Registers.
3.1 Identify Standard SCB present in the chassis 3.2 Clear fabric register values 3.2.1 Clear CB0's registers. Apply to MX960, MX480, and MX240 3.2.2 Clear CB1's registers. Apply to MX960, MX480, and MX240 3.2.3 Clear CB2's registers. Apply to MX960 only.
4. Identify components with fabric errors
4.1. FPC Offlined due to Unreachable Destination 4.2. Fabric Planes with Errors 4.2.1. Plane in check state 4.2.2. Link in error state 4.2.2.1. Destination errors 4.2.2.2. Link error 4.2.2.3. Unused links
check
error
5. Fabric Error Recovery Procedure
6. Sample Output for Working Fabric States
6.1 Output from show chassis fabric summary 6.2 Output from show chassis fabric fpc 6.2 Output from show chassis fabric plane
show chassis fabric summary
show chassis fabric fpc
show chassis fabric plane
1. Impacted Software Releases
FROM Releases: SDG 1.0 12.1 and earlier releases, includes any derived special and service releases. TO Releases: SDG 1.5 12.2 and later releases
2. Summary Check List and Corrective Procedures
A. Clear selected Standard SCB registers. This step applies to Standard SCB (MX SCB) only in ISSU upgrade for specified releases. Do NOT apply this step if ISSU is from 11.4R10/12.1R9/SDG1.5/12.2R1 or later releases and their derived service/special releases. ISSUs from all other releases, including derived special and service releases, must follow this procedure. B. Identify and correct all fabric errors before initiating ISSU. This step applies to all SCB types and is the best practice for ISSU.
Step B.1) FPC states Step B.2) Fabric plane states
3. Clearing Selected Standard SCB Registers. 3.1 Identify Standard SCB present in the chassis
root@testmx> show chassis hardware | match "MX SCB" CB 0 REV 03 710-021523 WY8117 MX SCB CB 1 REV 03 710-021523 XC9374 MX SCB
3.2 Clear fabric register values
For each of the MX SCBs identified above, perform the following CLI commands to clear the selected register values. Each CB has two fabric chips, so perform this on both fchip-id 0 and fchip-id 1.
3.2.1 Clear CB0's registers. Apply to MX960, MX480, and MX240.
test chassis fchip write-register address 0x10c data 0 cb-slot 0 fchip-id 0 test chassis fchip write-register address 0x110 data 0 cb-slot 0 fchip-id 1
3.2.2 Clear CB1's registers. Apply to MX960, MX480, and MX240.
test chassis fchip write-register address 0x10c data 0 cb-slot 1 fchip-id 0 test chassis fchip write-register address 0x110 data 0 cb-slot 1 fchip-id 1
3.2.3 Clear CB2's registers. Apply to MX960 only.
test chassis fchip write-register address 0x10c data 0 cb-slot 2 fchip-id 0 test chassis fchip write-register address 0x110 data 0 cb-slot 2 fchip-id 1
4. Identify components with fabric errors 4.1. FPC Offlined due to Unreachable Destination An FPC may report a destination error and be offlined by Fabric manager. Check if any FPC is offline due to unreachable destinations, using CLI command show chassis fpc . In the following output, FPC1 is offlined due to unreachable destinations. Please follow Section 5 Fabric Error Recovery Procedure to recover the failure. Repeat this for all FPCs that are offlined due to unreachable destinations.
show chassis fpc
root@testmx> show chassis fpc Temp CPU Utilization (%) Memory Utilization (%) Slot State (C) Total Interrupt DRAM (MB) Heap Buffer 0 Empty 1 Offline ---FPC offlined due to unreachable destinations--- 2 Online 40 5 0 2048 20 13 3 Empty 4 Empty 5 Online 38 4 0 2048 17 13 6 Empty 7 Empty 8 Empty 9 Online 31 4 0 1024 17 29 10 Empty 11 Empty
4.2. Fabric Planes with Errors 4.2.1. Plane in check state Fabric plane with any error will be shown in check state. Use the CLI command show chassis fabric summary to identify such fabric planes.
root@testmx> show chassis fabric summary Plane State Uptime 0 Offline 1 Online 1 day, 21 hours, 35 minutes, 55 seconds 2 Online 1 day, 20 hours, 33 minutes, 20 seconds 3 Check 1 day, 21 hours, 35 minutes, 55 seconds 4 Online 1 day, 20 hours, 25 minutes, 31 seconds 5 Spare 1 day, 20 hours, 31 minutes, 23 seconds
{MASTER.EN_US} In the above output, plane 0 is offlined, and plane 3 is in check state. If plane 0 is not offlined due to operation requirement, it is most likely offlined by fabric hardening and can be identified in log files. Please follow section 5 fabric errors recovery procedure to recover from the plane error. Plane 3 above is in check state indicating some fabric related errors have occurred on the plane. The error may be triggered by a failed fpc link, or a failed fabric link. If any of the plane is in check state, use CLI commands
show chassis fabric fpc and show chassis fabric plane
to identify the specific failure point.
4.2.2. Links in error state This section describes the details of the error states: Destination errors, Link errors, and Unused. The example uses output from the CLI command show chassis fabric fpc . Output from from show chassis fabric plane should also be verified. 4.2.2.1. Destination errors
root@testmx> show chassis fabric fpcs Fabric management FPC state: FPC 1 PFE #0 Plane 0: Links ok Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Destination error Plane 4: Plane enabled Plane 5: Links ok
In above output, FPC 1, PFE#0 is showing "Destination error" on plane 3. This indicates that plane 3 on the FPC is not usable by the FPC and cannot carry traffic. Please follow Section 5 Fabric Error Recovery procedure to recover from this error. 4.2.2.2. Link error
root@testmx> show chassis fabric fpcs Fabric management FPC state: FPC 2 PFE #0 Plane 0: Link error Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Plane enabled Plane 5: Links ok
In above output, FPC 2, PFE#0 is showing "Link error" on plane 0. This indicates that plane 0 on the PFE encounters either CRC errors, or serdes training error and MAY NOT be usable by the FPC to carry traffic. If the messages log file contains the string "CRC errors found", the link error is likely triggered by a CRC error. A CRC error is most likely caused by degraded hardware, or non-optimized serdes coefficient. Please contact JTAC to determine if upgrading to a software release supporting optimized serdes coefficient is required to correct the CRC error issue. If this is determined to be a serdes training issue, please follow Section 5 Fabric Error Recovery Procedure to recover from this error. 4.2.2.3. Unused links
root@testmx> show chassis fabric fpcs Fabric management FPC state: FPC 4 PFE #3 Plane 0: Unused Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Plane enabled Plane 5: Links ok
In the above output, FPC 4, PFE#3 is showing link state as Unused on plane 0. This indicates the serdes link is not trained and cannot be used to carry traffic without first training the serdes link. On MX480 and MX240, this can be the normal state for MPCs that have shared serdes links to fabric planes 1 and 5, and planes 3 and 7 where only one link of the pair can be trained up and the other will be in Unused or Disabled state. For example, MPC4E, and MPC5E etc. For all other types of linecards, the Unused link state is unexpected and indicates errors in fabric link bring up procedure. 5. Fabric Error Recovery Procedure If a plane or FPC is offlined by fabric hardening action, the following recovery procedure is recommended. Perform the actions in the order specified and stop when there are no errors present in the show output.
Unused
show
a. For FPC offlined due to unreachable destinations, restart these FPC one-by-one. Verify that the FPC comes up without error when the command show chassis fabric fpc is used. b. For fabric planes offlined by fabric hardening action, online these fabric planes one-by-one. Verify that the plane comes up without error by using the commands show chassis fabric fpc and show chassis fabric plane . c. Soft-restart or GRES chassisd d. Hard-restart chassisd
If after performing all the actions above some FPCs or plane/SCBs remain in an error state, contact Juniper JTAC for further investigation and action. 6. Sample Output for Working Fabric States The section shows a typical CLI output from an error-free MX960. 6.1 Output from show chassis fabric summary : Note that all plane states should be either online or spare. Typically there will be four online planes and two (four for MX480/MX240) spare planes if the device operates in redundant fabric mode. If the device operates in increased-bandwidth fabric mode, there will be six online planes for all MX devices and two additional spare planes for MX 480/240.
{MASTER.EN_US} root@oceanid> show chassis fabric summary Plane State Uptime 0 Online 48 minutes, 59 seconds 1 Online 56 minutes, 31 seconds 2 Online 56 minutes, 30 seconds 3 Online 43 minutes, 23 seconds 4 Spare 43 minutes, 20 seconds 5 Spare 56 minutes, 30 seconds
root@oceanid> show chassis fabric summary Plane State Uptime 0 Online 48 minutes, 59 seconds 1 Online 56 minutes, 31 seconds 2 Online 56 minutes, 30 seconds 3 Online 43 minutes, 23 seconds 4 Spare 43 minutes, 20 seconds 5 Spare 56 minutes, 30 seconds
{MASTER.EN_US}
6.2 Output from show chassis fabric fpc The PFE's plane state should be Plane enabled for online planes, and Links ok or Unused for spare planes.
Plane enabled
Links ok
root@testmx> show chassis fabric fpcs Fabric management FPC state: FPC 1 PFE #0 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok PFE #1 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok FPC 2 PFE #0 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok PFE #1 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok PFE #2 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok PFE #3 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok {MASTER.EN_US}
root@testmx> show chassis fabric fpcs Fabric management FPC state: FPC 1 PFE #0 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok PFE #1 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok FPC 2 PFE #0 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok PFE #1 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok PFE #2 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok PFE #3 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok
6.2 Output from show chassis fabric plane
All states should be Links ok or Unused if the plane state is ACTIVE or SPARE. Unused links may indicate error unless they are on planes 1 and 5, planes 3 and 7 and for MPC cards that use shared fabric links. Refer to section 5.2.3 for more details.
root@testmx> show chassis fabric plane Fabric management PLANE state Plane 0 Plane state: ACTIVE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 1 Plane state: ACTIVE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 2 Plane state: ACTIVE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 3 Plane state: ACTIVE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 4 Plane state: SPARE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 5 Plane state: SPARE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok {MASTER.EN_US}
root@testmx> show chassis fabric plane Fabric management PLANE state Plane 0 Plane state: ACTIVE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 1 Plane state: ACTIVE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 2 Plane state: ACTIVE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 3 Plane state: ACTIVE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 4 Plane state: SPARE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok Plane 5 Plane state: SPARE FPC 1 PFE 0 :Links ok PFE 1 :Links ok FPC 2 PFE 0 :Links ok PFE 1 :Links ok PFE 2 :Links ok PFE 3 :Links ok