This article reviews the basics behind MX Fabric Planes. It is referenced from KB23173 - Resolution Guide - Troubleshoot Fabric Plane [juniper.net] .
Describe MX Fabric (Planes and SCB).
Click one of the following links to jump to that topic:
Introduction to MX SCB
Packet Forwarding and Switch Fabric
MX960 Switch Fabric Diagram
MX480 / MX240 Switch Fabric
Fabric behavior when chassis has MX SCBE (1-3)
Fabric behavior when chassis has a mix of MPC and DPC cards and MX SCB
The MX Switch Control Board (SCB) provides Control Plane functions, Chassis management functions, and Switch Plane functions. It is also a carrier for the Routing Engine (RE). Below are various facts about the SCB:
Each SCB contains two fabric ASICs, and the same SCB card is usable in MX960, MX480, and MX240. The Control Plane functionality is provided primarily by an RE, an ACBC FPGA, and a gigabit Ethernet switch, all of which reside on the SCB. The RE is a Pentium processor subsystem that can be plugged in to the SCB. The RE uses two primary interfaces to the SCB: PCI bus and gigabit Ethernet. On the SCB, the RE’s PCI bus interfaces with the ACBC FPGA, which aggregates most of the system I/O and provides Chassis management functions. The RE’s gigabit Ethernet port connects to a gigabit Ethernet switch on the SCB. This gigabit Ethernet switch provides Ethernet connectivity to all of the processors in the Chassis, including the standby RE, for Control Plane communications. There are two SCBs in the system, which can operate the control plane function in redundant mode. Each SCB contains the mastership logic that determines which SCB/RE combination is the primary in the system. The primary is also called the Junos primary. The Junos primary controls all the components in the system, including the standby SCB/RE. Each SCB generates an output mastership signal to each FRU. Each SCB also generates an output operational signal to each FRU. On each FRU, the mastership signal is qualified by the operational signal and combined with local status signals to select the primary SCB that is used for FRU control. Each SCB contains two switch fabric chips (ASICs). However, the switch fabric ASICs are under the control of the current Junos primary. Each SCB provides two planes of switch fabric for packet forwarding among the DPCs/MPCs for MX960 and four planes for MX480/MX240. On MX480/MX240, each fabric ASIC is configured in virtual plane mode, where two virtual planes exist on one fabric ASIC. The MX960 chassis may contain up to three SCBs and hence 6 planes are available. The MX480/MX240 contains a maximum of 2 SCBs and hence 8 logical planes are available. On MX960, the two SCBs residing in slot 6 and slot 7 provide both Control and Switch Fabric features, whereas the third SCB residing in slot 8 of the chassis (hybrid slot) will only do fabric functions. Either an SCB or a DPC can be plugged in to slot 8. A third SCB is used only for Switch Fabric redundancy. Therefore, if an application does not require switch fabric redundancy, an FPC can be used in slot 8.
Each SCB contains two fabric ASICs, and the same SCB card is usable in MX960, MX480, and MX240.
The Control Plane functionality is provided primarily by an RE, an ACBC FPGA, and a gigabit Ethernet switch, all of which reside on the SCB.
The RE is a Pentium processor subsystem that can be plugged in to the SCB. The RE uses two primary interfaces to the SCB: PCI bus and gigabit Ethernet.
On the SCB, the RE’s PCI bus interfaces with the ACBC FPGA, which aggregates most of the system I/O and provides Chassis management functions.
The RE’s gigabit Ethernet port connects to a gigabit Ethernet switch on the SCB. This gigabit Ethernet switch provides Ethernet connectivity to all of the processors in the Chassis, including the standby RE, for Control Plane communications.
There are two SCBs in the system, which can operate the control plane function in redundant mode. Each SCB contains the mastership logic that determines which SCB/RE combination is the primary in the system. The primary is also called the Junos primary. The Junos primary controls all the components in the system, including the standby SCB/RE.
Each SCB generates an output mastership signal to each FRU. Each SCB also generates an output operational signal to each FRU. On each FRU, the mastership signal is qualified by the operational signal and combined with local status signals to select the primary SCB that is used for FRU control.
Each SCB contains two switch fabric chips (ASICs). However, the switch fabric ASICs are under the control of the current Junos primary.
Each SCB provides two planes of switch fabric for packet forwarding among the DPCs/MPCs for MX960 and four planes for MX480/MX240.
On MX480/MX240, each fabric ASIC is configured in virtual plane mode, where two virtual planes exist on one fabric ASIC. The MX960 chassis may contain up to three SCBs and hence 6 planes are available. The MX480/MX240 contains a maximum of 2 SCBs and hence 8 logical planes are available.
On MX960, the two SCBs residing in slot 6 and slot 7 provide both Control and Switch Fabric features, whereas the third SCB residing in slot 8 of the chassis (hybrid slot) will only do fabric functions. Either an SCB or a DPC can be plugged in to slot 8. A third SCB is used only for Switch Fabric redundancy. Therefore, if an application does not require switch fabric redundancy, an FPC can be used in slot 8.
The MX Series uses a distributed Packet Forwarding architecture. Each DPC/MPC contains 1/2/4 PFEs, each of which is self contained in terms of forwarding decisions. When the forwarding decision points to an output interface on a different PFE, which can be either on the same DPC/MPC or on a different DPC/MPC, the packet needs to be sent across the Fabric Plane. Since Packet Forwarding is implemented using a distributed architecture, the fabric architecture is both distributed and fault-tolerant. The fabric interconnect ASICs are housed in SCBs that are fully redundant on every platform. Each PFE implements the fabric queuing and flow control mechanisms that are required to communicate with multiple other PFEs on the chassis at the same time. A request-grant mechanism is used to implement flow control. Each PFE that wants to send a packet to a destination PFE sends a request, and only when the request is granted is it allowed to send the packet to the destination PFE.
The MX Series uses a distributed Packet Forwarding architecture. Each DPC/MPC contains 1/2/4 PFEs, each of which is self contained in terms of forwarding decisions. When the forwarding decision points to an output interface on a different PFE, which can be either on the same DPC/MPC or on a different DPC/MPC, the packet needs to be sent across the Fabric Plane.
Since Packet Forwarding is implemented using a distributed architecture, the fabric architecture is both distributed and fault-tolerant. The fabric interconnect ASICs are housed in SCBs that are fully redundant on every platform. Each PFE implements the fabric queuing and flow control mechanisms that are required to communicate with multiple other PFEs on the chassis at the same time. A request-grant mechanism is used to implement flow control. Each PFE that wants to send a packet to a destination PFE sends a request, and only when the request is granted is it allowed to send the packet to the destination PFE.
The MX960 Switch Fabric connectivity is as follows:
Each PFE is connected to all Fabric Planes.
The diagram shows only two FPCs in the chassis.
All other DPCs are also connected in the same way.
There are 4 Active Planes and 2 Spare Planes.
For additional details on the MX960 router, refer to MX960 3D Universal Edge Router - Components .
The MX480/MX240 Switch Fabric connectivity is as follows:
For additional details on the MX480 router, refer to MX480 3D Universal Edge Router - Components .
For additional details on the MX240 router, refer to MX240 3D Universal Edge Router - Components .
A chassis in the MX960 platform uses FOUR of the fabric planes in ACTIVE mode and TWO fabric planes in SPARE mode, thus providing fabric redundancy. The FPC cards will use only two of the SCBEs as ACTIVE and the third SCBE is a SPARE. If one of the two ACTIVE cards fail, then it will use the third SCBE as ACTIVE and will continue with full fabric bandwidth capability.
The MX480 and MX240 routers have similar behavior, where 4 fabric planes from one SCBE will be ACTIVE and 4 fabric planes from the second SCBE will be SPARE from the FPC perspective.
Note: The MX router can be configured with "chassis fabric redundancy-mode increased-bandwidth" for additional FPC throughput; however, if this configuration is done, plane redundancy will be lost.
A chassis (with DPC cards) in the MX960 platform uses FOUR of the fabric planes in ACTIVE mode and TWO fabric planes in SPARE mode, thus providing fabric redundancy. If MPC cards are added into the chassis, all SIX planes are in ACTIVE mode, thus providing more fabric bandwidth. However, with the MPC cards, there will not be any fabric redundancy; the DPC cards will still have fabric redundancy capabilities in a case where all three SCBs are present. The DPC cards will use only two of the SCBs as ACTIVE and the third SCB is a SPARE. If one of the two ACTIVE cards fail, then it will use the third SCB as ACTIVE and will continue with full fabric bandwidth capability. From a DPC card perspective, two SCBs are ACTIVE and one SCB is SPARE. From the chassis perspective and from MPC cards perspective, all 3 SCBs are ACTIVE.
The MX480 and MX240 routers have similar behavior, where 4 fabric planes from one SCB will be ACTIVE and 4 fabric planes from the second SCB will be SPARE from the DPC perspective. All eight planes will be ACTIVE from the MPC and chassis perspective.
The behavior of the chassis can be summarized as follows (for MX960, and MX480 and MX240 have similar behavior):
When an MPC card is inserted in the chassis, all 6 fabric planes are marked as ACTIVE in the chassis.
There will be NO concept of fabric redundancy from a chassis perspective. However, internally, fabric redundancy will be available for DPC cards ONLY.
The MPC will communicate with the DPC cards only through the FOUR planes that are considered ACTIVE in the chassis. These will be the planes that I-Chip considers as ACTIVE.
The MPC cards will communicate with other MPC cards through ALL 6 planes in the chassis.
The CLI will indicate that ALL 6 planes are ACTIVE.
Active fabric LED is turned “ON” for all three SCBs.
2022-03-03: Updated to include MX SBE 1-3 models and corresponding redundancy behavior