This article explains the factors influencing the prolonged timeframe for fabric healing on Juniper MX routers equipped with a single Control and Routing Engine (CB/RE).
On Juniper MX routers, fabric healing refers to the process of recovering fabric down condition which can be induced due to failed links or components. While fabric healing can vary in duration, it is noted that it may take prolonged timeframe under certain conditions.
In this user scenario, user faced a prolonged service outage when the only CB on the device failed due to hardware fault on control board, causing all fabric interfaces to go down creating a fabric down condition, to recover from this Fabric healing was triggered which took approximately 20 minutes (On MX240 Platform) of service out for customer (On MX240 Platform) before the traffic switched over to redundant device.
The Fabric boards and planes can go down due to multiple reasons which can transient or genuine hardware issue like voltage /power issues, planes being offlined by users, and other similar causes.
Fabric healing mechanisms occur only when blackholing events are detected. Either the loss of all fabric plane connections from a single PFE, or the detection of unreachable destination PFE’s can initiate fabric healing mechanisms due to the potential for ongoing disruption to network traffic.
Fabric healing mechanisms occur in three phases, attempting to automatically restore normal fabric connectivity with a minimum of disruption.
Phase-1 - Fabric Plane Restart : It restarts each fabric-plane component one at a time automatically. After executing Phase-1, if the issue re-occurs within 10 minutes, it is considered to be part of the same fabric healing event and Phase-2 is initiated.
Phase-2 - FPC and Fabric Plane Restart : This stage offlines any FPC’s that are currently experiencing destination errors. Performing this function results in interfaces on the afflicted line-card moving to a down state and allowing the network to reroute traffic around the blackholing router.
Phase-3 - FPC Offline : Any FPC’s that were not recovered by automatic healing mechanisms are offlined and remain offline until operator intervention. After the fabric healing actions have been performed, a 10 minute observation window begins. Recurring errors within this 10 minute period are treated as part of the same healing event and result in the FPC offline action being taken.
Errors occurring more than 10 minutes after fabric healing are treated as a new event.
Phase-1 is initiated for fabric component events. Unless all fabric components are offlined via CLI, FPC’s will have connections to some active planes so a total traffic blackhole does not exist. Fabric components are restarted and in the case of transient link errors, the error condition will likely be healed.
If restarting fabric components does not restore connectivity, Phase-2 and 3 continue to restart FPC’s and offline if necessary to prevent traffic being blackholed.
The Below may add more delays to the fabric healing procedure.
Hardware Configuration and Scale
Traffic Load
Automatic Healing Configurations
During a genuine Hardware failure of the CB/Fabric Card in the phase 1 of fabric healing there will be multiple attempts made (3 Re-Tries) to bring the plane online which may cause delayed outage for the end customer, in case there is no redundant CB/Fabric card available on the device. Hence in this specific scenario having a redundant CB/Fabric card on the device can help in avoiding the prolonged service outage caused due to absence of any spare planes.
There is a another feature available where the FPC's can be forcefully off-lined if a Fabric degradation is seen on the device as per user discretion.
The following is tested and FPC’s are off-lined only when the 3 planes are not available.
[edit]
labroot@salo-re0# show chassis
fpc 0 {
fabric {
bandwidth-degradation {
percentage 74;
action offline;
}
fpc 1 {
fpc 2 {
network-services enhanced-ip;
labroot@salo-re0# #
labroot@salo-re0> show chassis fabric degradation
Nov 03 21:07:06
Reqd/Curr Configured Current Time Last
FPC State Planes Degrad(%),action Degrad(%) Action Initiated
0 Online 4/4 74,offline none 2024-11-02 21:11:33 PDT
1 Online 4/4 74,offline none
2 Online 4/4 74,offline none
labroot@salo-re0>
After 1 plane is not available
Nov 03 21:07:50
0 Online 4/3 74,offline 25 2024-11-02 21:11:33 PDT
1 Online 4/3 74,offline 25
2 Online 4/3 74,offline 25
After two planes are not available
Nov 03 21:08:04
0 Online 4/2 74,offline 50 2024-11-02 21:11:33 PDT
1 Online 4/2 74,offline 50
2 Online 4/2 74,offline 50
labroot@salo-re0> show chassis fabric plane
Nov 03 21:08:23
Fabric management PLANE state
Plane 0
Plane state: OFFLINE
Plane 1
Plane 2
Plane state: ACTIVE
FPC 0
PFE 0 :Links ok
PFE 1 :Links ok
FPC 1
FPC 2
Plane 3
After 3 planes are not available
labroot@salo-re0> request chassis fabric plane 3 offline
Nov 03 21:08:34
Offline initiated, use "show chassis fabric plane" to verify
Nov 03 21:08:37
Nov 03 21:08:40
0 Online 4/1 74,offline 75 2024-11-02 21:11:33 PDT
1 Online 4/1 74,offline 75
2 Online 4/1 74,offline 75
labroot@salo-re0> show chassis alarms
Nov 03 21:15:29
5 alarms currently active
Alarm time Class Description
2024-11-03 21:08:46 PST Major FPC 2 offlined due to reduced fabric BW
2024-11-03 21:08:46 PST Major FPC 1 offlined due to reduced fabric BW
2024-11-03 21:08:46 PST Major FPC 0 offlined due to reduced fabric BW
2024-11-03 21:08:35 PST Minor CB 0 Fabric Chip 1 Not Online
2024-11-03 21:07:38 PST Minor CB 0 Fabric Chip 0 Not Online