Description

This article explains some important aspects and key hardware designs should be considered while SFB or FAN-Tray re-seat/replacement in MX-10008.

Symptoms

Incase of SFB replacement/re-seat you would disconnect FAN-TRAY and this will trigger chassis temperature to high threshold despite another FAN-TRAY is active and device will go down due to high temperature.

In this situation if you try to quickly connect the FAN-TRAY back to avoid device failure it will still give alarms for both FAN-TRAY and device will go down.
 

Example :

Task: SFB-3 re-seat or FAN_TRAY-1 replacement

Actions performed : Disconnected FAN-TRAY-1 and re-seat performed on FAN_TRAY-1.

FAN-TRAY-1 connected back however alarm raised for fan-failure for both FAN-TRAY's. This would cause the device to go down due to high temperature.

 

Router> show system alarms   

32 alarms currently active

Alarm time        Class Description

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 21 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 20 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 19 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 18 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 17 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 14 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 12 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 11 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 10 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 9 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 7 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 4 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 3 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 2 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 1 Failed

2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 0 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 20 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 19 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 18 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 17 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 14 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 12 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 11 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 10 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 9 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 7 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 4 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 3 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 2 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 1 Failed

2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 0 Failed

2024-04-16 05:31:03 UTC Major Fan Tray 1 Fan 21 Failed

 

 

Solution

If the device goes down, only recovery options is to drain the power completely by disconnecting PEM and connect PEM back to power-cycle the device.
 

To avoid this issue and have a smooth SFB/FAN-TRAY replacement/re-seat you can use below tested method. 

Step-1 : Use below hidden commands to set the both FAN-TRAY to full speed and keep it at least for 10 minutes before replacing the fans or SFBs.
Commands :"test chassis fan tray 0 speed full-speed" and "test chassis fan tray 1 speed full-speed"

 

Step-2 : Replace/Re-seat SFB/ FAN-TRAY and verify for any alarms.

 

Step-3 : Normalise the FAN speed to normal back using below command.

Commands : "test chassis fan tray 0 speed normal speed" and "test chassis fan tray 1 speed normal speed"

 

For removing and Installing MX10008 Cooling System Components: https://www.juniper.net/documentation/us/en/hardware/mx10008/topics/topic-map/mx10008-replace-cooling-system-components.html

 

SFB hardware guide for MX-10008 : https://www.juniper.net/documentation/us/en/hardware/mx10008/topics/topic-map/mx10008-16-switch-fabric.html

Modification History

2024-04-17 : Article Created