This article explains some important aspects and key hardware designs should be considered while SFB or FAN-Tray re-seat/replacement in MX-10008.
Incase of SFB replacement/re-seat you would disconnect FAN-TRAY and this will trigger chassis temperature to high threshold despite another FAN-TRAY is active and device will go down due to high temperature.In this situation if you try to quickly connect the FAN-TRAY back to avoid device failure it will still give alarms for both FAN-TRAY and device will go down.
Example :
Task: SFB-3 re-seat or FAN_TRAY-1 replacement
Actions performed : Disconnected FAN-TRAY-1 and re-seat performed on FAN_TRAY-1.
FAN-TRAY-1 connected back however alarm raised for fan-failure for both FAN-TRAY's. This would cause the device to go down due to high temperature.
Router> show system alarms 32 alarms currently active Alarm time Class Description 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 21 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 20 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 19 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 18 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 17 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 14 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 12 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 11 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 10 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 9 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 7 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 4 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 3 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 2 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 1 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 0 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 20 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 19 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 18 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 17 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 14 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 12 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 11 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 10 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 9 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 7 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 4 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 3 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 2 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 1 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 0 Failed 2024-04-16 05:31:03 UTC Major Fan Tray 1 Fan 21 Failed
If the device goes down, only recovery options is to drain the power completely by disconnecting PEM and connect PEM back to power-cycle the device.
To avoid this issue and have a smooth SFB/FAN-TRAY replacement/re-seat you can use below tested method.
Step-1 : Use below hidden commands to set the both FAN-TRAY to full speed and keep it at least for 10 minutes before replacing the fans or SFBs.Commands :"test chassis fan tray 0 speed full-speed" and "test chassis fan tray 1 speed full-speed"
Step-2 : Replace/Re-seat SFB/ FAN-TRAY and verify for any alarms.
Step-3 : Normalise the FAN speed to normal back using below command.
Commands : "test chassis fan tray 0 speed normal speed" and "test chassis fan tray 1 speed normal speed"
For removing and Installing MX10008 Cooling System Components: https://www.juniper.net/documentation/us/en/hardware/mx10008/topics/topic-map/mx10008-replace-cooling-system-components.html
SFB hardware guide for MX-10008 : https://www.juniper.net/documentation/us/en/hardware/mx10008/topics/topic-map/mx10008-16-switch-fabric.html