This article explains some important aspects and key hardware designs should be considered while SIB or FAN-Tray re-seat/replacement in PTX-10008 and PTX-100016.
In case of FAN-TRAY or SIB replacement/re-seat you would disconnect FAN-TRAY and this will trigger chassis temperature to high threshold despite another FAN-TRAY is active and device will go down due to high temperature.
In this situation if you try to quickly connect the FAN-TRAY back to avoid device failure it will still give alarms for both FAN-TRAY and device will go down.
Example :
Task: FAN_TRAY-1 replacement
Actions performed : Disconnected FAN-TRAY-1 and re-seat/ reboot performed on FAN_TRAY-1.
FAN-TRAY-1 connected back however alarm raised for fan-failure for both FAN-TRAY's. This would cause the device to go down due to high temperature.
Router> show system alarms 32 alarms currently active Alarm time Class Description 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 21 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 20 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 19 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 18 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 17 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 14 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 12 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 11 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 10 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 9 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 7 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 4 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 3 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 2 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 1 Failed 2024-04-16 05:44:48 UTC Major Fan Tray 0 Fan 0 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 20 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 19 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 18 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 17 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 14 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 12 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 11 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 10 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 9 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 7 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 4 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 3 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 2 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 1 Failed 2024-04-16 05:44:18 UTC Major Fan Tray 1 Fan 0 Failed 2024-04-16 05:31:03 UTC Major Fan Tray 1 Fan 21 Failed
NOTE: If the device goes down, the only recovery option is to drain the power completely by disconnecting PEM and connect PEM back to power-cycle the device.
To avoid this issue and have a smooth FAN-TRAY or SIB replacement/re-seat, you can usethe below tested method:
Step-1 : Use below hidden commands below to set both FAN-TRAYs to full speed and keep them at least for 10 minutes before replacing the fans or SFBs.
Router> test chassis fan tray 0 speed full-speed Router> test chassis fan tray 1 speed full-speed
Step-2 : Please have the ISP promptly replace or re-seat the FAN-TRAY or SIB, and verify the system for any active alarms. This step should be completed as soon as possible to prevent chassis overheating.
Step-3 : Monitor the chassis components temperature throughout the activity using below command:
Router> show chassis environment Router> show chassis environment fpc Router> show chassis environment sib Router> show chassis ambient-temperature
Step-4: In case a high-temperature alarm is observed on the SIB or FPC, the re-seated FAN tray can be set to high-speed mode again using the same hidden command to improve chassis cooling.
Router> test chassis fan tray <> speed full-speed
Step-5 : After confirming that all chassis components have returned to normal temperature and no temperature alarms are active, set the FAN speed back to normal using the following command.
Router> test chassis fan tray 0 speed normal Router> test chassis fan tray 1 speed normal
Example : Router> show chassis environment | match fan Fans Fan Tray 0 Fan 0 OK Spinning at normal speed Fan Tray 0 Fan 1 OK Spinning at normal speed Fan Tray 0 Fan 2 OK Spinning at normal speed Fan Tray 0 Fan 3 OK Spinning at normal speed Fan Tray 0 Fan 4 OK Spinning at normal speed Fan Tray 0 Fan 5 OK Spinning at normal speed Fan Tray 0 Fan 6 OK Spinning at normal speed Fan Tray 0 Fan 7 OK Spinning at normal speed Fan Tray 0 Fan 8 OK Spinning at normal speed Fan Tray 0 Fan 9 OK Spinning at normal speed Fan Tray 0 Fan 10 OK Spinning at normal speed Router> test chassis fan tray 0 speed full-speed Action successful Router> show chassis environment | match fan Fans Fan Tray 0 Fan 0 OK Spinning at full speed Fan Tray 0 Fan 1 OK Spinning at full speed Fan Tray 0 Fan 2 OK Spinning at full speed Fan Tray 0 Fan 3 OK Spinning at full speed Fan Tray 0 Fan 4 OK Spinning at full speed Fan Tray 0 Fan 5 OK Spinning at full speed Fan Tray 0 Fan 6 OK Spinning at full speed Fan Tray 0 Fan 7 OK Spinning at full speed Fan Tray 0 Fan 8 OK Spinning at full speed Fan Tray 0 Fan 9 OK Spinning at full speed Fan Tray 0 Fan 10 OK Spinning at full speed Router> test chassis fan tray 0 speed normal Action successful Router> show chassis environment | match "Fan Tray 0" Fans Fan Tray 0 Fan 0 OK Spinning at normal speed Fan Tray 0 Fan 1 OK Spinning at normal speed Fan Tray 0 Fan 2 OK Spinning at normal speed Fan Tray 0 Fan 3 OK Spinning at normal speed Fan Tray 0 Fan 4 OK Spinning at normal speed Fan Tray 0 Fan 5 OK Spinning at normal speed Fan Tray 0 Fan 6 OK Spinning at normal speed Fan Tray 0 Fan 7 OK Spinning at normal speed Fan Tray 0 Fan 8 OK Spinning at normal speed Fan Tray 0 Fan 9 OK Spinning at normal speed Fan Tray 0 Fan 10 OK Spinning at normal speed