Description

This article describes the scenario when CHASSISD_HIGH_TEMP_CONDITION syslog message is generated, minor chassis alarm "Temperature Warm" is raised and corrective actions that should be taken.

Symptoms

  • Minor chassis alarm "Temperature Warm" is active:
user@device> show chassis alarms 
1 alarms currently active
Alarm time      Class  Description
<date> <time>   Minor  Temperature Warm
  • CHASSISD_HIGH_TEMP_CONDITION syslog message is generated:
<date> <time>  device chassisd[13585]: %DAEMON-5-CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed [system warm]
<date> <time>  device alarmd[14725]: %DAEMON-4: Alarm set: Temp sensor id=33554534, color=YELLOW, class=CHASSIS, reason=Temperature Warm
<date> <time>  device craftd[13588]: %DAEMON-4:  Minor alarm set, Temperature Warm
<date> <time>  device chassisd[13585]: %DAEMON-4-CHASSISD_HIGH_TEMP_CONDITION: Chassis temperature over 90 degrees C (but no fan/impeller failure detected)

Solution

CHASSISD_HIGH_TEMP_CONDITION syslog message and "Temperature Warm" alarm are generated when the lower temperature threshold is exceeded for one or more FRUs: 
user@device> help syslog CHASSISD_HIGH_TEMP_CONDITION
Name:          CHASSISD_HIGH_TEMP_CONDITION
Message:       Chassis temperature over <temperature> degrees C (<message>)
Help:          Temperature exceeded lower threshold
Description:   The temperature of one or more components (field-replaceable units, or FRUs) exceeded the indicated temperature, which is the lower of two thresholds. The fans (and impellers, if applicable) were in the indicated state.
               The chassis process (chassisd) increased the speed of all functioning fans and impellers to full speed.
Type:          Event: This message reports an event, not an error
Severity:      warning
Facility:      LOG_DAEMON
Action:        Increase cooling in the area around the chassis.
The current component temperatures can be monitored by "show chassis environment" command, for example: 
user@device> show chassis environment
Class Item                           Status     Measurement
Temp  PEM 0                          OK         45 degrees C / 113 degrees F
      PEM 1                          OK         50 degrees C / 122 degrees F
      PEM 2                          OK         45 degrees C / 113 degrees F
      PEM 3                          OK         45 degrees C / 113 degrees F
      Routing Engine 0               OK         37 degrees C / 98 degrees F
      Routing Engine 0 CPU           OK         33 degrees C / 91 degrees F
      Routing Engine 1               OK         36 degrees C / 96 degrees F
      Routing Engine 1 CPU           OK         32 degrees C / 89 degrees F
      CB 0 Intake                    OK         35 degrees C / 95 degrees F
      CB 0 Exhaust A                 OK         32 degrees C / 89 degrees F
      CB 0 Exhaust B                 OK         41 degrees C / 105 degrees F
      CB 0 ACBC                      OK         40 degrees C / 104 degrees F
      CB 0 XF A                      OK         54 degrees C / 129 degrees F
      CB 0 XF B                      OK         53 degrees C / 127 degrees F
      CB 1 Intake                    OK         35 degrees C / 95 degrees F
      CB 1 Exhaust A                 OK         33 degrees C / 91 degrees F
      CB 1 Exhaust B                 OK         42 degrees C / 107 degrees F
      CB 1 ACBC                      OK         39 degrees C / 102 degrees F
      CB 1 XF A                      OK         57 degrees C / 134 degrees F
      CB 1 XF B                      OK         55 degrees C / 131 degrees F
      CB 2 Intake                    OK         33 degrees C / 91 degrees F
      CB 2 Exhaust A                 OK         31 degrees C / 87 degrees F
      CB 2 Exhaust B                 OK         42 degrees C / 107 degrees F
      CB 2 ACBC                      OK         37 degrees C / 98 degrees F
      CB 2 XF A                      OK         56 degrees C / 132 degrees F
      CB 2 XF B                      OK         55 degrees C / 131 degrees F
      FPC 10 Intake                  OK         42 degrees C / 107 degrees F
      FPC 10 Exhaust A               OK         60 degrees C / 140 degrees F
      FPC 10 Exhaust B               OK         55 degrees C / 131 degrees F
      FPC 10 XL TSen                 OK         51 degrees C / 123 degrees F
      FPC 10 XL Chip                 OK         52 degrees C / 125 degrees F
      FPC 10 XL_XR0 TSen             OK         51 degrees C / 123 degrees F
      FPC 10 XL_XR0 Chip             OK         52 degrees C / 125 degrees F
      FPC 10 XL_XR1 TSen             OK         51 degrees C / 123 degrees F
      FPC 10 XL_XR1 Chip             OK         53 degrees C / 127 degrees F
      FPC 10 XQ TSen                 OK         51 degrees C / 123 degrees F
      FPC 10 XQ Chip                 OK         57 degrees C / 134 degrees F
      FPC 10 XQ_XR0 TSen             OK         51 degrees C / 123 degrees F
      FPC 10 XQ_XR0 Chip             OK         61 degrees C / 141 degrees F
      FPC 10 XQ_XR1 TSen             OK         51 degrees C / 123 degrees F
      FPC 10 XQ_XR1 Chip             OK         61 degrees C / 141 degrees F
      FPC 10 XM 0 TSen               OK         60 degrees C / 140 degrees F
      FPC 10 XM 0 Chip               OK         69 degrees C / 156 degrees F
      FPC 10 XM 1 TSen               OK         60 degrees C / 140 degrees F
      FPC 10 XM 1 Chip               OK         53 degrees C / 127 degrees F
      FPC 10 PLX PCIe Switch TSen    OK         60 degrees C / 140 degrees F
      FPC 10 PLX PCIe Switch Chip    OK         81 degrees C / 177 degrees F
Fans  Top Fan Tray Temp              OK         46 degrees C / 114 degrees F
      Top Tray Fan 1                 OK         Spinning at normal speed
      Top Tray Fan 2                 OK         Spinning at normal speed
      Top Tray Fan 3                 OK         Spinning at normal speed
      Top Tray Fan 4                 OK         Spinning at normal speed
      Top Tray Fan 5                 OK         Spinning at normal speed
      Top Tray Fan 6                 OK         Spinning at normal speed
      Top Tray Fan 7                 OK         Spinning at normal speed
      Top Tray Fan 8                 OK         Spinning at normal speed
      Top Tray Fan 9                 OK         Spinning at normal speed
      Top Tray Fan 10                OK         Spinning at normal speed
      Top Tray Fan 11                OK         Spinning at normal speed
      Top Tray Fan 12                OK         Spinning at normal speed
      Bottom Fan Tray Temp           OK         37 degrees C / 98 degrees F
      Bottom Tray Fan 1              OK         Spinning at normal speed
      Bottom Tray Fan 2              OK         Spinning at normal speed
      Bottom Tray Fan 3              OK         Spinning at normal speed
      Bottom Tray Fan 4              OK         Spinning at normal speed
      Bottom Tray Fan 5              OK         Spinning at normal speed
      Bottom Tray Fan 6              OK         Spinning at normal speed
      Bottom Tray Fan 7              OK         Spinning at normal speed
      Bottom Tray Fan 8              OK         Spinning at normal speed
      Bottom Tray Fan 9              OK         Spinning at normal speed
      Bottom Tray Fan 10             OK         Spinning at normal speed
      Bottom Tray Fan 11             OK         Spinning at normal speed
      Bottom Tray Fan 12             OK         Spinning at normal speed
The current temperature thresholds can be checked by using "show chassis temperature-thresholds" command: 
user@device> show chassis temperature-thresholds
                                        Fan speed      Yellow alarm      Red alarm      Fire Shutdown
                                       (degrees C)      (degrees C)     (degrees C)      (degrees C)
Item                                  Normal  High   Normal  Bad fan   Normal  Bad fan     Normal
Chassis default                           48    54       65       55       80       65        100
Routing Engine 0                          70    80       95       95      110      110        112
Routing Engine 1                          70    80       95       95      110      110        112
FPC 10 Intake                             52    60       70       65       81       76        100
FPC 10 Exhaust A                          72    80       91       86      102       97        115
FPC 10 Exhaust B                          73    81       85       80       96       91        109
FPC 10 XL Chip                            85    95       95       92      105      100        110
FPC 10 XL_XR0 Chip                        85    95       95       92      105      100        110
FPC 10 XL_XR1 Chip                        85    95       95       92      105      100        110
FPC 10 XQ Chip                            85    95       95       92      105      100        110
FPC 10 XQ_XR0 Chip                        85    95       95       92      105      100        110
FPC 10 XQ_XR1 Chip                        85    95       95       92      105      100        110
FPC 10 XM 0 Chip                          75    80       90       87      100       95        105
FPC 10 XM 1 Chip                          75    80       90       87      100       95        105
FPC 10 PLX PCIe Switch Chip               80    85       95       95      105      105        120
Top Fan Tray 0                            48    54       77       67       90       80        120
Bottom Fan Tray 1                         48    54       77       67       90       80        120
To identify which component triggered this alarm, /var/log/chassisd file can be checked. In the example below the alarm was raised at 19:58:21 and cleared at 19:58:31 due to FPC 10 PLX PCIe Switch Chip temperature exceeding the lower temperature threshold: 
Nov 27 19:58:21  device chassisd[13585]: %DAEMON-5-CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed [system warm]
Nov 27 19:58:21  device alarmd[14725]: %DAEMON-4: Alarm set: Temp sensor id=33554534, color=YELLOW, class=CHASSIS, reason=Temperature Warm
Nov 27 19:58:21  device craftd[13588]: %DAEMON-4:  Minor alarm set, Temperature Warm
Nov 27 19:58:21  device chassisd[13585]: %DAEMON-4-CHASSISD_HIGH_TEMP_CONDITION: Chassis temperature over 90 degrees C (but no fan/impeller failure detected)
Nov 27 19:58:31  device alarmd[14725]: %DAEMON-4: Alarm cleared: Temp sensor id=33554534, color=YELLOW, class=CHASSIS, reason=Temperature Warm
Nov 27 19:58:31  device craftd[13588]: %DAEMON-4: Minor alarm cleared, Temperature Warm
Chassisd log file contains more information: 
Nov 27 19:58:20  i2c_check_temp_limits: FPC 10 temperature sensor PLX PCIe Switch Chip has temperature 96 above yellow alarm temperature 95
Nov 27 19:58:21 CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed [system warm]
Nov 27 19:58:21  send: yellow alarm set, device Temp Sensor, reason Temperature Warm
Nov 27 19:58:21 CHASSISD_HIGH_TEMP_CONDITION: Chassis temperature over 90 degrees C (but no fan/impeller failure detected)
Nov 27 19:58:26  i2c_check_temp_limits: FPC 10 temperature sensor PLX PCIe Switch Chip has temperature 96 above yellow alarm temperature 95
Nov 27 19:58:31  send: yellow alarm clear, device Temp Sensor, reason Temperature Warm

Follow these steps to troubleshoot this issue:

  1. Check the surrounding environment to the chassis, to verify that airflow to and from the unit is not being restricted. Air must be allowed to enter and exit via only the air vents on the chassis. Vent restriction will lower the effectiveness of the cooling for the system. Also check to make sure that air inflow is not next to the exhaust of another system generating heated air, as this will also reduce cooling efficiently.
  2. Check the air filters on the platform and verify that they are clean (please refer platform specific documentation regarding the air filter maintenance). For some platforms it's recommended to replace air filters every 6 months.
  3. Verify that empty FPC slots have a blank cover plates installed, to preserve the integrity of the internal airflow inside the chassis. All system components (FPCs, CBs, PICs, REs, etc.) should fit snugly in their slots and not have gaps that allow air flow around the component to the outside environment.
  4. Verify that the fans are operating by either visual and/or audio observation. If a fan or fan tray does not seem to be running as expected, or there are log messages reporting a fan failure, try reseating the fan in its slot. If there are still messages reporting a fan failure, contact Support for further investigation.
  5. If a specific component is reporting as being over temperature, or is reporting a temperature sensor failure, try reseating that component. Some components, such as FPCs, may also be moved to a different slot in the chassis to see if that resolves the issue.
  6. If there are still messages reporting component temperature rising steadily, or over temperature, or a temperature sensor failure at this point, contact Support for further investigation.

Modification History

2023-12-13 : Article Created

Related Information

https://www.juniper.net/documentation/us/en/hardware/mx960/topics/task/air-filter-mx960-maintain.html
https://supportportal.juniper.net/s/article/SRX-Understanding-how-temperature-sensors-affect-FAN-speed-and-Alarm