On PTX3000 Series devices, customers may observe a high temperature syslog message for the backup Routing Engine and an "Over Temperature! SNMP trap.
This article clarifies that high temperature for the backup RE does not cause any service impact and therefore, customers can ignore the syslog message and the Simple Network Management Protocol (SNMP) trap.
While the high temperature is seen, we always see the JUNOScript root access with the backup RE.
*** RE0 messages *** Dec 13 04:41:21.887 PTX3K-re0 chassisd[12941]: %DAEMON-3-CHASSISD_RE_OVER_TEMP_WARNING: Routing Engine 1 temperature (96 C) over 82 degrees C, Routing Engine will shutdown in 240 seconds if condition persists Dec 13 04:41:21.887 PTX3K-re0 alarmd[13439]: %DAEMON-4: Alarm set: RE color=RED, class=CHASSIS, reason=Host 1 Temperature Hot Dec 13 04:41:21.887 PTX3K-re0 craftd[12944]: %DAEMON-4: Major alarm set, Host 1 Temperature Hot Dec 13 04:41:21.935 PTX3K-re0 chassisd[12941]: %DAEMON-5-CHASSISD_ZONE_BLOWERS_SPEED_FULL: Fans and impellers in zone 1 being set to full speed [system warm] Dec 13 04:41:23.249 PTX3K-re0 chassisd[12941]: %DAEMON-5- CHASSISD_SNMP_TRAP6: SNMP trap generated: Over Temperature! (jnxContentsContainerIndex 9, jnxContentsL1Index 2, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr Routing Engine 1, jnxOperatingTemp 96) Dec 13 04:41:26.965 PTX3K-re0 chassisd[12941]: %DAEMON-3-CHASSISD_RE_OVER_TEMP_WARNING: Routing Engine 1 temperature (96 C) over 82 degrees C, Routing Engine will shutdown in 235 seconds if condition persists Dec 13 04:41:27.516 PTX3K-re0 chassisd[12941]: %DAEMON-6: chassisd SIGCHLD handler: pid=98901 Dec 13 04:41:27.516 PTX3K-re0 chassisd[12941]: %DAEMON-6: chassisd SIGCHLD handler: pid=-1 Dec 13 04:41:31.905 PTX3K-re0 chassisd[12941]: %DAEMON-3-CHASSISD_RE_OVER_TEMP_WARNING: Routing Engine 1 temperature (96 C) over 82 degrees C, Routing Engine will shutdown in 230 seconds if condition persists Dec 13 04:41:32.438 PTX3K-re0 chassisd[12941]: %DAEMON-6: chassisd SIGCHLD handler: pid=98902 Dec 13 04:41:32.438 PTX3K-re0 chassisd[12941]: %DAEMON-6: chassisd SIGCHLD handler: pid=-1 Dec 13 04:41:37.591 PTX3K-re0 chassisd[12941]: %DAEMON-3-CHASSISD_RE_OVER_TEMP_WARNING: Routing Engine 1 temperature (96 C) over 82 degrees C, Routing Engine will shutdown in 224 seconds if condition persists Dec 13 04:41:38.143 PTX3K-re0 chassisd[12941]: %DAEMON-6: chassisd SIGCHLD handler: pid=98907 Dec 13 04:41:38.144 PTX3K-re0 chassisd[12941]: %DAEMON-6: chassisd SIGCHLD handler: pid=-1 Dec 13 04:41:41.949 PTX3K-re0 alarmd[13439]: %DAEMON-4: Alarm cleared: RE color=RED, class=CHASSIS, reason=Host 1 Temperature Hot Dec 13 04:41:41.949 PTX3K-re0 craftd[12944]: %DAEMON-4: Major alarm cleared, Host 1 Temperature Hot Dec 13 04:41:52.129 PTX3K-re0 chassisd[12941]: %DAEMON-5-CHASSISD_ZONE_BLOWERS_SPEED: Fans and impellers in zone 1 are now running at normal speed Dec 13 04:41:53.255 PTX3K-re0 chassisd[12941]: %DAEMON-5- CHASSISD_SNMP_TRAP6: SNMP trap generated: Temperature back to normal (jnxContentsContainerIndex 9, jnxContentsL1Index 2, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr Routing Engine 1, jnxOperatingTemp 44)
*** RE1 messages *** Dec 13 04:41:27.407 PTX3K-re1 mgd[29972]: %INTERACT-6-UI_CMDLINE_READ_LINE: User '(authentication in progress)', command 'rpc command .set auth environment user root logname root host PTX3K-re0 agent mgd current-directory /var/tmp pid 98901 ppid 12941 ' Dec 13 04:41:27.407 PTX3K-re1 mgd[29972]: %INTERACT-6-UI_JUNOSCRIPT_CMD: User '(authentication in progress)' used JUNOScript client to run command 'request-authentication user=root logname=root host=PTX3K-re0 agent=mgd current-directory=/var/tmp pid=98901 ppid=12941' Dec 13 04:41:27.409 PTX3K-re1 mgd[29972]: %DAEMON-7: check_regex_add: 1588 regex_add = 0 Dec 13 04:41:27.409 PTX3K-re1 mgd[29972]: %INTERACT-6-UI_AUTH_EVENT: Authenticated user 'root' at permission level 'super-user' Dec 13 04:41:27.409 PTX3K-re1 mgd[29972]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' master[29972], ssh-connection '', client-mode 'junoscript' Dec 13 04:41:27.713 PTX3K-re1 mgd[29972]: %INTERACT-6-UI_JUNOSCRIPT_CMD: User 'root' used JUNOScript client to run command 'request-end-session' Dec 13 04:41:27.713 PTX3K-re1 mgd[29972]: %INTERACT-6-UI_LOGOUT_EVENT: User 'root' logout Dec 13 04:41:32.325 PTX3K-re1 mgd[29973]: %INTERACT-6-UI_CMDLINE_READ_LINE: User '(authentication in progress)', command 'rpc command .set auth environment user root logname root host PTX3K-re0 agent mgd current-directory /var/tmp pid 98902 ppid 12941 ' Dec 13 04:41:32.326 PTX3K-re1 mgd[29973]: %INTERACT-6-UI_JUNOSCRIPT_CMD: User '(authentication in progress)' used JUNOScript client to run command 'request-authentication user=root logname=root host=PTX3K-re0 agent=mgd current-directory=/var/tmp pid=98902 ppid=12941' Dec 13 04:41:32.327 PTX3K-re1 mgd[29973]: %DAEMON-7: check_regex_add: 1588 regex_add = 0 Dec 13 04:41:32.328 PTX3K-re1 mgd[29973]: %INTERACT-6-UI_AUTH_EVENT: Authenticated user 'root' at permission level 'super-user' Dec 13 04:41:32.328 PTX3K-re1 mgd[29973]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' master[29973], ssh-connection '', client-mode 'junoscript' Dec 13 04:41:32.632 PTX3K-re1 mgd[29973]: %INTERACT-6-UI_JUNOSCRIPT_CMD: User 'root' used JUNOScript client to run command 'request-end-session' Dec 13 04:41:32.632 PTX3K-re1 mgd[29973]: %INTERACT-6-UI_LOGOUT_EVENT: User 'root' logout Dec 13 04:41:38.029 PTX3K-re1 mgd[29975]: %INTERACT-6-UI_CMDLINE_READ_LINE: User '(authentication in progress)', command 'rpc command .set auth environment user root logname root host PTX3K-re0 agent mgd current-directory /var/tmp pid 98907 ppid 12941 ' Dec 13 04:41:38.029 PTX3K-re1 mgd[29975]: %INTERACT-6-UI_JUNOSCRIPT_CMD: User '(authentication in progress)' used JUNOScript client to run command 'request-authentication user=root logname=root host=PTX3K-re0 agent=mgd current-directory=/var/tmp pid=98907 ppid=12941' Dec 13 04:41:38.030 PTX3K-re1 mgd[29975]: %DAEMON-7: check_regex_add: 1588 regex_add = 0 Dec 13 04:41:38.031 PTX3K-re1 mgd[29975]: %INTERACT-6-UI_AUTH_EVENT: Authenticated user 'root' at permission level 'super-user' Dec 13 04:41:38.031 PTX3K-re1 mgd[29975]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' master[29975], ssh-connection '', client-mode 'junoscript' Dec 13 04:41:38.336 PTX3K-re1 mgd[29975]: %INTERACT-6-UI_JUNOSCRIPT_CMD: User 'root' used JUNOScript client to run command 'request-end-session' Dec 13 04:41:38.336 PTX3K-re1 mgd[29975]: %INTERACT-6-UI_LOGOUT_EVENT: User 'root' logout
User@PTX3K-re0> show chassis hardware Hardware inventory: Item Version Part number Serial number Description Chassis JN1218DE1AJC PTX3000 Midplane REV 17 750-044645 ACAL0001 Backplane FPM REV 07 760-044663 ACDK9595 Front Panel Display PSM 0 REV 04 740-044980 1EDJ3420182 DC 12V Power Supply PSM 1 REV 04 740-044980 1EDJ3420193 DC 12V Power Supply PSM 2 REV 04 740-044980 1EDJ3420196 DC 12V Power Supply Routing Engine 0 BUILTIN BUILTIN RE-PTX-2X00x6 Routing Engine 1 BUILTIN BUILTIN RE-PTX-2X00x6 CB 0 REV 22 750-053916 ACPR6926 RCB P2 CB 1 REV 22 750-053916 ACPT2975 RCB P2 RCB-CC 0 REV 05 750-060320 ACPR3312 PTX3K CMPN RCB RCB-CC 1 REV 05 750-060320 ACPR3290 PTX3K CMPN RCB
The following event-option configuration triggers the outputs under /var/log when over temperature is seen.
/var/log
set event-options policy temp_issue1 events chassisd_re_over_temp_warning set event-options policy temp_issue1 within 1 trigger on set event-options policy temp_issue1 within 1 trigger 1 set event-options policy temp_issue1 then execute-commands commands "show chassis environment" set event-options policy temp_issue1 then execute-commands commands "show system processes extensive" set event-options policy temp_issue1 then execute-commands commands "show chassis fan" set event-options policy temp_issue1 then execute-commands commands "show system uptime" set event-options policy temp_issue1 then execute-commands output-filename temp_issue_fail set event-options policy temp_issue1 then execute-commands destination local set event-options policy temp_issue1 then execute-commands output-format text set event-options destinations local archive-sites /var/log/
###### temp_issue_fail.txt ######
The backup RE 1 is 96 degree C.
User@PTX3K-re0> show chassis environment Class Item Status Measurement Temp PSM 0 OK 33 degrees C / 91 degrees F PSM 1 OK 27 degrees C / 80 degrees F PSM 2 OK 30 degrees C / 86 degrees F PSM 3 Absent PSM 4 Absent Routing Engine 0 OK 49 degrees C / 120 degrees F Routing Engine 0 CPU OK 48 degrees C / 118 degrees F Routing Engine 1 OK 96 degrees C / 204 degrees F Routing Engine 1 CPU OK 43 degrees C / 109 degrees F
Class Item Status Measurement Temp PSM 0 OK 33 degrees C / 91 degrees F PSM 1 OK 27 degrees C / 80 degrees F PSM 2 OK 30 degrees C / 86 degrees F PSM 3 Absent PSM 4 Absent Routing Engine 0 OK 49 degrees C / 120 degrees F Routing Engine 0 CPU OK 48 degrees C / 118 degrees F Routing Engine 1 OK 96 degrees C / 204 degrees F Routing Engine 1 CPU OK 43 degrees C / 109 degrees F
The primary RE 0 CPU usage is not so high.
User@PTX3K-re0> show system processes extensive PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 64K CPU0 0 560.0H 100.00% idle{idle: cpu0} 11 root 155 ki31 0K 64K CPU3 3 514.6H 100.00% idle{idle: cpu3} 11 root 155 ki31 0K 64K RUN 2 516.3H 97.56% idle{idle: cpu2} 11 root 155 ki31 0K 64K RUN 1 519.0H 97.27% idle{idle: cpu1} 6013 root 33 0 855M 99196K CPU1 1 45.8H 9.28% chassisd{CHASSISD } 6024 root 20 0 6417M 5460M kqread 2 267.9H 0.10% rpd{RPD } 0 root -16 - 0K 400K swapin 3 28.0H 0.00% kernel{SWAPPER }
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 64K CPU0 0 560.0H 100.00% idle{idle: cpu0} 11 root 155 ki31 0K 64K CPU3 3 514.6H 100.00% idle{idle: cpu3} 11 root 155 ki31 0K 64K RUN 2 516.3H 97.56% idle{idle: cpu2} 11 root 155 ki31 0K 64K RUN 1 519.0H 97.27% idle{idle: cpu1} 6013 root 33 0 855M 99196K CPU1 1 45.8H 9.28% chassisd{CHASSISD } 6024 root 20 0 6417M 5460M kqread 2 267.9H 0.10% rpd{RPD } 0 root -16 - 0K 400K swapin 3 28.0H 0.00% kernel{SWAPPER }
Fans and impellers in zone 1 are now running at full speed.
User@PTX3K-re0> show chassis fan Item Status % RPM Measurement Fan Tray 0 Fan 1 OK 44% 7080 RPM Fan Tray 0 Fan 2 OK 45% 7200 RPM Fan Tray 0 Fan 3 OK 44% 7080 RPM Fan Tray 0 Fan 4 OK 44% 7080 RPM Fan Tray 0 Fan 5 OK 44% 7080 RPM Fan Tray 0 Fan 6 OK 43% 6960 RPM Fan Tray 0 Fan 7 OK 43% 6840 RPM Fan Tray 0 Fan 8 OK 50% 8040 RPM Fan Tray 0 Fan 9 OK 50% 8040 RPM Fan Tray 0 Fan 10 OK 50% 8040 RPM Fan Tray 0 Fan 11 OK 50% 7920 RPM Fan Tray 0 Fan 12 OK 50% 8040 RPM Fan Tray 0 Fan 13 OK 52% 8280 RPM Fan Tray 0 Fan 14 OK 50% 8040 RPM Fan Tray 1 Fan 1 OK 70% 11160 RPM Fan Tray 1 Fan 2 OK 68% 10920 RPM Fan Tray 1 Fan 3 OK 68% 10920 RPM Fan Tray 1 Fan 4 OK 68% 10920 RPM Fan Tray 1 Fan 5 OK 68% 10920 RPM Fan Tray 1 Fan 6 OK 68% 10920 RPM Fan Tray 1 Fan 7 OK 68% 10920 RPM Fan Tray 1 Fan 8 OK 75% 11880 RPM Fan Tray 1 Fan 9 OK 75% 11880 RPM Fan Tray 1 Fan 10 OK 75% 11880 RPM Fan Tray 1 Fan 11 OK 75% 12000 RPM Fan Tray 1 Fan 12 OK 75% 12000 RPM Fan Tray 1 Fan 13 OK 76% 12120 RPM Fan Tray 1 Fan 14 OK 76% 12120 RPM
Item Status % RPM Measurement Fan Tray 0 Fan 1 OK 44% 7080 RPM Fan Tray 0 Fan 2 OK 45% 7200 RPM Fan Tray 0 Fan 3 OK 44% 7080 RPM Fan Tray 0 Fan 4 OK 44% 7080 RPM Fan Tray 0 Fan 5 OK 44% 7080 RPM Fan Tray 0 Fan 6 OK 43% 6960 RPM Fan Tray 0 Fan 7 OK 43% 6840 RPM Fan Tray 0 Fan 8 OK 50% 8040 RPM Fan Tray 0 Fan 9 OK 50% 8040 RPM Fan Tray 0 Fan 10 OK 50% 8040 RPM Fan Tray 0 Fan 11 OK 50% 7920 RPM Fan Tray 0 Fan 12 OK 50% 8040 RPM Fan Tray 0 Fan 13 OK 52% 8280 RPM Fan Tray 0 Fan 14 OK 50% 8040 RPM Fan Tray 1 Fan 1 OK 70% 11160 RPM Fan Tray 1 Fan 2 OK 68% 10920 RPM Fan Tray 1 Fan 3 OK 68% 10920 RPM Fan Tray 1 Fan 4 OK 68% 10920 RPM Fan Tray 1 Fan 5 OK 68% 10920 RPM Fan Tray 1 Fan 6 OK 68% 10920 RPM Fan Tray 1 Fan 7 OK 68% 10920 RPM Fan Tray 1 Fan 8 OK 75% 11880 RPM Fan Tray 1 Fan 9 OK 75% 11880 RPM Fan Tray 1 Fan 10 OK 75% 11880 RPM Fan Tray 1 Fan 11 OK 75% 12000 RPM Fan Tray 1 Fan 12 OK 75% 12000 RPM Fan Tray 1 Fan 13 OK 76% 12120 RPM Fan Tray 1 Fan 14 OK 76% 12120 RPM
This symptom has been observed with VMhost based RE-PTX-2X00x6 RCB in Junos OS release 17.3R3 and 17.4R2, and has not been observed with legacy RE-DUO-2600.
The backup temperature spike is related to JUNOScript.
Customers can safely ignore the syslog message and the SNMP trap because the backup RE high temperature does not cause any service impact. The backup RE temperature comes back to normal immediately by the use of full speed fans before the chassis is shut down.
2021-03-25: Updated the article terminology to align with Juniper's Inclusion & Diversity initiatives