The customer observed their PTX10001 RE rebooted with catastrophic error.
Routing Engine status:
Slot 0:
[...]
Model RE-JNP10001-36MR
Serial ID BCEB5050
Start time 2024-03-05 02:46:00 UTC
Uptime 23 minutes, 12 seconds
Load averages: 1 minute 5 minute 15 minute
1.87 2.06 1.86
Last reboot reason catastrophic error/hw misc <<<<<<<<<
Firmware:
========
RE 0 BIOS ROM 7 28.01 28.01 OK <<<<< BIOS version 28.01
sh-4.3# cat /sys/module/intel_idle/parameters/max_cstate
9
sh-4.3# grep . /sys/devices/system/cpu/cpu*/cpuidle/state3/name
/sys/devices/system/cpu/cpu0/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu1/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu10/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu11/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu12/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu13/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu14/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu15/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu16/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu17/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu18/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu19/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu2/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu20/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu21/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu22/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu23/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu3/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu4/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu5/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu6/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu7/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu8/cpuidle/state3/name:C6-SKX
/sys/devices/system/cpu/cpu9/cpuidle/state3/name:C6-SKX
sh-4.3#
sh-4.3# cat /sys/devices/system/cpu/cpu0/cpuidle/state3/usage
21097775
21098591
21099352
21100242
21101056
21101789
21102525
21103613
21105244
21106618
21107465
21108222
21109034
21110318
21110926
21111757
21112430
Mar 5 02:47:06 nzakl-orb-p-1 rpd[13810]: RPD_ISIS_NO_ROUTERID: IS-IS instance does not have a valid router ID
Mar 5 02:47:08 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruPowerOn for /Chassis[0]/Cb[0]
Mar 5 02:47:08 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruInsertion for /Chassis[0]/Cb[0]
Mar 5 02:47:08 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruInsertion for /Chassis[0]/Psm[1]
Mar 5 02:47:08 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruOnline for /Chassis[0]/Fan[5]
Mar 5 02:47:08 nzakl-orb-p-1 hwdre: HWD_FRU_ONLINE_NOTICE: FRU online fan5
Mar 5 02:47:08 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruInsertion for /Chassis[0]/Fan[5]
Mar 5 02:47:09 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruOnline for /Chassis[0]/Fan[3]
Mar 5 02:47:09 nzakl-orb-p-1 hwdre: HWD_FRU_ONLINE_NOTICE: FRU online fan3
Mar 5 02:47:09 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruInsertion for /Chassis[0]/Fan[3]
Mar 5 02:47:09 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruOnline for /Chassis[0]/Fan[1]
Mar 5 02:47:09 nzakl-orb-p-1 hwdre: HWD_FRU_ONLINE_NOTICE: FRU online fan1
Mar 5 02:47:09 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruInsertion for /Chassis[0]/Fan[1]
Mar 5 02:47:11 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruOnline for /Chassis[0]/Cb[0]
Mar 5 02:47:11 nzakl-orb-p-1 hwdre: HWD_FRU_ONLINE_NOTICE: FRU online cb0
Mar 5 02:47:11 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruPowerOn for /Chassis[0]/Sib[0]
Mar 5 02:47:11 nzakl-orb-p-1 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruInsertion for /Chassis[0]/Sib[0]
Mar 5 02:47:12 nzakl-orb-p-1 hwdre: HWD_FRU_ONLINE_NOTICE: FRU online fpc0
Mar 5 02:47:16 nzakl-orb-p-1 hwdre: HWD_FRU_REBOOT_REASON_REG_NOTICE: reason reg0 byte_offset 0x208 = 0x84
Mar 5 02:47:16 nzakl-orb-p-1 last message repeated 2 times
Mar 5 02:47:16 nzakl-orb-p-1 hwdre: HWD_FRU_EACH_REBOOT_REASON_NOTICE: each_reason_string=catastrophic error/hw misc
Mar 5 02:47:16 nzakl-orb-p-1 hwdre: HWD_FRU_REBOOT_REASON_REG_NOTICE: reason reg0 byte_offset 0x208 = 0x80
Mar 5 02:47:16 nzakl-orb-p-1 last message repeated 4 times
Mar 5 02:47:16 nzakl-orb-p-1 hwdre: HWD_FRU_EACH_REBOOT_REASON_NOTICE: each_reason_string=software reboot
Mar 5 02:47:16 nzakl-orb-p-1 hwdre: HWD_FRU_REBOOT_REASON_REG_NOTICE: reason reg1 byte_offset 0x207 = 0x0
Mar 5 02:47:16 nzakl-orb-p-1 hwdre: HWD_FRU_REBOOT_REASON_REG_NOTICE: reason reg2 byte_offset 0x20a = 0x0
Mar 5 02:47:16 nzakl-orb-p-1 hwdre: HWD_FRU_REBOOT_REASON_NOTICE: reboot reason string = catastrophic error/hw misc
Based on the troubleshooting so far, I think the issue hit the known problem - PR/1793605.
Platform
PTX10001-36MR: PTX10002-36QDD <<<<< platform matched, reboot reason also matched
Dev-Introduced-RLI-Or-PR
Day-1
Root-Cause
Intel CPU specific issue, DAY-1. Power save c-state support thats enabled by default can cause Machine Check errors. This PR fix disables c-states to improve platform reliability.
The software fix had been submit and it is under review right now.
JTAC had provided two workaround methods:
1) Python script
2) Event script
PR/1793605