There was power issue seen on ptx5000.
There are 2x PDU's and each PDU has 6x PSM.
PDU 1 failed on the router
root@> show chassis alarms no-forwarding 11 alarms currently active Alarm time Class Description 2023-09-11 21:31:33 PDT Major Insufficient Power - FRU(s) went offline 2023-09-11 21:20:52 PDT Major PDU 1 PSM 6 Not OK 2023-09-11 21:20:52 PDT Major PDU 1 PSM 5 Not OK 2023-09-11 21:20:52 PDT Major PDU 1 PSM 4 Not OK 2023-09-11 21:20:52 PDT Major PDU 1 PSM 3 Not OK 2023-09-11 21:20:52 PDT Major PDU 1 PSM 2 Not OK 2023-09-11 21:20:52 PDT Major PDU 1 PSM 1 Not OK 2023-09-11 21:20:52 PDT Major PDU 1 PSM 0 Not OK 2023-09-11 21:20:52 PDT Major PDU 1 Not OK 2023-09-11 21:20:44 PDT Major PDU 1 Converter Failed 2023-05-22 11:57:13 PDT Minor No Redundant Power for System
>> Below are the initial logs in chassisd logs. This is being reported by multiple PEMs.
Sep 11 21:20:13 psm_sng_calculate_input_usage: fru PSM 0 invalid input 1 Sep 11 21:20:13 psm_sng_jvision_property_get: fru PSM 0 calc input2 usage failed Sep 11 21:20:13 psm_sng_calculate_input_usage: fru PSM 1 invalid input 1 Sep 11 21:20:13 psm_sng_jvision_property_get: fru PSM 1 calc input2 usage failed
>> After some time, PDU failure logs were seen
Sep 11 21:20:43 psm_sng_calculate_input_usage: fru PSM 0 invalid input 1 Sep 11 21:20:43 psm_sng_jvision_property_get: fru PSM 0 calc input2 usage failed Sep 11 21:20:43 psm_sng_calculate_input_usage: fru PSM 1 invalid input 1 Sep 11 21:20:43 psm_sng_jvision_property_get: fru PSM 1 calc input2 usage failed Sep 11 21:20:43 psm_sng_calculate_input_usage: fru PSM 2 invalid input 1 Sep 11 21:20:43 psm_sng_jvision_property_get: fru PSM 2 calc input2 usage failed Sep 11 21:20:43 psm_sng_calculate_input_usage: fru PSM 3 invalid input 1 Sep 11 21:20:43 psm_sng_jvision_property_get: fru PSM 3 calc input2 usage failed Sep 11 21:20:43 psm_sng_calculate_input_usage: fru PSM 4 invalid input 1 Sep 11 21:20:43 psm_sng_jvision_property_get: fru PSM 4 calc input2 usage failed Sep 11 21:20:43 psm_sng_calculate_input_usage: fru PSM 5 invalid input 1 Sep 11 21:20:43 psm_sng_jvision_property_get: fru PSM 5 calc input2 usage failed Sep 11 21:20:43 psm_sng_calculate_input_usage: fru PSM 6 invalid input 1 Sep 11 21:20:43 psm_sng_jvision_property_get: fru PSM 6 calc input2 usage failed Sep 11 21:20:44 pdu_teq_read_36v_conv_ph_status:PDU 1 Conv Failed Sep 11 21:20:44 send: red alarm set, device PDU 1, reason PDU 1 Converter Failed
>> First PDU 1 and later All the PEM under PDU 1 reported major alarm
Sep 11 21:20:44.842 alarmd[8260]: Alarm set: PDU id=134218118, color=RED, class=CHASSIS, reason=PDU 1 Converter Failed Sep 11 21:20:44.842 craftd[7924]: Major alarm set, PDU 1 Converter Failed Sep 11 21:20:52.266 alarmd[8260]: Alarm set: PDU id=33554822, color=RED, class=CHASSIS, reason=PDU 1 Not OK Sep 11 21:20:52.266 craftd[7924]: Major alarm set, PDU 1 Not OK Sep 11 21:20:52.277 chassisd[7914]: CHASSISD_PDU_NOT_OK: status failure for PDU 1 (status bits: 0x01) Sep 11 21:20:52.328 alarmd[8260]: Alarm set: PSM id=33554823, color=RED, class=CHASSIS, reason=PDU 1 PSM 0 Not OK Sep 11 21:20:52.350 chassisd[7914]: CHASSISD_PSM_NOT_OK: status failure for PDU 1 PSM 0 (status bits: 0x5F1E) Sep 11 21:20:52.398 alarmd[8260]: Alarm set: PSM id=34603399, color=RED, class=CHASSIS, reason=PDU 1 PSM 1 Not OK Sep 11 21:20:52.420 chassisd[7914]: CHASSISD_PSM_NOT_OK: status failure for PDU 1 PSM 1 (status bits: 0x5F1E) Sep 11 21:20:52.469 alarmd[8260]: Alarm set: PSM id=35651975, color=RED, class=CHASSIS, reason=PDU 1 PSM 2 Not OK Sep 11 21:20:52.490 chassisd[7914]: CHASSISD_PSM_NOT_OK: status failure for PDU 1 PSM 2 (status bits: 0x5F1E) Sep 11 21:20:52.546 alarmd[8260]: Alarm set: PSM id=36700551, color=RED, class=CHASSIS, reason=PDU 1 PSM 3 Not OK Sep 11 21:20:52.568 chassisd[7914]: CHASSISD_PSM_NOT_OK: status failure for PDU 1 PSM 3 (status bits: 0x5F1E) Sep 11 21:20:52.617 chassisd[7914]: CHASSISD_PSM_NOT_OK: status failure for PDU 1 PSM 4 (status bits: 0x5F1E) Sep 11 21:20:52.617 alarmd[8260]: Alarm set: PSM id=37749127, color=RED, class=CHASSIS, reason=PDU 1 PSM 4 Not OK Sep 11 21:20:52.667 chassisd[7914]: CHASSISD_PSM_NOT_OK: status failure for PDU 1 PSM 5 (status bits: 0x5F1E) Sep 11 21:20:52.667 alarmd[8260]: Alarm set: PSM id=38797703, color=RED, class=CHASSIS, reason=PDU 1 PSM 5 Not OK Sep 11 21:20:52.716 chassisd[7914]: CHASSISD_PSM_NOT_OK: status failure for PDU 1 PSM 6 (status bits: 0x5F1E) Sep 11 21:20:52.716 alarmd[8260]: Alarm set: PSM id=39846279, color=RED, class=CHASSIS, reason=PDU 1 PSM 6 Not OK Sep 11 21:20:52.716 chassisd[7914]: chassisd SIGCHLD handler: pid=-1 Sep 11 21:20:53.296 craftd[7924]: Major alarm set, PDU 1 PSM 0 Not OK Sep 11 21:20:54.329 craftd[7924]: Major alarm set, PDU 1 PSM 1 Not OK Sep 11 21:20:55.365 craftd[7924]: Major alarm set, PDU 1 PSM 2 Not OK
>> As all the PEMs on PDU 1 went down, we could see Insufficient power alarm for the chassis, because of which FPC 6 and FPC 7 went offline
Sep 11 21:20:55.718 alarmd[8260]: Alarm set: Power Zone id=83886984, color=RED, class=CHASSIS, reason=Insufficient Power - FRU(s) went offline Sep 11 21:20:55.718 chassisd[7914]: CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 7 offline: No power
Slot 6 information: State Offline Reason No power Slot 7 information: State Offline Reason No power