Description

    For MX2000 Router with the AC PSM (MX2000-PSM-AC 740-045051 REV11 and below), there have been cases reported that the customer-site AC power source circuit breaker gets tripped, and multiple PSMs raise alarm of "Not OK", "Output Failure", "INPx Failure" and others. 

Symptoms

The alarms similar to the following may be raised:

> show chassis alarms
40 alarms currently active
Alarm time Class Description
2023-12-15 11:05:30 GMT Major PSM 10 Output 5V Fail
2023-12-15 11:05:30 GMT Major PSM 10 Output Failure
2023-12-15 11:04:50 GMT Minor N+1 Power redundancy missing in Upper zone
2023-12-15 11:04:50 GMT Major Wrong Input Feed Selection for PSM 16
2023-12-15 11:04:50 GMT Minor Redundant feed missing for PSM 16
2023-12-15 11:04:50 GMT Major Wrong Input Feed Selection for PSM 15
2023-12-15 11:04:50 GMT Minor Redundant feed missing for PSM 15
2023-12-15 11:04:50 GMT Major Input feed missing for PSM 14
2023-12-15 11:04:50 GMT Major Input feed missing for PSM 13
2023-12-15 11:04:50 GMT Major Input feed missing for PSM 12
2023-12-15 11:04:50 GMT Major PSM 17 Output 5V Fail
2023-12-15 11:04:50 GMT Major PSM 9 Not OK
2023-12-15 11:04:45 GMT Major PSM 17 Output Failure
2023-12-15 11:04:45 GMT Major PSM 17 INP0 Failure
2023-12-15 11:04:45 GMT Major PSM 17 Not OK
2023-12-15 11:04:45 GMT Major PSM 16 INP0 Failure
2023-12-15 11:04:45 GMT Major PSM 15 INP0 Failure
2023-12-15 11:04:45 GMT Major PSM 14 Output 5V Fail
2023-12-15 11:04:45 GMT Major PSM 14 Output Failure
2023-12-15 11:04:45 GMT Major PSM 14 INP1 Failure
2023-12-15 11:04:45 GMT Major PSM 14 INP0 Failure
2023-12-15 11:04:45 GMT Major PSM 13 Output 5V Fail
2023-12-15 11:04:45 GMT Major PSM 13 Output Failure
2023-12-15 11:04:45 GMT Major PSM 13 INP1 Failure
2023-12-15 11:04:45 GMT Major PSM 13 INP0 Failure
2023-12-15 11:04:45 GMT Major PSM 12 Output 5V Fail
2023-12-15 11:04:45 GMT Major PSM 12 Output Failure
2023-12-15 11:04:45 GMT Major PSM 12 INP1 Failure
2023-12-15 11:04:45 GMT Major PSM 12 INP0 Failure
2023-12-15 11:04:45 GMT Major PSM 11 INP1 Failure
2023-12-15 11:04:45 GMT Major PSM 9 Output 5V Fail
2023-12-15 11:04:45 GMT Major PSM 9 Output Failure
2023-12-15 11:04:30 GMT Major Wrong Input Feed Selection for PSM 9
2023-12-15 11:04:30 GMT Minor Redundant feed missing for PSM 9
2023-12-15 11:04:25 GMT Major PSM 11 Output 5V Fail
2023-12-15 11:04:25 GMT Major PSM 11 Output Failure
2023-12-15 11:04:25 GMT Major PSM 11 INP0 Failure
2023-12-15 11:04:25 GMT Major PSM 11 Not OK
2023-12-15 11:04:25 GMT Major PSM 10 Not OK
2023-12-15 11:04:25 GMT Major PSM 9 INP0 Failure

 

Solution

The immediate objective after this issue is observed is to contain the problem and is to limit the collateral consequences, by means of having all the PSMs that share the same AC power source replaced out. 

NOTE: Do not switch on the dedicated customer-site circuit breaker that tripped already, before following the procedure below.

Follow the procedure below:
1. Physically trace the AC power cords from the customer-site AC power source circuit breaker that is tripped, to MX2000 Power Distribution Module (PDM) and PSM.
    1.1. For an MX2000 Router with Three-Phase Wye/Delta AC PDM, one feed (e.g. PDM2/Input0 (Right)) maps to multiple PSMs (e.g. PSM slot 12, 13, 14, 15, 16, and 17). Refer to Mapping Input Power from AC Power Distribution Modules to AC Power Supply Modules on MX2000 Routers.
    1.2. For an MX2000 system with Single-Phase AC PDM, the internal wiring and the circuit breaker of the customer-site AC power source need to be checked.
2. Issue RMA for the PSMs that share the same AC power source.
3. Schedule a maintenance window (MW) with outage.
    3.1. For MX2020, either one side of chassis (top or bottom) may need an outage; MPCs in slot 10 through 19 (top), or MPCs in slot 0 through 9 (bottom).
    3.2. An outage MW may not be needed depending on how many PSMs to be replaced and the power budget. 
4. At the MW
    4.1. Remove all the PSMs that need to be replaced as identified through procedure 1. and 2., by turning the power switch to the off (O) position. Follow relevant part of procedure.
    4.2. Insert all the PSMs with the PSM switch in the off (O) position. Follow relevant part of procedure.
    4.3. Switch on the customer-site circuit breakers.
    4.4. Move the power switch on the PSMs to the (I) position.
 

Modification History

  • 09/15/2024: Add some small fix to the article
  • 01/30/2024: Remove some contents
  • 01/30/2024: Initial draft