This article explains a scenario where too many power failures will lead to power shortage on the system (less than the required power budget) and lead to FRU disconnects. In this case, one of the FPC got rebooted due to power shortage.

1. Multiple PEMs (2 PEMs in this case) failing at the same time
2. Below log messages will be seen on the router indicating too many PEM failures and indication on router experiencing Power shortage

Nov 12 06:15:22 router-name craftd[30057]: %DAEMON-4: Minor alarm cleared, Power Budget:Chassis experiencing power shortage

Nov 12 06:37:50 router-name alarmd[30326]: %DAEMON-4: Alarm set: Power Supply id=65663, color=RED, class=CHASSIS, reason=Too many supplies missing

Nov 12 06:37:50 router-name craftd[30057]: %DAEMON-4: Receive FX craftd set alarm message: color: 1 class: 100 object: 127 slot: 0 silent: 0 short_reason=Supplies missing in long_reason=Too many supplies missing id=65663 reason=134217728

1. FRUs (FPC in this case) will get disconnected or rebooted due to this power shortage. For example one of the FPC in this case got disconnected due to power shortage

Nov 12 06:37:58 router-name : %PFE-3: fpc5 CMLC: Master closed connection (errno=0)

From the logs we see that 2 x PEMs failed at the same time multiple times. This has caused issues in total available power since only 2 PEMs were powered on when the other 2 PEMs failed.

Nov 12 06:15:22 router-name craftd[30057]: %DAEMON-4: Minor alarm cleared, Power Budget:Chassis experiencing power shortage

Nov 12 06:37:50 router-name alarmd[30326]: %DAEMON-4: Alarm set: Power Supply id=65663, color=RED, class=CHASSIS, reason=Too many supplies missing

Nov 12 06:37:50 router-name craftd[30057]: %DAEMON-4: Receive FX craftd set alarm message: color: 1 class: 100 object: 127 slot: 0 silent: 0 short_reason=Supplies missing in long_reason=Too many supplies missing id=65663 reason=134217728

Here's a detailed explanation on required power budget for the system.

Each PEM has a total output capacity of 2700 W.

When 2 PEMs failed, the system was left only with 5400 W available power. With all the FRUs in the router, below is the total power calculation that was needed to keep the FRUs powered on.

This case had below FRUs in the router:

1xRE, 1xCB, 6xLC, 6xSFB, 2xFTC (22 Fans)

Power required for 6xSFB   - (x6) = 1410 W

Power required for 2xFan   - (x2) = 950 W

Power required for 1xRE0   - (x1) = 350 W

Power required for 6xFPC   - 520 W (x6) = 3120 W

Total Power        = 5830 W

 Below Juniper external documentation that explains power requirement for each FRU type and total power budget calculation for the system.

https://www.juniper.net/documentation/us/en/hardware/ptx10008/topics/topic-map/ptx10008-power-specifications.html

With 2 PEMs only operational (and 2 going bad at some point), system was left with only 5400 W which is less than actual and redundant power budget of the system and thus FPC5 should have gone offline. So, looks like FPC5 was just a victim while the 2x PEMs failing is the actual culprit.