This article is related to the 'microcode failed to ack" logs which occurs when PFE go in a bad state after performing optics insertion/removal on a port on QFX5100-24Q-2P running Junos 14.1X53-D26.3 that subsequently caused issues on the box. 

This triggered PFE Manager CPU spike and eventually leading to microcode failure

%PFE-3: fpc0 84328: microcode failed to ack: u=0 p=104 1.c841=4280 1.c843=8804 1.ca18=0008 checks=10000

On the QFX5100 line of switches, inserting or removing optics on a port might cause a Packet Forwarding Engine Manager CPU spike and an eventual microcode failure.

We can see the error log "microcode failed to ack" in log messages when the issue is hit.

Dec 16 21:17:57 Lab-Device@host : %PFE-3: fpc0 0:_phy_84328_intf_update:

Dec 16 21:17:57 Lab-Device@host : %PFE-3: fpc0 84328: microcode failed to ack: u=0 p=104 1.c841=4280 1.c843=8804 1.ca18=0008 checks=10000

Dec 16 21:18:06 Lab-Device@host : %PFE-3: fpc0 CMQFX: PIC 2 is removed!!!

Dec 16 21:18:09 Lab-Device@host : %PFE-3: fpc0 fxpc_opus_qic_en_reset: POWER not GOOD for QIC slot: 1

Dec 16 21:18:10 Lab-Device@host : %PFE-3: fpc0 PIC: 2 S/W Init...

Dec 16 21:18:11 Lab-Device@host : %PFE-3: fpc0 fxpc_opus_sfp_qsfp_reset: Failed to get the reset SFP/QSFP for pic 2

Dec 16 21:18:43 Lab-Device@host : %PFE-3: fpc0 CMQFX: PIC 2 is removed!!!

This issue might be seen if the following conditions are met:
* On QFX5000 Series platforms running on Junos 14.1X53-D26.3 or lower versions
* If performing optics insertion/removal on one port

PFE reboot re-initializes the ports and that will fix the issue. This issue is getting resolved in Junos 14.1X53-D49 and higher. 

For more details regarding this issue, please refer to the below PR link:

https://prsearch.juniper.net/problemreport/PR1372041

2023-12-29 : Initial Version