This article explains about the Error logs "fpc* hmc_reg_rd HMC * pio error * " observed and the workaround for the same.

1) Below "FPC Major Errors" or "FPC Minor Errors" alarm is observed on the box:

user@TEXT_MX_RE> show chassis alarms

1 alarms currently active

Alarm time Class Description

2024-12-26 05:12:05 UTC Major FPC 0 Major Errors

2) Log messages record the below HMC Errors:

Dec 26 05:12:05 TEXT_MX_RE : %PFE-3: fpc0 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 1 group 5 addr 0x10 prio 0 flags 0x15 failed status 0x1125757.077:  

Dec 26 05:12:05 TEXT_MX_RE : %PFE-3: fpc0 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 read reg 0x2b0004 size 4 failed 

Dec 26 05:12:05 TEXT_MX_RE : %PFE-3: fpc0 hmc_reg_rd HMC 68 pio error 1 

Dec 26 05:12:05 TEXT_MX_RE : %PFE-3: fpc0 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 1 group 5 addr 0x10 prio 0 flags 0x15 failed status 0x1125757.077:  

Dec 26 05:12:05 TEXT_MX_RE : %PFE-3: fpc0 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 read reg 0x2b0004 size 4 failed 

Dec 26 05:12:05 TEXT_MX_RE : %PFE-3: fpc0 hmc_reg_rd HMC 68 pio error 2 

Dec 26 05:12:06 TEXT_MX_RE : %PFE-3: fpc0 mpcs_i2c_smem_read: MPCS(0) ctlr 0 smem_idx 1 group 5 addr 0x10 prio 0 flags 0x15 failed status 0x1125757.077:  

Dec 26 05:12:06 TEXT_MX_RE : %PFE-3: fpc0 cmtfpc_hmc_iofunc HMC 0 0x45/0x10 read reg 0x2b0004 size 4 failed 

Dec 26 05:12:06 TEXT_MX_RE : %PFE-3: fpc0 hmc_reg_rd HMC 68 pio error 3 

Dec 26 05:12:06 TEXT_MX_RE : %PFE-6: fpc0 hmc_asic_cmerror_set_error EA0_HMC0 reports CM err HMC Fatal Err PIO, err code id 0 

Dec 26 05:12:06 TEXT_MX_RE : %PFE-3: fpc0 hmc_eri_config_access, HMC 68, read eri 10 ext_req read error, at 31 ms polling 0 

Dec 26 05:12:06 TEXT_MX_RE : %PFE-3: fpc0 micron_hmc_get_temp_info, eri 10 error 

These log messages points towards a fatal error on the Hybrid Memory Cube (HMC) module, and thus the said errors are detected on the PFE. The issue is due to a board specific and a generic internal fault, which should be a soft event.

Even though the HMC error is an auto corrected transient hardware error, it may cause wedge condition sometimes. This will result in a disabled PFE action and shut down the interface on the affected PFE to avoid the impact being spread across other PFEs.

Workaround:

As a workaround, please reboot the FPC in question to clear the alarm and bring up the disabled PFEs. It will also clear the HMC error if it is not auto corrected. Use the below command to clear the alarm:

request chassis fpc restart slot <slot-number>

If the device has a built in FPC [example: QFX10002-60c, QFX10002-36Q, QFX10002-72Q] then please reboot the device

If restart of FPC/Device did not resolve the issue, please contact JTAC for further debugging.