Description

This article explains the behavior when "HMC::FATAL ERROR!! due to PFE4-HMC1: eri timeout error" appears.

Symptoms

There seems to be an issue with hybrid memory cube (HMC) and hence, errors are detected on PFE.
Even though the HMC error is an auto corrected transient hardware error, it may cause wedge condition sometimes.
This will result in a disabled PFE and shut down the interface on the affected PFE to avoid the impact being spread across other PFEs.

May 10 07:04:27.781 2024  fpc0 hmc_eri_config_access, HMC 57, read eri 10 timeout error 
May 10 07:04:27.781 2024  fpc0 micron_hmc_get_runtime_info, hmc 57 eri 10 error 
May 10 07:04:27.781 2024  fpc0 hmc_get_runtime_info vector NULL or failure return from vector 
May 10 07:04:27.781 2024  fpc0 cmsngfpc_temp_check, HMC 57 PFE4-HMC1-DIE run time info read error 
 
PFE Disable logs :-

May 10 07:04:27.781 2024  fpc0 ISSUING ifd-down for et-0/0/10 (449) on pfe-4 
May 10 07:04:27.781 2024  fpc0 ISSUING ifd-down for et-0/0/11 (450) on pfe-4 
May 10 07:04:27.781 2024  fpc0 ISSUING ifd-down for et-0/0/12 (451) on pfe-4 
May 10 07:04:27.781 2024  fpc0 ISSUING ifd-down for et-0/0/13 (452) on pfe-4 
May 10 07:04:27.781 2024  fpc0 ISSUING ifd-down for et-0/0/14 (453) on pfe-4 
 

May 10 07:04:27.781 2024  fpc0 pechip_set_pfe_disabled:976: PECHIP[4]: PFE Marked Disabled !!! 
May 10 07:04:27.796 2024  fpc0 pechip_set_pfe_disabled:983: PECHIP[4]: Deregisterd TOE PIO Vectors !!! 
May 10 07:04:27.796 2024  fpc0 JPRDS_FDB:ERR:jprds_fdb_set_pfe_disabled(),8400: pfe 4 marked disabled  
May 10 07:04:27.796 2024  fpc0 Cmerror Op Set: Generic HMC::FATAL ERROR!! from PFE4-HMC1: eri timeout error  
May 10 07:05:10.809 2024  fpc0 Cmerror Op Clear: PE Chip: pe4[4]: IQM: clear pkt_timeout error  
May 10 07:05:20.436 2024  fpc0 Cmerror Op Clear: PE Chip: pe4[4]: HOSTSYS HIF Clear Write NACK for Aperture 0  
May 10 07:12:18.458 2024  fpc0 hmc_eri_config_access, HMC 57, read eri 10 timeout error 
May 10 07:12:18.530 2024  fpc0 micron_hmc_get_runtime_info, hmc 57 eri 10 error 
May 10 07:12:18.550 2024  fpc0 hmc_get_runtime_info vector NULL or failure return from vector 
May 10 07:12:18.551 2024  fpc0 cmsngfpc_temp_check, HMC 57 PFE4-HMC1-DIE run time info read error 
 
May 10 07:12:18.751 2024  fpc0 pechip_set_pfe_disabled:976: PECHIP[4]: PFE Marked Disabled !!! 
May 10 07:12:18.751 2024  fpc0 pechip_set_pfe_disabled:983: PECHIP[4]: Deregisterd TOE PIO Vectors !!! 
May 10 07:12:18.751 2024  fpc0 JPRDS_FDB:ERR:jprds_fdb_set_pfe_disabled(),8400: pfe 4 marked disabled  
May 10 07:12:18.751 2024  fpc0 Cmerror Op Set: Generic HMC::FATAL ERROR!! from PFE4-HMC1: eri timeout error 

Solution

As a workaround, please reboot fpc to clear the alarm and bring up the disabled PFEs.
It will also clear the HMC error if it is not auto-corrected.

You can use "request chassis fpc slot 0 restart "command.

Modification History

Initial Draft : 28/05/2024