Description

The "number of corrected single-bit errors from HMC" message reports an excessive number of corrected memory errors in the EACHIP HMC section. 

Symptoms

Customer reported FPC encountered errors on EA chips based line card . 

show chassis alarms
2 alarms currently active
Alarm time Class Description
2024-03-19 06:47:23 UTC Major FPC 1 Hard errors
2024-02-26 08:10:08 UTC Major FPC 1 Major Errors - EA Error code: 0x2400c5

Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-1/1/0 199 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-1/1/1 200 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-1/1/2 201 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-1/0/0:0 151 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-1/0/0:1 152 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-1/0/0:2 153 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-1/0/0:3 154 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-1/0/1:0 155 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-1/0/1:1 156 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-1/0/1:2 157 
Feb 26 08:10:27   fpc1 PFE 0: 'PFE Disable' action performed. Bringing down ifd xe-1/0/1:3 158 
Feb 26 08:10:28   fpc1 Cmerror Op Set: EA[0:0]: EA[0:0]: HMCIF Rx: Link3: total number of corrected singlebit errors from HMC 1 exceeded threshold 32  
Feb 26 08:10:28   fpc1 Cmerror Op Set: EA[0:0]: EA[0:0]: HMCIF Rx: Link3: total number of corrected singlebit errors from HMC 1 exceeded threshold 32  
Feb 26 08:10:58   fpc1 Cmerror Op Set: EA[0:0]: EA[0:0]: HMCIF Rx: Link3: total number of corrected singlebit errors from HMC 1 exceeded threshold 32  
Feb 26 08:12:25   fpc1 Cmerror Op Set: EA[0:0]: EA[0:0]: HMCIF Rx: Link3: total number of corrected singlebit errors from HMC 1 exceeded threshold 32  
Feb 26 08:12:28   fpc1 Cmerror Op Set: EA[0:0]: EA[0:0]: HMCIF Rx: Link3: total number of corrected singlebit errors from HMC 1 exceeded threshold 32  
Feb 26 08:14:18   fpc1 Cmerror Op Set: EA[0:0]: EA[0:0]: HMCIF Rx: Link3: total number of corrected singlebit errors from HMC 1 exceeded threshold 32 




 

Solution

On MX platforms with EA Line cards, the error messages might be seen as follows when the single-bit errors event exceeds the threshold of 32. 
This is due to a transient hardware issue in the EACHIP Hybrid Memory Cube.  HMC Firmware logic does automatically repair this events and there is no operational impact. 

Perform these steps to determine the cause and resolve the problem (if any).  Continue through each step until the problem is resolved.
  1. Collect the show command output.
Capture the output to a file (in case you have to open a technical support case). To do this, configure each SSH client/terminal emulator to log your session.
show log messages
show log chassisd
start shell network pfe <fpc#>
show nvram
show syslog messages
show cmerror module brief 
show ea-asic <asic-number> hmc correct-sbe
  1. Analyze the show command output.
In the 'show log messages', review the events that occurred at or just before the appearance of the "number of corrected single-bit errors from HMC" message. Frequently, these events help identify the cause.
  • A single occurrence of the occurrence should be non-impacting and would clear automatically .
  • If there are multiple errors being reported continuously and PFE gets disabled due to the error then reboot of the FPC should clear the alarms.
  • If these errors keep reporting even after FPC reboot then RMA would be required.

Modification History

2024-03-22 : Article Created