HMC Multi-bit uncorrectable error (MUE) analysis and recommendations - ERRSTATE=0x1F or ERRSTATE=31.
This article describes the issue and step to auto recover, without manual intervention whenever this type of error is reported.
HMC Multi-bit uncorrectable error (MUE) analysis and recommendations - ERRSTATE=0x1F or ERRSTATE=31 reported on a QFX10008 and caused an outage.
Error state 0x001F means MUE error was detected in the DRAM data in read response packet data. DRAM inside HMC is ECC-protected. ECC logic can correct single-bit DRAM errors automatically. However, it cannot correct multi-bit errors in a word, and occurrence of these errors are indicated in ERRSTAT register as MUE. Usually FPC reset is suggested to recover HMC errors. If reset doesn’t help it indicates HMC part issue and replacement is suggested. Unlike HMC ERRSTAT=0x7F that are classified as fatal, for which FPCs are configured to reset upon encountering this error, there is no such possibility for HMC error 0x1F as it's not classified as fatal. fpc 0 { error { fatal { action reset; } } } Auto-recovery: Either the FPC/Line card remains off or it resets itself to recover from the error condition avoiding traffic blackhole. Classify 0x1F error as a Fatal error. So that fatal error reset configuration can kick in and reset the card. Configuration Steps to classify 0x1F error from Major to Fatal: 0x1F Error maps to PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR. This error is categorized as MAJOR error. You can change the severity of the error from MAJOR to FATAL error using configuration. Classify ‘0x1F’ error as fatal error on HMC revision version 2.2 and 2.2A via cli by using the following: · Check the FPC models if they are QFX10000-30C or QFX10000-36Q. hostname> show chassis hardware clei-models | match fpc FPC 0 REV 41 xxx-xxxxxx xxxxxxxxxx QFX10000-30C FPC 1 REV 13 xxx-xxxxxx xxxxxxxxxx QFX10000-36Q FPC 4 REV 41 xxx-xxxxxx xxxxxxxxxx QFX10000-30C FPC 5 REV 13 xxx-xxxxxx xxxxxxxxxx QFX10000-36Q
· On QFX10000-30C FPC model with 6 PE chips the CMERR module is module 27.
root@hostname:RE:0% cprod -A fpc4 -c "show cmerr mod 27" | grep PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 0 Major 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 1 Major 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 2 Major 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 3 Major 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 4 Major 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 5 Major 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
Commands:
set chassis fpc 4 error /fpc/4/pfe/0/cm/0/PE_Chip/0/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 4 error /fpc/4/pfe/0/cm/0/PE_Chip/1/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 4 error /fpc/4/pfe/0/cm/0/PE_Chip/2/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 4 error /fpc/4/pfe/0/cm/0/PE_Chip/3/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 4 error /fpc/4/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 4 error /fpc/4/pfe/0/cm/0/PE_Chip/5/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 4 error scope pfe category memory fatal action reset
0x210087 0 Fatal* 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 1 Fatal* 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 2 Fatal* 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 3 Fatal* 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 4 Fatal* 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
0x210087 5 Fatal* 1 0 0 0 0 PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
· On QFX10000-36Q FPC model with 3 PE chips the CMERR module is module 15.
root@hostname:RE:0% cprod -A fpc5 -c "show cmerr mod 15" | grep PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR
root@hostname:RE:0% vty fpc5
set chassis fpc 5 error /fpc/5/pfe/0/cm/0/PE_Chip/0/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 5 error /fpc/5/pfe/0/cm/0/PE_Chip/1/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 5 error /fpc/5/pfe/0/cm/0/PE_Chip/2/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 5 error scope pfe category memory fatal action reset
Note:
To configure for all line cards regardless of the PE chip counts do the following:
set chassis fpc 5 error /fpc/5/pfe/0/cm/0/PE_Chip/3/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 5 error /fpc/5/pfe/0/cm/0/PE_Chip/4/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal
set chassis fpc 5 error /fpc/5/pfe/0/cm/0/PE_Chip/5/PECHIP_CMERROR_HMCIF_LINK0_INT_REG_HMC_ERR_MAJOR severity fatal