In Node Slicing or Standalone mx2010/mx2020 setup, MPC11E FPC logged the below error (this also applies to MX-Series MPC10E).
GNF1_RE0> show log messages | match cmerror Mar 13 17:36:47.119 GNF1_RE0 bsys_RE0 fpc1 cmerror_set_error: Module: ZT[7:0]-HBM-IO, pfe id:7, error_id:0x351032 reason:ZT[7:0]: inst 0 channel 1 TOP vector 3 error GNF1_RE0>
Below error log reported mpc11e card fpc1 pfe7 HBM IO memory error, and the error_id:0x351032 is HBM_IO_SGL_BIT_ECC_ERR_CHNL1 (HBM IO single bit ECC Error for channel: 1)
Mar 13 17:36:47.116 BSYS_RE0 fpc1 cmerror_set_error:Module: ZT[7:0]-HBM-IO, pfe id:7, error_id:0x351032reason:ZT[7:0]: inst 0 channel 1 TOP vector 3 error
1. First, get the cmerror module # for ZT[7:0]-HBM-IO for the problem fpc (Check this in the GNF if it is node slicing setup)
GNF1_RE0> request pfe execute command "show cmerror module brief" target fpc1 SENT: Ukern command: show cmerror module brief ------------------------------------------------------------------------- Module Name Active Errors PFE Callback ModuleData Specific Function ------------------------------------------------------------------------- 1 fab-pfe@0 0 Yes 0x08089500 0x00000000 2 GE Switch 0 No 0x08089500 0x00000000 3 ZT[0:0] 0 No 0x08089500 0x00000000 4 ZT[1:0] 0 No 0x08089500 0x00000000 5 ZT[2:0] 0 No 0x08089500 0x00000000 6 ZT[3:0] 0 No 0x08089500 0x00000000 7 ZT[4:0] 0 No 0x08089500 0x00000000 8 ZT[5:0] 0 No 0x08089500 0x00000000 9 ZT[6:0] 0 No 0x08089500 0x00000000 10 ZT[7:0] 0 No 0x08089500 0x00000000 11 ZT[5:0]-HBM-IO 0 Yes 0x08089500 0x00000000 12 ZT[7:0]-HBM-IO 0 Yes 0x08089500 0x00000000.<--cmerror module# 12 is for the ZT[7:0]-HBM-IO module errors 13 ZT[1:0]-HBM-IO 0 Yes 0x08089500 0x00000000 14 ZT[2:0]-HBM-IO 0 Yes 0x08089500 0x00000000 15 ZT[0:0]-HBM-IO 0 Yes 0x08089500 0x00000000 16 ZT[4:0]-HBM-IO 0 Yes 0x08089500 0x00000000 17 ZT[6:0]-HBM-IO 0 Yes 0x08089500 0x00000000 18 ZT[3:0]-HBM-IO 0 Yes 0x08089500 0x00000000 ...
2. Next, check module# 12 for the error_id:0x351032 for the problem fpc1 pfe7. Then we know this error_id:0x351032 is for HBM_IO_SGL_BIT_ECC_ERR_CHNL1
GNF1_RE0> request pfe execute command "show cmerror module 12 error 0x351032 pfe 7" target fpc1 SENT: Ukern command: show cmerror module 12 error 0x351032 pfe 7 Error-id : 0x351032 Error Name : HBM_IO_SGL_BIT_ECC_ERR_CHNL1 Identifier : /fpc/1/platformd/0/cm/0/hbm/7/HBM_IO_SGL_BIT_ECC_ERR_CHNL1 Description : HBM IO single bit ECC Err channel: 1 State : enabled Scope : pfe Category : memory PFE : 7 Configured Level : Minor Default Level : Minor Count : 2 Threshold : 100 Error Limit : 0 Occur Count : 2 Clear Count : 0 Last-occurred(ms ago) : 174979427
Logs:
---------------------------------------------------------- Index Time Sub-Err State Description ---------------------------------------------------------- GNF1_RE0>
Now we know this error log is for the fpc1 pfe7 memory HBM IO single bit ECC Err channel 1. The single bit ECC error on HBM IO block is a transient memory error in nature. The single BIT ECC error usually should not affect the services and it was automatically corrected when it is detected. For this error, the threshold is 100 and, in this example, you can see it only occurred (count) 2 such errors, so no action is needed for a small number of such error counter.
If the error counter is big or repeats quickly in a short while, we can take action to reboot the fpc in a safe MW. Normally for such a transient H/W issue rebooting should resolve the issue. If the error still occurs after rebooting, then contact your JTAC Representative for further investigation.
2023-04-07: Initial publication