Description

This article explains the meaning of the following syslog message and how to resolve it:

PQ3 DDR single bit error minor alarm​
Error Number: CMERROR_PQ3_DDR_1_BIT_MINOR (0x160001)
 
PQ3 DDR single bit error major alarm​
Error Number: CMERROR_PQ3_DDR_1_BIT_MAJOR (0x160002)

Symptoms

The errors that are logged on the FPC are single bit errors that are registered on the PQ3 chip:

chassisd[13928]: <FPC 11> ASIC Error detected errorno 0x00160001 
alarmd[16943]: Alarm set: FPC color=YELLOW, class=CHASSIS, reason=FPC 11 Minor Errors - PQ3 Error code: 0x160001
craftd[13932]:  Minor alarm set, FPC 11 Minor Errors - PQ3 Error code: 0x160001
chassisd[13928]: FPC 11 sent action 0x5 that is not configured 0x8 
alarmd[16943]: Alarm cleared: FPC color=YELLOW, class=CHASSIS, reason=FPC 11 Minor Errors - PQ3 Error code: 0x160001
craftd[13932]: Minor alarm cleared, FPC 11 Minor Errors - PQ3 Error code: 0x160001
mgd[49345]: UI_CMDLINE_READ_LINE: User 'rlozano', command 'show chassis fpc 11 detail '
chassisd[13928]: <FPC 11> ASIC Error detected errorno 0x00160001 
alarmd[16943]: Alarm set: FPC color=YELLOW, class=CHASSIS, reason=FPC 11 Minor Errors - PQ3 Error code: 0x160001
craftd[13932]:  Minor alarm set, FPC 11 Minor Errors - PQ3 Error code: 0x160001
chassisd[13928]: FPC 11 sent action 0x5 that is not configured 0x8 
alarmd[16943]: Alarm cleared: FPC color=YELLOW, class=CHASSIS, reason=FPC 11 Minor Errors - PQ3 Error code: 0x160001
craftd[13932]: Minor alarm cleared, FPC 11 Minor Errors - PQ3 Error code: 0x160001
mgd[20451]: UI_CMDLINE_READ_LINE: User 'eodhiambo', command 'show chassis fpc 11 detail '
chassisd[13928]: <FPC 11> ASIC Error detected errorno 0x00160001 
alarmd[16943]: Alarm set: FPC color=YELLOW, class=CHASSIS, reason=FPC 11 Minor Errors - PQ3 Error code: 0x160001
craftd[13932]:  Minor alarm set, FPC 11 Minor Errors - PQ3 Error code: 0x160001
chassisd[13928]: FPC 11 sent action 0x5 that is not configured 0x8 
alarmd[16943]: Alarm cleared: FPC color=YELLOW, class=CHASSIS, reason=FPC 11 Minor Errors - PQ3 Error code: 0x160001
craftd[13932]: Minor alarm cleared, FPC 11 Minor Errors - PQ3 Error code: 0x160001
chassisd[13928]: <FPC 11> ASIC Error detected errorno 0x00160001 
alarmd[16943]: Alarm set: FPC color=YELLOW, class=CHASSIS, reason=FPC 11 Minor Errors - PQ3 Error code: 0x160001
craftd[13932]:  Minor alarm set, FPC 11 Minor Errors - PQ3 Error code: 0x160001
chassisd[13928]: FPC 11 sent action 0x5 that is not configured 0x8 
craftd[13932]: Minor alarm cleared, FPC 11 Minor Errors - PQ3 Error code: 0x160001
alarmd[16943]: Alarm cleared: FPC color=YELLOW, class=CHASSIS, reason=FPC 11 Minor Errors - PQ3 Error code: 0x160001
 
 
fpc11 Cmerror Op Set: PQ3 Chip: PQ3 DDR single bit error minor alarm raised 
fpc11 Cmerror Op Set: PQ3 Chip: PQ3 DDR single bit error minor alarm raised 
fpc11 Cmerror Op Set: PQ3 Chip: PQ3 DDR single bit error minor alarm raised 
fpc11 Cmerror Op Set: PQ3 Chip: PQ3 DDR single bit error minor alarm raised 
fpc11 Cmerror Op Set: PQ3 Chip: PQ3 DDR single bit error minor alarm raised 
fpc11 Cmerror Op Set: PQ3 Chip: PQ3 DDR single bit error minor alarm raised 
fpc11 Cmerror Op Set: PQ3 Chip: PQ3 DDR single bit error minor alarm raised 
 
 
fpc11 DDR: last error at addr 0x6d5d6350, bad data/mask 0x084ba471004eb2aa/0x0800000000000000 bad ecc/mask=0x52/0x00 
fpc11 DDR: last error at addr 0x6d5d6350, bad data/mask 0x084ba471004eb2aa/0x0800000000000000 bad ecc/mask=0x52/0x00 
fpc11 DDR: last error at addr 0x6d5d6350, bad data/mask 0x084ba471004eb2aa/0x0800000000000000 bad ecc/mask=0x52/0x00 
fpc11 DDR: last error at addr 0x6d5d6350, bad data/mask 0x084ba471004eb2aa/0x0800000000000000 bad ecc/mask=0x52/0x00 
fpc11 DDR: last error at addr 0x6d5d6350, bad data/mask 0x084ba471004eb2aa/0x0800000000000000 bad ecc/mask=0x52/0x00 
fpc11 DDR: last error at addr 0x6d5d6350, bad data/mask 0x084ba471004eb2aa/0x0800000000000000 bad ecc/mask=0x52/0x00 
fpc11 DDR: last error at addr 0x6d5d6350, bad data/mask 0x084ba471004eb2aa/0x0800000000000000 bad ecc/mask=0x52/0x00

Solution

Run the following command from the affected FPC and verify:

> start shell pfe network fpc <slot>
#sh cmerror module 1 error 0x160001
#show  cmerror module

Example output:

xxxxxxxxxx# sh cmerror module 1 error 0x160001
 
Error-id              : 0x160001
 
Error Name            : CMERROR_PQ3_DDR_1_BIT_MINOR
Identifier            : /fpc/9/pfe/0/cm/0/PQ3_Chip/0/CMERROR_PQ3_DDR_1_BIT_MINOR
Description           : PQ3 DDR single bit minor error
State                 : enabled
Scope                 : board
Category              : memory
PFE                   : 0
Configured Level      : Minor
Default Level         : Minor
Count                 : 0
Threshold             : 1
Error Limit           : 1
Occur Count           : 2540
Clear Count           : 2540
Last-occurred(ms ago) : 91189

Logs:

----------------------------------------------------------
Index  Time                 Sub-Err   State    Description
----------------------------------------------------------
0      03/31/20 23:37:12    0         Clear    PQ3 DDR single bit error minor alarm cleared
1      03/31/20 23:36:42    0         Set      PQ3 DDR single bit error minor alarm raised
2      03/31/20 23:34:42    0         Clear    PQ3 DDR single bit error minor alarm cleared
3      03/31/20 23:34:12    0         Set      PQ3 DDR single bit error minor alarm raised
4      03/31/20 23:32:42    0         Clear    PQ3 DDR single bit error minor alarm cleared
5      03/31/20 23:32:12    0         Set      PQ3 DDR single bit error minor alarm raised
6      03/31/20 23:30:42    0         Clear    PQ3 DDR single bit error minor alarm cleared
7      03/31/20 23:30:12    0         Set      PQ3 DDR single bit error minor alarm raised
8      03/31/20 23:28:12    0         Clear    PQ3 DDR single bit error minor alarm cleared
9      03/31/20 23:27:42    0         Set      PQ3 DDR single bit error minor alarm raised
10     03/31/20 23:26:12    0         Clear    PQ3 DDR single bit error minor alarm cleared
11     03/31/20 23:25:42    0         Set      PQ3 DDR single bit error minor alarm raised
12     03/31/20 23:24:12    0         Clear    PQ3 DDR single bit error minor alarm cleared
13     03/31/20 23:23:12    0         Set      PQ3 DDR single bit error minor alarm raised
14     03/31/20 23:21:42    0         Clear    PQ3 DDR single bit error minor alarm cleared
15     03/31/20 23:21:12    0         Set      PQ3 DDR single bit error minor alarm raised
16     03/31/20 23:19:43    0         Clear    PQ3 DDR single bit error minor alarm cleared
17     03/31/20 23:19:12    0         Set      PQ3 DDR single bit error minor alarm raised
18     03/31/20 23:17:12    0         Clear    PQ3 DDR single bit error minor alarm cleared
19     03/31/20 23:16:42    0         Set      PQ3 DDR single bit error minor alarm raised
20     03/31/20 23:15:12    0         Clear    PQ3 DDR single bit error minor alarm cleared
21     03/31/20 23:14:42    0         Set      PQ3 DDR single bit error minor alarm raised
22     03/31/20 23:13:12    0         Clear    PQ3 DDR single bit error minor alarm cleared

These errors are auto-correctable and no impact should be seen in forwarding traffic on the PFE.

The 'PFE: SDRAM ECC' error message is reported in the system message file whenever a SDRAM ECC error is detected by a system component (such as FPC).

The PFE components include error checking and correction (ECC) to protect their SDRAM. This ECC correction provides error detection of single-bit and double-bit errors and correction of single-bit errors for the SDRAM as follows:

  • If ECC detects a single-bit error, it logs the error, automatically corrects the error, and operation continues.
  • ECC errors have two main causes; hardware failure (bad memory, failed board components) or transient anomaly (transient noise, external radiation, and so on).

 These errors appear when reading a certain memory zone which may be corrupted and generates these errors.

Recommended Action:

 This is most likely a transient anomaly that does not require further action to be performed other than monitoring it. If the issue continues to occur, then an FPC reboot would be the next action to take to clear these errors.

Modification History

2022-11-15 Update to add Major error (0x160002), and "Recommend Action" section
2022-10-21 Marks KB as public