This article explains the meaning of the "CPU Error code: 0x280003", "CPU Error code: 0x280002", and "CPU Error code: 0x280001" alarms and also provides the steps to resolve them.
Syslogs and Alarms
The alarm and logs for "CPU Error 0x280003" are given below:
Active Alarm
user@host> show chassis alarms 1 alarms currently active Alarm time Class Description 2020-06-09 22:01:11 UTC Major FPC 2 Major Errors - CPU Error code: 0x280003
Logs
Jun 9 22:01:11 alarmd[72819]: Alarm set: FPC color=RED, class=CHASSIS, reason=FPC 2 Major Errors - CPU Error code: 0x280003 Jun 9 22:01:11 alarmd[72819]: Alarm set: FPC color=RED, class=CHASSIS, reason=FPC 2 Major Errors - CPU Error code: 0x280003 Jun 9 22:01:11 fpc2 Cmerror: Draining ASIC error message queue Jun 9 22:01:11 fpc2 cmerror_process_queue: module = CPU Error
The alarm and logs for CPU Error 0x280002 are given below:
user@host> show chassis alarms 1 alarms currently active Alarm time Class Description 2020-01-14 04:57:44 UTC Minor FPC 2 Minor Errors - CPU Error code: 0x280002
Jun 14 04:48:12 alarmd[39959]: Alarm set: FPC color=RED, class=CHASSIS, reason=FPC 2 Minor Errors - CPU Error code: 0x280002 Jun 14 04:58:12 alarmd[39959]: Alarm set: FPC color=RED, class=CHASSIS, reason=FPC 2 Minor Errors - CPU Error code: 0x280002 Jan 14 04:58:13 last message repeated 15 times Jan 14 04:58:13 last message repeated 8 times Jan 14 04:58:14 chassisd[3858]: %DAEMON-3: ASIC Error detected errorno 0x00280002
Explanation
The CPU errors may be generated due to memory errors on the ASIC. There are two types of errors, correctable memory errors, and uncorrectable errors. These errors generate syslog and alarms as follows:
The correctable memory errors alarm is cleared automatically when the error stops. There is no need to reboot the FPC to clear the alarm. In some cases, the FPC may reboot to clear the alarm on its own. See below for how to check for the history of correctable memory errors.
user@host>show chassis fpc detail Slot 2 information: State Online Temperature 29 Total CPU DRAM 3200 MB Total RLDRAM 516 MB Total DDR DRAM 8192 MB Start time 2020-06-09 21:59:45 UTC Uptime 6 hours, 12 minutes, 55 seconds Max power consumption 465 Watts
Steps
Check the FPC alarm count and the active alarm
user@host> show chassis fpc errors >> Total FPC Minor/Major/Fatal summary FPC Level Occurred Cleared Threshold Action-Taken Action <snip> 2 Minor 38 38 10 38 LOG| Major 4 4 1 4 GET STATE|CM ALARM| Fatal 0 0 1 0 RESET user@host> show chassis errors active detail fpc-slot 2 Slot 2 -------------------------------------------------- Location : FPC 2 Identifier : /fpc/2/pfe/0/cm/0/CPU_Error/0/CPU_CMERROR_DDR_CORRECTABLE_MINOR Error : CPU_CMERROR_DDR_CORRECTABLE_MINOR Scope : board Category : processing Severity : Minor Details : Minor CPU DDR correctable error Count : 38 Support : No help info provided
Check for the FPC alarm module.
user@host> request pfe execute command “show cmerror module” target fpc2
This output will list a number of modules with a module number for each cmerror. Make a note of the module number.
Example Output
<snip> Module (4) (CPU Error) >>>>> Make a note of the module number. Error-id PFE Level Threshold Count Occurred Cleared Last-occurred(ms ago) Description -------------------------------------------------------------------------------------------- 0x280001 0 Major 1 0 0 0 0 CPU_CMERROR_DDR_UNCORRECTABLE 0x280002 0 Minor 1 0 2 2 28646703 Minor CPU DDR correctable error 0x280003 0 Major 1 0 0 0 0 Major CPU DDR correctable error <snip>
The error module for the FPC, in this case, is 4.
Check the error information to determine the reason.
user@host> request pfe execute command "show cmerror module 4 error 0x280002" target fpc2 >> Input error code as seen in alarm Description : Minor CPU DDR correctable error PFE : 0 State : enabled Configured Level : Minor Default Level : Minor Count : 0 Threshold : 1 Error Limit : 0 Occur Count : 2 Clear Count : 2 Last-occurred(ms ago) : 28646703
Logs
---------------------------------------------------------- Index Time Sub-Err State Description ---------------------------------------------------------- 0 02/04/20 21:00:06 0 Clear Clear CPU DDR correctable minor alarm 1 02/04/20 20:59:36 0 Set Raise CPU DDR correctable minor alarm 2 02/04/20 20:54:06 0 Clear Clear CPU DDR correctable minor alarm 3 02/04/20 20:53:36 0 Set Raise CPU DDR correctable minor alarm 4 02/04/20 17:40:58 0 Clear Clear CPU DDR correctable minor alarm 5 02/04/20 17:40:28 0 Set Raise CPU DDR correctable minor alarm user@host> request pfe execute command "show cmerror module 4 error 0x280003" target fpc2 >> Input error code as seen in alarm Description : Major CPU DDR correctable error PFE : 0 State : enable Configured Level : Major Default Level : Major <snip>
Since the cause is a hardware memory error, corrected errors are transparent to the system. If the message/alarm occurs just once or very sporadically, it can be ignored.
If the FPC is offline due to the error, there is a possibility that this error is fatal in nature. To recover from this error, ideally, the FPC resets on its own. However, if the FPC alarm has not auto-cleared, you can manually restart the FPC, as shown below.
Restart the FPC if the FPC is offline
user@host> request chassis fpc (offline | online | restart) slot slot-number
In this example:
user@host> request chassis fpc restart slot 2
If a restart does not solve the issue or clear the alarm, contact Support for detailed analysis. This could be a possible hardware issue.
Collect the following logs for troubleshooting and analysis, in addition to the ones collected above:
show log messages show log chassisd request execute pfe command "show nvram" target fpc<FPC#> request execute pfe command "show syslog messages" target fpc<FPC#>
Note: For line cards MX2K-MPC7/8/9, the default threshold for single-bit correctable ECC errors on PMB DDR memory is increased from 1 to 10 before declaring a minor alarm. Refer to PR1539779 for enhancement details.
2024-05-28: Added clarification about detection interval of 24 hours for 0x280002 and 0x280003; minor formatting changes.2023-06-26: Added note on enhancement of threshold increased to 10. 2022-12-27: Fixing typo and reformat2022-11-04: Update to add condition 0x280001. Clarify that "reboot" is not required for errorno 0x280002 and 0x280003. The alarm will be clear after the error stop for 24 hours.2022-06-27: Checked article based on feedback and made minor modifications; article found to be valid and accurate2023-12-25: explanation and content is valid for PTX/QFX boxes as well.
[MX] How to disable a specific CMERROR alarm via configuration