Description

This article explains the solution for the "kernel: mce: [Hardware Error]: Machine check events logged" message seen on the host-log of an RE which went for an abnormal reboot

Symptoms

Unexpected reboot of routing reported and the last reboot reason was set as "0x2000:hypervisor reboot". 


root@re1> show chassis routing-engine no-forwarding

...

Routing Engine status:

Slot 0:

 Current state     Backup

 Election priority    Master (default)

...

 Uptime       4 hours, 50 minutes, 8 seconds

 Last reboot reason    0x2000:hypervisor reboot





The host logs of the specific RE show "kernel: mce: [Hardware Error]: Machine check events logged" 


2024-03-21T19:27:22.999924-04:00 re0 re_fpga: libhostd_open: mq is ready. mqdes 3, msgsize 1024, maxmsg 512, curmsgs 0.

2024-03-21T19:27:23.134886-04:00 re0 root: New timezone: /usr/share/zoneinfo/EST5EDT

2024-03-21T19:27:23.139384-04:00 re0 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.22.0" x-pid="3815" x-info=http://www.rsyslog.com] exiting on signal 15.

2024-03-21T19:27:23.495520-04:00 re0 re_fpga: libhostd_open: mq is ready. mqdes 3, msgsize 1024, maxmsg 512, curmsgs 0.

2024-03-21T19:27:32.095793-04:00 re0 refpga_cmd: MQ related information - mqdes 3, flag 0, msgsize 127, maxmsg 50, curmsgs 0

2024-03-21T19:27:32.097011-04:00 re0 refpga_cmd: msg stat - Tx 1, bad_des 0, bad_length 0, intr 0, tx_timeout 0, q_full 0, unknown 0

2024-03-21T19:27:32.097069-04:00 re0 refpga_cmd: librefpga_log_stat: Response stat: Rx 0, errors - Open 0, MQRemove 0, ID 0, Read 0, Length 0, Wrong type 0

2024-03-21T19:29:20.283146-04:00 re0 kernel: mce: [Hardware Error]: Machine check events logged


Solution

The unexpected reboot of RE can be due to on of the following 


- Data corruption detected in the CPU caches

- In main memory by an integrated memory controller

- Data transfer errors on the front side bus

- CPU interconnect errors 

- Due to other internal errors. 


Routing Engine works well after the reboot. Replace Routing Engine if the errors are repeating


Modification History

2024-04-17 : Article Created