On cRPD, users may observe the following syslog message, which indicates that the CPU might have crashed.

[4314574.496136] NMI watchdog: Watchdog detected hard LOCKUP on cpu 1

This article provides more information about this error and explains what needs to be done when users see this message.

Users may see the following log after a CPU crash on cRPD.

[4314574.496136] NMI watchdog: Watchdog detected hard LOCKUP on cpu 1
[4314574.496137] Modules linked in: tcp_diag udp_diag inet_diag dummy nf_tables ipip ip6_tunnel tunnel6 sit tunnel4 ip_gre ip_tunnel gre veth xt_nat nf_log_ipv4 nf_log_common xt_LOG xt_limit ipt_REJECT nf_reject_ipv4 iptable_nat bridge nf_conntrack_netlink nfnetlink xfrm_user nf_nat overlay 8021q garp stp mrp llc bonding xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_multiport xt_comment iptable_filter bpfilter coretemp sb_edac x86_pkg_temp_thermal kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd i2c_i801 cryptd glue_helper lpc_ich i2c_core sg ioatdma mfd_core ipmi_si ipmi_devintf evdev ipmi_msghandler acpi_pad button raid1 dm_raid raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq sd_mod ixgbe ehci_pci ahci xfrm_algo ehci_hcd dca libahci hwmon mdio libata ptp crc32c_intel usbcore scsi_mod pps_core usb_common libphy dm_mirror dm_region_hash dm_log dm_mod dax
[4314574.496169] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Not tainted 5.4.172-100.336.amzn2int.x86_64 #1
[4314574.496169] Hardware name: QUANTA BLACKFOOT15/S2VF, BIOS S2V_3A24 08/08/2017
[4314574.496170] RIP: 0010:acpi_processor_ffh_cstate_enter+0x73/0xc0
[4314574.496172] Code: 48 8b 04 25 00 af 01 00 48 89 d1 0f 01 c8 48 8b 00 a8 08 75 15 e9 07 00 00 00 0f 00 2d 86 07 57 00 48 89 f0 48 89 f9 0f 01 c9 <65> 48 8b 04 25 00 af 01 00 f0 80 60 02 df f0 83 44 24 fc 00 48 8b
[4314574.496172] RSP: 0018:ffffc9000631be28 EFLAGS: 00000046
[4314574.496173] RAX: 0000000000000010 RBX: ffff88a03eff2400 RCX: 0000000000000001
[4314574.496174] RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000001
[4314574.496174] RBP: ffff88a03eff2498 R08: 0000000000000002 R09: 000000000002dc00
[4314574.496175] R10: ffffc9000631be58 R11: 000000000000561b R12: 0000000000000002
[4314574.496176] R13: 0000000000000002 R14: 0000000000000002 R15: ffffffff827634a0
[4314574.496176] FS:  0000000000000000(0000) GS:ffff88903f640000(0000) knlGS:0000000000000000
[4314574.496177] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4314574.496178] CR2: 00007f7966d12aa0 CR3: 0000001fe409c001 CR4: 00000000003606e0
[4314574.496178] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4314574.496179] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[4314574.496179] Call Trace:
[4314574.496180]  acpi_idle_enter+0xea/0x290
[4314574.496180]  cpuidle_enter_state+0x72/0x400
[4314574.496181]  cpuidle_enter+0x29/0x40
[4314574.496181]  do_idle+0x200/0x270
[4314574.496182]  cpu_startup_entry+0x19/0x20
[4314574.496182]  start_secondary+0x155/0x1a0
[4314574.496183]  secondary_startup_64+0xa4/0xb0

The Linux kernel can act as a watchdog to detect both soft and hard lockups.

A "hardlockup" is defined as a bug that causes the CPU to loop in kernel mode for more than 10 seconds without letting other interrupts have a chance to run. The current stack trace is displayed upon detection and the system stays locked up unless the default behavior is changed, which can be done through a sysctl.

Since this issue is on the host side, users would need to contact the hardware vendor where the cRPD container is running to troubleshoot the problem.

2022-06-07: Version 1