Description

During the normal operation, the device suddenly becomes unresponsive and nothing is recorded in the JunOS or the Host-OS logs. The article describes these symptoms and how to recover the device.

Symptoms

  • No console access as the device is completely unresponsive.
  • No hardware issues on the device or logs seen which could point to a hardware issue.
  • Device Doesn't forward traffic and there are no unusual CPU spikes are noticed before the issue
  • No Junos logs are recorded.
  • There are no core-dumps on the JunOS or the Host-OS as the kernel stops responding
  • Host-logs are recorded as below before the device goes unresponsive.

 

2024-01-14T00:34:13+00:00 dcpfe0 kernel: BUG: sleeping function called from invalid context at /b/dc-builder/yocto-x86/JUNOS_184_R2_BRANCH-x86-64-ovp-2019-05-15_22-45-43/build-qfx-5e-ovp-secure/x86-64-ovp/bitbake_build/tmp/work-shared/intel-x86-64/kernel-source/arch/x86/mm/fault.c:1160

2024-01-14T00:34:13+00:00 dcpfe0 kernel: in_atomic(): 0, irqs_disabled(): 1, pid: 18725, name: ls

2024-01-14T00:34:13+00:00 dcpfe0 kernel: CPU: 4 PID: 18725 Comm: ls Tainted: P   O 3.14.52-rt50-WR7.0.0.9_ovp #1

2024-01-14T00:34:13+00:00 dcpfe0 kernel: Hardware name: Juniper Networks 0C4F/Type2 - Board Product Name1, BIOS QFXS_SFP_00.33_00.01 04/18/2017

2024-01-14T00:34:13+00:00 dcpfe0 kernel: 0000000000000008 ffff88022df1b938 ffffffff818e3860 0000000000000002

2024-01-14T00:34:13+00:00 dcpfe0 kernel: ffff88022df1b948 ffffffff8107f480 ffff88022df1ba48 ffffffff81042983

2024-01-14T00:34:13+00:00 dcpfe0 kernel: 0000000000000000 0000000000000002 ffff880448550090 ffff88022df1ba68

2024-01-14T00:34:13+00:00 dcpfe0 kernel: Call Trace:

2024-01-14T00:34:13+00:00 dcpfe0 kernel: [<ffffffff818e3860>] dump_stack+0x4d/0x66

2024-01-14T00:34:13+00:00 dcpfe0 kernel: [<ffffffff8107f480>] __might_sleep+0xd0/0x100

2024-01-14T00:34:13+00:00 dcpfe0 kernel: [<ffffffff81042983>] __do_page_fault+0x133/0x530

 

 

Solution

This seems to be a hardware transient issue as there are no evident triggers or imminent hardware failures noticed for the issue. It is likely that the kernel communication was broken due to a hardware transient issue and hence the hypervisor logs are seen which eventually led the device to be in the unresponsive state.

To recover the device from issue state, power cycle the device. If the issue is seen in multiple instances or the logs are reported continuously, contact JTAC for further investigation.

Modification History

2024-01-22 : Initial Publication
​​​​​​​2024-07-29 - Visibility modified