Description

For QFX5110 devices we may observe that after upgrade/reboot/power cycle, the device does not come up and on the console we observe continuous scrolling of boot logs.

Symptoms

Following logs seen on the console.

 

BUG: unable to handle kernel NULL pointer dereference at 00000000000002c0
IP: [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
PGD 446fd6067 PUD 27fa84067 PMD 0
Oops: 0000 [#364] SMP
Modules linked in: linux_user_bde(PO+) linux_kernel_bde(PO) xt_NFQUEUE vhost_net vhost macvtap vfio_iommu_type1 vfio_pci vfio act_mirred sch_ingress sch_htb ifb nbd aufs pci_stub iptable_mangle ipt_REJECT xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nfsd bios_drv(O) re_lpc_fpga_drv(O) intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crct10dif_common aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd iTCO_wdt gpio_ich iTCO_vendor_support i2c_i801 lpc_ich fan igb(O)
CPU: 5 PID: 11333 Comm: dcpfe Tainted: P      D    O 3.14.52-rt50-WR7.0.0.9_ovp #1
Hardware name: Juniper Networks 0C4F/Type2 - Board Product Name1, BIOS QFXS_SFP_00.33_00.01 04/18/2017
task: ffff880339fb4490 ti: ffff88033c3f8000 task.ti: ffff88033c3f8000
RIP: 0010:[<ffffffffa003390f>]  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
RSP: 0018:ffff88033c3fbdb0  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000020000000 RCX: 0000000000000000
RDX: 0000000040000000 RSI: ffffea0000e00000 RDI: 0000000000000098
RBP: ffff880040000000 R08: 00000000ff84f930 R09: 000000000000000c
R10: 00000000f7780b20 R11: 0000000000000000 R12: ffff88033c3fbdf8
R13: ffff88033c3fbdf0 R14: ffff88033c3fbde8 R15: 000000000000000c
FS:  0000000000000000(0000) GS:ffff88045fd40000(0063) knlGS:00000000f737bac0
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 00000000000002c0 CR3: 000000027faa8000 CR4: 00000000001427e0
Stack:
 0000000000004c05 00000000ff84f8d4 0000000000000000 ffff88044925b340
 ffff88044925b340 ffffffffa004d4b2 ffff880300000000 ffff880339ec5b98
 000000042f02d865 000000803c3fbe18 ffffffff000000a9 0000000000000062
Call Trace:
 [<ffffffffa004d4b2>] ? _ioctl+0x225/0x76e [linux_user_bde]
 [<ffffffffa004ddc8>] ? _gmodule_compat_ioctl+0x1c/0x20 [linux_user_bde]
 [<ffffffff811de580>] ? compat_sys_ioctl+0xc0/0x13e0
 [<ffffffff810b5e09>] ? __getnstimeofday+0x39/0xd0
 [<ffffffff81042dbc>] ? do_page_fault+0xc/0x10
 [<ffffffff818ecd8f>] ? sysenter_dispatch+0x7/0x1e
Code: 48 01 f0 48 be 00 00 00 00 00 ea ff ff 48 c1 e8 0c 48 6b c0 38 48 01 c6 48 8b 05 4d 94 00 00 48 89 c7 48 81 c7 98 00 00 00 74 0c <4c> 8b 90 c0 02 00 00 4d 85 d2 75 07 4c 8b 15 4e 45 de e1 45 31
RIP  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
 RSP <ffff88033c3fbdb0>
CR2: 00000000000002c0
---[ end trace 713078497440876b ]---
BUG: unable to handle kernel NULL pointer dereference at 00000000000002c0
IP: [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
PGD 44571b067 PUD 33bce8067 PMD 0
Oops: 0000 [#365] SMP
Modules linked in: linux_user_bde(PO+) linux_kernel_bde(PO) xt_NFQUEUE vhost_net vhost macvtap vfio_iommu_type1 vfio_pci vfio act_mirred sch_ingress sch_htb ifb nbd aufs pci_stub iptable_mangle ipt_REJECT xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nfsd bios_drv(O) re_lpc_fpga_drv(O) intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crct10dif_common aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd iTCO_wdt gpio_ich iTCO_vendor_support i2c_i801 lpc_ich fan igb(O)
CPU: 5 PID: 11770 Comm: dcpfe Tainted: P      D    O 3.14.52-rt50-WR7.0.0.9_ovp #1
Hardware name: Juniper Networks 0C4F/Type2 - Board Product Name1, BIOS QFXS_SFP_00.33_00.01 04/18/2017
task: ffff880447dfa250 ti: ffff88044722c000 task.ti: ffff88044722c000
RIP: 0010:[<ffffffffa003390f>]  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
RSP: 0018:ffff88044722fdb0  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000020000000 RCX: 0000000000000000
RDX: 0000000040000000 RSI: ffffea0000e00000 RDI: 0000000000000098
RBP: ffff880040000000 R08: 00000000ff82c6a0 R09: 000000000000000c
R10: 00000000f774bb20 R11: 0000000000000000 R12: ffff88044722fdf8
R13: ffff88044722fdf0 R14: ffff88044722fde8 R15: 000000000000000c
FS:  0000000000000000(0000) GS:ffff88045fd40000(0063) knlGS:00000000f7346ac0
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 00000000000002c0 CR3: 0000000446fd6000 CR4: 00000000001427e0
Stack:
 0000000000004c05 00000000ff82c644 0000000000000000 ffff880448e80280
 ffff880448e80280 ffffffffa004d4b2 ffff880400000000 ffff880339e0cb98
 0000000303504865 000000804722fe18 ffffffff000000a9 0000000000000062
Call Trace:
 [<ffffffffa004d4b2>] ? _ioctl+0x225/0x76e [linux_user_bde]
 [<ffffffffa004ddc8>] ? _gmodule_compat_ioctl+0x1c/0x20 [linux_user_bde]
 [<ffffffff811de580>] ? compat_sys_ioctl+0xc0/0x13e0
 [<ffffffff810b5e09>] ? __getnstimeofday+0x39/0xd0
 [<ffffffff81042dbc>] ? do_page_fault+0xc/0x10
 [<ffffffff818ecd8f>] ? sysenter_dispatch+0x7/0x1e
Code: 48 01 f0 48 be 00 00 00 00 00 ea ff ff 48 c1 e8 0c 48 6b c0 38 48 01 c6 48 8b 05 4d 94 00 00 48 89 c7 48 81 c7 98 00 00 00 74 0c <4c> 8b 90 c0 02 00 00 4d 85 d2 75 07 4c 8b 15 4e 45 de e1 45 31
RIP  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
 RSP <ffff88044722fdb0>
CR2: 00000000000002c0
---[ end trace 713078497440876c ]---
BUG: unable to handle kernel NULL pointer dereference at 00000000000002c0
IP: [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
PGD 27facc067 PUD 4490f3067 PMD 0
Oops: 0000 [#366] SMP
Modules linked in: linux_user_bde(PO+) linux_kernel_bde(PO) xt_NFQUEUE vhost_net vhost macvtap vfio_iommu_type1 vfio_pci vfio act_mirred sch_ingress sch_htb ifb nbd aufs pci_stub iptable_mangle ipt_REJECT xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nfsd bios_drv(O) re_lpc_fpga_drv(O) intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crct10dif_common aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd iTCO_wdt gpio_ich iTCO_vendor_support i2c_i801 lpc_ich fan igb(O)
CPU: 5 PID: 12191 Comm: dcpfe Tainted: P      D    O 3.14.52-rt50-WR7.0.0.9_ovp #1
Hardware name: Juniper Networks 0C4F/Type2 - Board Product Name1, BIOS QFXS_SFP_00.33_00.01 04/18/2017
task: ffff880339c8a3d0 ti: ffff880446618000 task.ti: ffff880446618000
RIP: 0010:[<ffffffffa003390f>]  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
RSP: 0018:ffff88044661bdb0  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000020000000 RCX: 0000000000000000
RDX: 0000000040000000 RSI: ffffea0000e00000 RDI: 0000000000000098
RBP: ffff880040000000 R08: 00000000ffee98e0 R09: 000000000000000c
R10: 00000000f77b8b20 R11: 0000000000000000 R12: ffff88044661bdf8
R13: ffff88044661bdf0 R14: ffff88044661bde8 R15: 000000000000000c
FS:  0000000000000000(0000) GS:ffff88045fd40000(0063) knlGS:00000000f73b3ac0
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 00000000000002c0 CR3: 0000000339c97000 CR4: 00000000001427e0
Stack:
 0000000000004c05 00000000ffee9884 0000000000000000 ffff880446899d80
 ffff880446899d80 ffffffffa004d4b2 ffff880300000000 ffff88027fa7eb98
 0000000431801865 000000804661be18 ffffffff000000a9 0000000000000062
Call Trace:
 [<ffffffffa004d4b2>] ? _ioctl+0x225/0x76e [linux_user_bde]
 [<ffffffffa004ddc8>] ? _gmodule_compat_ioctl+0x1c/0x20 [linux_user_bde]
 [<ffffffff811de580>] ? compat_sys_ioctl+0xc0/0x13e0
 [<ffffffff810b5e09>] ? __getnstimeofday+0x39/0xd0
 [<ffffffff81042dbc>] ? do_page_fault+0xc/0x10
 [<ffffffff818ecd8f>] ? sysenter_dispatch+0x7/0x1e
Code: 48 01 f0 48 be 00 00 00 00 00 ea ff ff 48 c1 e8 0c 48 6b c0 38 48 01 c6 48 8b 05 4d 94 00 00 48 89 c7 48 81 c7 98 00 00 00 74 0c <4c> 8b 90 c0 02 00 00 4d 85 d2 75 07 4c 8b 15 4e 45 de e1 45 31
RIP  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
 RSP <ffff88044661bdb0>
CR2: 00000000000002c0
---[ end trace 713078497440876d ]---
BUG: unable to handle kernel NULL pointer dereference at 00000000000002c0
IP: [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
PGD 4443de067 PUD 339d0b067 PMD 0
Oops: 0000 [#367] SMP
Modules linked in: linux_user_bde(PO+) linux_kernel_bde(PO) xt_NFQUEUE vhost_net vhost macvtap vfio_iommu_type1 vfio_pci vfio act_mirred sch_ingress sch_htb ifb nbd aufs pci_stub iptable_mangle ipt_REJECT xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nfsd bios_drv(O) re_lpc_fpga_drv(O) intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crct10dif_common aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd iTCO_wdt gpio_ich iTCO_vendor_support i2c_i801 lpc_ich fan igb(O)
CPU: 5 PID: 12632 Comm: dcpfe Tainted: P      D    O 3.14.52-rt50-WR7.0.0.9_ovp #1
Hardware name: Juniper Networks 0C4F/Type2 - Board Product Name1, BIOS QFXS_SFP_00.33_00.01 04/18/2017
task: ffff880447dfa250 ti: ffff88033c300000 task.ti: ffff88033c300000
RIP: 0010:[<ffffffffa003390f>]  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
RSP: 0018:ffff88033c303db0  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000020000000 RCX: 0000000000000000
RDX: 0000000040000000 RSI: ffffea0000e00000 RDI: 0000000000000098
RBP: ffff880040000000 R08: 00000000ff9aada0 R09: 000000000000000c
R10: 00000000f76f2b20 R11: 0000000000000000 R12: ffff88033c303df8
R13: ffff88033c303df0 R14: ffff88033c303de8 R15: 000000000000000c
FS:  0000000000000000(0000) GS:ffff88045fd40000(0063) knlGS:00000000f72edac0
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 00000000000002c0 CR3: 0000000446617000 CR4: 00000000001427e0
Stack:
 0000000000004c05 00000000ff9aad44 0000000000000000 ffff88033c006180
 ffff88033c006180 ffffffffa004d4b2 ffff880400000000 ffff8804460bfb98
 0000000430a67865 000000803c303e18 ffffffff000000a9 0000000000000062
Call Trace:
 [<ffffffffa004d4b2>] ? _ioctl+0x225/0x76e [linux_user_bde]
 [<ffffffffa004ddc8>] ? _gmodule_compat_ioctl+0x1c/0x20 [linux_user_bde]
 [<ffffffff811de580>] ? compat_sys_ioctl+0xc0/0x13e0
 [<ffffffff810b5e09>] ? __getnstimeofday+0x39/0xd0
 [<ffffffff81042dbc>] ? do_page_fault+0xc/0x10
 [<ffffffff818ecd8f>] ? sysenter_dispatch+0x7/0x1e
Code: 48 01 f0 48 be 00 00 00 00 00 ea ff ff 48 c1 e8 0c 48 6b c0 38 48 01 c6 48 8b 05 4d 94 00 00 48 89 c7 48 81 c7 98 00 00 00 74 0c <4c> 8b 90 c0 02 00 00 4d 85 d2 75 07 4c 8b 15 4e 45 de e1 45 31
RIP  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
 RSP <ffff88033c303db0>
CR2: 00000000000002c0
---[ end trace 713078497440876e ]---
BUG: unable to handle kernel NULL pointer dereference at 00000000000002c0
IP: [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
PGD 448e81067 PUD 4491ae067 PMD 0
Oops: 0000 [#368] SMP
Modules linked in: linux_user_bde(PO+) linux_kernel_bde(PO) xt_NFQUEUE vhost_net vhost macvtap vfio_iommu_type1 vfio_pci vfio act_mirred sch_ingress sch_htb ifb nbd aufs pci_stub iptable_mangle ipt_REJECT xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nfsd bios_drv(O) re_lpc_fpga_drv(O) intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crct10dif_common aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd iTCO_wdt gpio_ich iTCO_vendor_support i2c_i801 lpc_ich fan igb(O)
CPU: 5 PID: 13069 Comm: dcpfe Tainted: P      D    O 3.14.52-rt50-WR7.0.0.9_ovp #1
Hardware name: Juniper Networks 0C4F/Type2 - Board Product Name1, BIOS QFXS_SFP_00.33_00.01 04/18/2017
task: ffff880446920410 ti: ffff880447018000 task.ti: ffff880447018000
RIP: 0010:[<ffffffffa003390f>]  [<ffffffffa003390f>] lkbde_get_dma_info+0x93/0x156 [linux_kernel_bde]
RSP: 0018:ffff88044701bdb0  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000020000000 RCX: 0000000000000000
RDX: 0000000040000000 RSI: ffffea0000e00000 RDI: 0000000000000098
RBP: ffff880040000000 R08: 00000000fffe4680 R09: 000000000000000c
R10: 00000000f77a2b20 R11: 0000000000000000 R12: ffff88044701bdf8
R13: ffff88044701bdf0 R14: ffff88044701bde8 R15: 000000000000000c
FS:  0000000000000000(0000) GS:ffff88045fd40000(0063) knlGS:00000000f739dac0
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 00000000000002c0 CR3: 00000004455b8000 CR4: 00000000001427e0
Stack:
 0000000000004c05 00000000fffe4624 0000000000000000 ffff880086347140
 ffff880086347140 ffffffffa004d4b2 ffff880400000000 ffff8804450f5b98
 000000042fb0f865 000000804701be18 ffffffff000000a9 0000000000000062
Call Trace:
 [<ffffffffa004d4b2>] ? _ioctl+0x225/0x76e [linux_user_bde]
 [<ffffffffa004ddc8>] ? _gmodule_compat_ioctl+0x1c/0x20 [linux_user_bde]
 [<ffffffff811de580>] ? compat_sys_ioctl+0xc0/0x13e0
 [<ffffffff810b5e09>] ? __getnstimeofday+0x39/0xd0
 [<ffffffff81042dbc>] ? do_page_fault+0xc/0x10
 [<ffffffff818ecd8f>] ? sysenter_dispatch+0x7/0x1e
Code: 48 01 f0 48 be 00 00 00 00 00 ea ff ff 48 c1 e8 0c 48 6b c0 38 48 01 c6 48 8b 05 4d 94 00 00 48 89 c7 48 81 c7 98 00 00 00 74 0c <4c> 8b 90 c0 02 00 00 4d 85 d2 75 07 4c 8b 15 4e 45 de e1 45 31


The above console logs can also be seen in the host logs.
 

Solution

  1. Try to power cycle the device to recover.
  2. If power cycle doesnt work then proceed with the RMA.

Modification History

2024-07-05 : Article Created

Related Information

For More information refer to the following internal PRs.

https://gnats.juniper.net/web/default/1502847#audit_tab
https://gnats.juniper.net/web/default/1463976#audit_tab
https://gnats.juniper.net/web/default/1765127#audit_tab