Article Detail

Description

This article is about QFX5100 which may experience reboot with multiple process core dump.

Symptoms

Multiple core dumps on the device

user@QFX5100> show system core-dumps 

-fpc0:

---------------------------------------------------------------------------

--rw-rw---- 1 root field   140680 Nov 18 03:38 /var/tmp/bfdd.core.0.gz

--rw-rw---- 1 root field   139340 Nov 18 03:38 /var/tmp/bfdd.core.1.gz

--rw-rw---- 1 root field   140045 Nov 18 03:39 /var/tmp/bfdd.core.2.gz

--rw-rw---- 1 root field   138551 Nov 18 03:39 /var/tmp/bfdd.core.3.gz

--rw-rw---- 1 root field   140092 Nov 18 03:39 /var/tmp/bfdd.core.4.gz

--rw-rw---- 1 root field  1114112 Nov 18 03:52 /var/tmp/chassisd.core.0.gz

--rw-rw---- 1 root field   196025 Nov 18 03:45 /var/tmp/clksyncd.core.0.gz

--rw-rw---- 1 root field   13213 Nov 18 03:47 /var/tmp/clksyncd.core.1.gz

--rw-rw---- 1 root field   13213 Nov 18 03:47 /var/tmp/clksyncd.core.2.gz

--rw-rw---- 1 root field   13213 Nov 18 03:49 /var/tmp/clksyncd.core.3.gz

--rw-rw---- 1 root field     0 Nov 18 03:56 /var/tmp/clksyncd.core.4.gz

--rw-rw---- 1 root field   601466 Nov 18 03:38 /var/tmp/dfcd.core.0.gz

--rw-rw---- 1 root field   566253 Nov 18 03:39 /var/tmp/dfcd.core.1.gz

--rw-rw---- 1 root field   566252 Nov 18 03:39 /var/tmp/dfcd.core.2.gz

--rw-rw---- 1 root field   566251 Nov 18 03:39 /var/tmp/dfcd.core.3.gz

--rw-rw---- 1 root field   566251 Nov 18 03:40 /var/tmp/dfcd.core.4.gz

--rw-rw---- 1 root field   300399 Nov 18 03:38 /var/tmp/dfwd.core.0.gz

--rw-rw---- 1 root field   151629 Nov 18 03:38 /var/tmp/dfwd.core.1.gz

--rw-rw---- 1 root field   151640 Nov 18 03:39 /var/tmp/dfwd.core.2.gz

--rw-rw---- 1 root field   151633 Nov 18 03:39 /var/tmp/dfwd.core.3.gz

--rw-rw---- 1 root field   151634 Nov 18 03:39 /var/tmp/dfwd.core.4.gz

--rw-rw---- 1 root field   791258 Nov 18 03:38 /var/tmp/dot1xd-secure.core.0.gz

--rw-rw---- 1 root field   383816 Nov 18 03:39 /var/tmp/dot1xd-secure.core.1.gz

--rw-rw---- 1 root field   383815 Nov 18 03:39 /var/tmp/dot1xd-secure.core.2.gz

--rw-rw---- 1 root field   383835 Nov 18 03:39 /var/tmp/dot1xd-secure.core.3.gz

--rw-rw---- 1 root field   383785 Nov 18 03:39 /var/tmp/dot1xd-secure.core.4.gz

--rw-rw---- 1 root field  8518768 Nov 18 03:45 /var/tmp/fxpc.core.0.gz

--rw-rw---- 1 root field   180234 Nov 18 03:52 /var/tmp/fxpc.core.1.gz

--rw-rw---- 1 root field   117030 Nov 18 03:38 /var/tmp/iccpd.core.0.gz

--rw-rw---- 1 root field   117453 Nov 18 03:38 /var/tmp/iccpd.core.1.gz

--rw-rw---- 1 root field   117229 Nov 18 03:39 /var/tmp/iccpd.core.2.gz

--rw-rw---- 1 root field   117275 Nov 18 03:39 /var/tmp/iccpd.core.3.gz

--rw-rw---- 1 root field   117291 Nov 18 03:39 /var/tmp/iccpd.core.4.gz

--rw-rw---- 1 root field   809220 Nov 18 03:38 /var/tmp/l2ald.core.0.gz

--rw-rw---- 1 root field  6012342 Nov 18 03:39 /var/tmp/l2ald.core.1.gz

--rw-rw---- 1 root field  6012342 Nov 18 03:40 /var/tmp/l2ald.core.2.gz

--rw-rw---- 1 root field  6012383 Nov 18 03:43 /var/tmp/l2ald.core.3.gz

--rw-rw---- 1 root field   15133 Nov 18 03:57 /var/tmp/l2ald.core.4.gz

--rw-rw---- 1 root field  1391591 Nov 18 03:38 /var/tmp/l2cpd.core.0.gz

--rw-rw---- 1 root field  1081800 Nov 18 03:38 /var/tmp/l2cpd.core.1.gz

--rw-rw---- 1 root field  1086850 Nov 18 03:39 /var/tmp/l2cpd.core.2.gz

--rw-rw---- 1 root field  1085896 Nov 18 03:39 /var/tmp/l2cpd.core.3.gz

--rw-rw---- 1 root field  1089358 Nov 18 03:39 /var/tmp/l2cpd.core.4.gz

--rw-rw---- 1 root field  5888864 Nov 18 03:55 /var/tmp/mgd.core.0.gz

--rw-rw---- 1 root field  2339641 Nov 18 03:46 /var/tmp/mib2d.core.0.gz

--rw-rw---- 1 root field    7436 Nov 18 03:47 /var/tmp/mib2d.core.1.gz

--rw-rw---- 1 root field    7436 Nov 18 03:47 /var/tmp/mib2d.core.2.gz

--rw-rw---- 1 root field    7436 Nov 18 03:48 /var/tmp/mib2d.core.3.gz

--rw-rw---- 1 root field     0 Nov 18 03:57 /var/tmp/mib2d.core.4.gz

--rw-rw---- 1 root field   537777 Nov 18 03:41 /var/tmp/mplsoamd.core.0.gz

--rw-rw---- 1 root field   156239 Nov 18 03:41 /var/tmp/mplsoamd.core.1.gz

--rw-rw---- 1 root field   156241 Nov 18 03:43 /var/tmp/mplsoamd.core.2.gz

--rw-rw---- 1 root field   156249 Nov 18 03:43 /var/tmp/mplsoamd.core.3.gz

--rw-rw---- 1 root field   156257 Nov 18 03:45 /var/tmp/mplsoamd.core.4.gz

--rw-rw---- 1 root field   205877 Nov 18 03:38 /var/tmp/ppmd.core.0.gz

--rw-rw---- 1 root field   144636 Nov 18 03:38 /var/tmp/ppmd.core.1.gz

--rw-rw---- 1 root field   129180 Nov 18 03:39 /var/tmp/ppmd.core.2.gz

--rw-rw---- 1 root field   139057 Nov 18 03:39 /var/tmp/ppmd.core.3.gz

--rw-rw---- 1 root field   151277 Nov 18 03:39 /var/tmp/ppmd.core.4.gz

--rw-rw---- 1 root field   260037 Nov 18 03:38 /var/tmp/rmopd.core.0.gz

--rw-rw---- 1 root field   264254 Nov 18 03:38 /var/tmp/rmopd.core.1.gz

--rw-rw---- 1 root field   264344 Nov 18 03:39 /var/tmp/rmopd.core.2.gz

--rw-rw---- 1 root field   257222 Nov 18 03:39 /var/tmp/rmopd.core.3.gz

--rw-rw---- 1 root field   264336 Nov 18 03:39 /var/tmp/rmopd.core.4.gz

--rw-rw---- 1 root field  29928838 Nov 18 03:43 /var/tmp/rpd.core.0.gz

--rw-rw---- 1 root field  7392781 Nov 18 03:45 /var/tmp/rpd.core.1.gz

--rw-rw---- 1 root field   86691 Nov 18 03:45 /var/tmp/rpd.core.2.gz

--rw-rw---- 1 root field   79031 Nov 18 03:47 /var/tmp/rpd.core.3.gz

--rw-rw---- 1 root field     0 Nov 18 03:56 /var/tmp/rpd.core.4.gz

--rw-rw---- 1 root wheel   208540 Nov 18 03:41 /var/tmp/sdk-vmmd.core.0.gz

--rw-rw---- 1 root field   205101 Nov 18 03:45 /var/tmp/sdk-vmmd.core.1.gz

--rw-rw---- 1 root field    8238 Nov 18 03:46 /var/tmp/sdk-vmmd.core.2.gz

--rw-rw---- 1 root field    8238 Nov 18 03:47 /var/tmp/sdk-vmmd.core.3.gz

--rw-rw---- 1 root field    8238 Nov 18 03:57 /var/tmp/sdk-vmmd.core.4.gz

--rw-rw---- 1 root field    3107 Nov 18 03:45 /var/tmp/sh.core.0.gz

--rw-rw---- 1 root field 155654227 Nov 18 07:14 /var/tmp/vmcore.0.gz

DMA Read Errors seen on the device

-Nov 18 03:38:24 2024 QFX5100 /kernel: ad0: FAILURE - READ_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=1053711

-Nov 18 03:38:24 2024 QFX5100 /kernel: g_vfs_done():ad0s1a[READ(offset=539459584, length=32768)]error = 5

-Nov 18 03:38:24 2024 QFX5100 /kernel: vn_read_compressed_block: read error = 5; offset = 116623680, size = 65536

-Nov 18 03:38:24 2024 QFX5100 /kernel: vm_fault: pager read error, pid 2183 (l2cpd)

-Nov 18 03:38:24 2024 QFX5100 /kernel: BAD_PAGE_FAULT: pid 2183 (l2cpd), uid 0: pc 0x89330ad4 got a read fault at 0x8a1e37a, x86 fault flags = 0x4

-Nov 18 03:38:24 2024 QFX5100 /kernel: Trapframe Register Dump:

-Nov 18 03:38:24 2024 QFX5100 /kernel:   eax: 0000006c  ecx: 00000000  edx: 08a1e37a  ebx: 8936892c

-Nov 18 03:38:24 2024 QFX5100 /kernel:   esp: afbfb920  ebp: afbfb948  esi: 0000006c  edi: 09fe500f

-Nov 18 03:38:24 2024 QFX5100 /kernel:   eip: 89330ad4  eflags: 00210293

-Nov 18 03:38:24 2024 QFX5100 /kernel:   cs: 0033    ss: 003b    ds: afbf003b  es: 8d4003b

-Nov 18 03:38:24 2024 QFX5100 /kernel:   fs: afbf003b  trapno: 0000000c    err: 00000004

-Nov 18 03:38:24 2024 QFX5100 /kernel: Page table info for PC address 0x89330ad4: PDE = 0x41a4067, PTE = 1954e425

-Nov 18 03:38:24 2024 QFX5100 /kernel: Dumping 16 bytes starting at PC address 0x89330ad4:

-Nov 18 03:38:24 2024 QFX5100 /kernel:   0f b6 02 42 89 55 e4 3d ff 00 00 00 76 0a 89 04

Solution

These core dumps are generated due to the transient SSD failure related to Read DMA failure and a Power cycle would be needed to clear the error condition.

Modification History

2024-11-18 : Article Created