Customer reported they observed core-dump on upgrading QFX5100 to 21.4R3-S9.
Core-dump was observed after upgrade.
Version: ======== root@QFX5K> show version detail no-forwarding Hostname: QFX5K Model: qfx5100-96s-8q Junos: 21.4R3-S9.5
root@QFX5K> show system uptime no-forwarding Current time: 2024-12-04 18:31:48 UTC Time Source: LOCAL CLOCK System booted: 2024-12-02 17:41:34 UTC (2d 00:50 ago) Protocols started: 2024-12-02 17:44:07 UTC (2d 00:47 ago) Last configured: 2024-12-04 18:22:08 UTC (00:09:40 ago) by aecore 6:31PM up 2 days, 50 mins, 3 users, load averages: 0.37, 0.19, 0.15
root@QFX5K> show system core-dumps no-forwarding -rw-rw---- 1 root field 79705407 Dec 2 17:42 /var/tmp/vmcore.0.gz
Before upgrading the device, i.e. on Nov 30, the device went on reboot due to power failure/cycle.
Nov 30 11:00:36 QFX5K /kernel: Automatic reboot in progress...
Nov 30 11:00:36 QFX5K /kernel: savecore: Reboot reason(s): 0x1: power cycle/failure
Device upgraded to 21.4R3-S9.5 on Dec 2 17:01:02
Dec 2 17:01:02 QFX5K mgd[10085]: UI_CMDLINE_READ_LINE: User 'aecore', command 'request system reboot hypervisor '
Dec 2 17:01:08 QFX5K mgd[10085]: UI_REBOOT_EVENT: System rebooted by 'aecore'
Dec 2 17:06:56 QFX5K /kernel: savecore: Reboot reason(s): 0x2000: hypervisor reboot
The core is generated around 2024-12-02 17:42. Adding below the logs around this time interval.
Dec 2 17:43:48 QFX5K eventd: sendto: No route to host
Dec 2 17:43:48 QFX5K eventd[1327]: SYSTEM_ABNORMAL_SHUTDOWN: System abnormally shut down
Dec 2 17:43:48 QFX5K eventd[1327]: SYSTEM_OPERATIONAL: System is operational
Dec 2 17:43:48 QFX5K /kernel: savecore: reboot after panic: Hardware watchdog timeout
Dec 2 17:43:48 QFX5K /kernel: savecore: writing compressed core to vmcore.0.gz
Dec 2 17:43:48 QFX5K /kernel: savecore: currently booted kernel copied
Dec 2 17:43:48 QFX5K /kernel: savecore: writing compressed kernel to /var/crash/kernel.0.gz
Dec 2 17:43:59 QFX5K /kernel: savecore: Reboot reason(s): 0x2000: hypervisor reboot
Dec 2 17:43:59 QFX5K savecore: Reboot reason(s): 0x2000: hypervisor reboot
Around these time interval, checked the logs to find the reason for abnormal shutdown and core.
"missing keepalive" logs are seen for a long time and it is seen throughout the file ( more than a year ).
Dec 2 17:38:36 localhost vehostd[3773]: vjunos_vmm_periodic: [vjunos0] vehostd missing keepalive
Dec 2 17:38:41 localhost vehostd[3773]: vjunos_vmm_periodic: [vjunos0] vehostd missing keepalive
Dec 2 17:38:46 localhost vehostd[3773]: vjunos_vmm_periodic: [vjunos0] vehostd missing keepalive
But this Keep alive miss count ( KA miss_count ) crossed boot timeout value of 400 on Dec 2 17:39:22.
Dec 2 17:05:17 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 2 boot_timeout: 400 online_timeout: 60
Dec 2 17:06:08 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 12 boot_timeout: 400 online_timeout: 60
Dec 2 17:06:59 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 22 boot_timeout: 400 online_timeout: 60
Dec 2 17:07:50 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 32 boot_timeout: 400 online_timeout: 60
Dec 2 17:08:41 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 42 boot_timeout: 400 online_timeout: 60
Dec 2 17:09:32 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 52 boot_timeout: 400 online_timeout: 60
Dec 2 17:10:23 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 62 boot_timeout: 400 online_timeout: 60
Dec 2 17:11:14 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 72 boot_timeout: 400 online_timeout: 60
Dec 2 17:12:05 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 82 boot_timeout: 400 online_timeout: 60
Dec 2 17:12:56 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 92 boot_timeout: 400 online_timeout: 60
Dec 2 17:13:47 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 102 boot_timeout: 400 online_timeout: 60
Dec 2 17:14:38 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 112 boot_timeout: 400 online_timeout: 60
Dec 2 17:15:29 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 122 boot_timeout: 400 online_timeout: 60
Dec 2 17:16:20 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 132 boot_timeout: 400 online_timeout: 60
Dec 2 17:17:11 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 142 boot_timeout: 400 online_timeout: 60
Dec 2 17:18:02 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 152 boot_timeout: 400 online_timeout: 60
Dec 2 17:18:53 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 162 boot_timeout: 400 online_timeout: 60
Dec 2 17:19:44 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 172 boot_timeout: 400 online_timeout: 60
Dec 2 17:20:35 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 182 boot_timeout: 400 online_timeout: 60
Dec 2 17:21:26 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 192 boot_timeout: 400 online_timeout: 60
Dec 2 17:22:17 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 202 boot_timeout: 400 online_timeout: 60
Dec 2 17:23:08 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 212 boot_timeout: 400 online_timeout: 60
Dec 2 17:23:59 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 222 boot_timeout: 400 online_timeout: 60
Dec 2 17:24:50 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 232 boot_timeout: 400 online_timeout: 60
Dec 2 17:25:41 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 242 boot_timeout: 400 online_timeout: 60
Dec 2 17:26:32 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 252 boot_timeout: 400 online_timeout: 60
Dec 2 17:27:23 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 262 boot_timeout: 400 online_timeout: 60
Dec 2 17:28:14 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 272 boot_timeout: 400 online_timeout: 60
Dec 2 17:29:05 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 282 boot_timeout: 400 online_timeout: 60
Dec 2 17:29:56 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 292 boot_timeout: 400 online_timeout: 60
Dec 2 17:30:47 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 302 boot_timeout: 400 online_timeout: 60
Dec 2 17:31:38 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 312 boot_timeout: 400 online_timeout: 60
Dec 2 17:32:29 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 322 boot_timeout: 400 online_timeout: 60
Dec 2 17:33:20 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 332 boot_timeout: 400 online_timeout: 60
Dec 2 17:34:11 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 342 boot_timeout: 400 online_timeout: 60
Dec 2 17:35:02 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 352 boot_timeout: 400 online_timeout: 60
Dec 2 17:35:53 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 362 boot_timeout: 400 online_timeout: 60
Dec 2 17:36:44 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 372 boot_timeout: 400 online_timeout: 60
Dec 2 17:37:35 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 382 boot_timeout: 400 online_timeout: 60
Dec 2 17:38:26 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 392 boot_timeout: 400 online_timeout: 60
Dec 2 17:39:22 localhost vehostd[3773]: vjunos_db_entry_print: KA miss_count: 401 boot_timeout: 400 online_timeout: 60
After this interval, vmcore is generated.
Looks like it is HW related issue and one off issue.