Description

The article provides reasons due to which system might shutdown abnormally

Symptoms

The device abruptly rebooted on its own with below log messages,

 

Apr 1 02:13:22 labroot eventd[6025]: %SYSLOG-5-SYSTEM_ABNORMAL_SHUTDOWN: System abnormally shut down

Apr 1 02:13:22 labroot eventd[6025]: %SYSLOG-5-SYSTEM_OPERATIONAL: System is operational

 

And the last reason on "show chassis routing-engine" shows "0x4000:VJUNOS reboot"

Solution

  1. Check for log messages before the device went down to check if there any hardware errors or logs due to which system went down.
  2. Check if there were any reboots being done by use due to which system went down from messages file or from interactive command file during the issue time.
  3. The forwarding process stopped receiving keepalives from the control plane, it kicked off the watchdog timer that causes it to reboot to recover after multiple unsuccessful retries:

 

2024-04-21T22:30:50.798456+00:00 labroot vehostd[7323]: syslog_notify_re_state_change: The RE 192.168.1.2 is no more syslog server. Resetting..

2024-04-21T22:30:52+00:00 dcpfe0 dcpfe[2447]: CMLC: Master closed connection (errno=0)

2024-04-21T22:30:52+00:00 dcpfe0 dcpfe[2447]: CMLC: Going disconnected; Routing engine chassis socket closed abruptly

2024-04-21T22:30:52+00:00 dcpfe0 dcpfe[2447]: CMQFX: Connection dropped, Trying to reconnect

2024-04-21T22:30:52+00:00 dcpfe0 dcpfe[2447]: #012 CMQFX(cmlc_platform_periodic): Resetting the pfeman resync delay time

2024-04-21T22:30:52.956989+00:00 labroot vehostd[7323]: vjunos_vmm_periodic: [vjunos0] vehostd missing keepalive

2024-04-21T22:30:55.957388+00:00 labroot vehostd[7323]: vjunos_vm_current_state: [vjunos0] pid 12191 not found

2024-04-21T22:30:55.957401+00:00 labroot vehostd[7323]: vjunos_db_entry_print: Index: 0    Name: vjunos0   State: ONLINE PID: 12191

2024-04-21T22:30:55.957410+00:00 labroot vehostd[7323]: vjunos_db_entry_print: uuid: ffffffaaffffffbbffffffccffffffdd-ffffffeeffffffff-42ffffffe3-0000-4a554e4f5330

2024-04-21T22:30:55.957416+00:00 labroot vehostd[7323]: vjunos_db_entry_print: Cap: Monitor: 1  Recovery: 1 PTY: telnet

2024-04-21T22:30:55.957421+00:00 labroot vehostd[7323]: vjunos_db_entry_print: KA miss_count: 1 boot_timeout: 400 online_timeout: 60

2024-04-21T22:30:55.957426+00:00 labroot vehostd[7323]: vjunos_db_entry_print: Relaunch count: 1 max_count: 3  

2024-04-21T22:30:55.957431+00:00 labroot vehostd[7323]: vjunos_online: [vjunos0] unresponsive : vm state ONLINE to OFFLINE

2024-04-21T22:30:56+00:00 dcpfe0 dcpfe[2447]: SYSIPC:Remote socket shutdown, restart PFE process

 

In this case there are no much logs to confirm exact reason but there might be below possibility

 

1- Control plane (Junos VM) hung and stopped responding

 

2- Host contantly exchanges keepalives with the Junos VM, missed keeaplives due to the CP being hung

 

3- The watchdog triggers the reboot of the box since the CP is not responding



Please reachout  to JTAC for detailed investigation on the logs 

Modification History

2024-05-19 : Article Created