Description

In PTX10016 and MX240, which is Linux based Junos, the router may run into an unexpected situation where the following syslog messages and chassis alarm are seen: 

Under 'show chassis alarms'

1 alarms currently active
Alarm time               Class  Description
2020-12-30 19:39:35 UTC  Minor  VMHost RE 0 host vehostd Application failed

Under syslog:

Dec 30 19:39:35  TL.ptx10016 alarmd[10613]: Alarm set: RE color=YELLOW, class=CHASSIS, reason=VMHost RE 0 host vehostd Application failed
Dec 30 19:39:35  TL.ptx10016 craftd[7323]: Receive FX craftd set alarm message: color: 2 class: 100 object: 105 slot: 0 silent: 0 short_reason=VMHost RE 0 host ve long_reason=VMHost RE 0 host vehostd Application failed id=738197609 reason=738197504
Dec 30 19:39:35  TL.ptx10016 craftd[7323]:  Minor alarm set, VMHost RE 0 host vehostd Application failed

Symptoms

This issue is seen with the following conditions:

  • The NG-RE (e.g. RE-S-2X00x6, RE-PTX-X8-128G-S, RE-S-X6-64G-LT) is installed
  • The vehostd is re-spawned with a new PID (process ID) length shorter than the previous PID

The process vehostd may crash without the core file and automatic restart of vehostd may fail unexpectedly. Vehostd is an internal process or daemon for managing the lifecycle of system-critical Junos OS VMs in the system. If the process vehostd gets in crash state, it will impact the management of Junos OS VMs.

Solution

The vehostd uses 5 digit PID and stores that PID in a file. In an affected release, when vehostd is killed, a re-spawned PID is performed, then stored to the same file. However, the new 4 digit PID overwrites 4 digits of the older 5 digit PID. Hence, the 5th digit still remains in the file and causes the subsequent PID check to fail, due to the stale PID.  This is because the older file is appended without first deleting the file. This causes the monitor process to fail to identify that vehostd is running, which leads to multiple re-spawns of vehostd but still unable to locate the right process. Eventually, vehostd goes into un-monitor state.

Refer to  PR1448413 - The vehostd application fails to generate a minor alarm.

Truncate the PID file before creating a new one.

Restoration:

The vehostd can be started manually:

CLI> start shell user root
Junos shell# vhclient -s
Linux shell# vehostd
Linux shell# ps aux | egrep vehostd$

Example:

root@MX-re0-node:~# ps aux | egrep vehostd$
root      9510  0.0  0.0 149708  3248 ?        Ssl  02:07   0:00 vehostd
root@MX-re0-node:~#