Description

The vhclient connection is refused and ':error[1]' errors are continuously logged in the following format:

[date][time][nodename] vhclient.[pid].daemon: [em1 internal communication v4 address]: [error]

Symptoms

The device will not connect and the following messages are continuously logged by default in /var/log/messages :

May 30 14:16:28.637 lab-router vhclient.18619.daemon: 192.168.1.1: Connection refused :error[1]

This may cause some 'show vmhost <*>' commands to hang without ever displaying output and require ^c on the CLI to break out and regain CLI prompt.

labroot@ l ab-router> show vmhost version  
^C[abort]

The Junos VM compute node will show Failed status. 

labroot@lab-router> show vmhost    status
Compute cluster: rainier-re-cc
  Compute Node: rainier-re-cn, Online
    Note: Compute node has the following errors:
          - Failed

While in this state, Junos upgrades may fail as the upgrade requires communication between the Host OS and Junos VM. Pings to the Host OS over em1/virbr0 port (192.168.1.0/24) private instance network may succeed while in this state. 

labroot@lab-router> ping 192.168.1.1 routing-instance __juniper_private4__
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.389 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.708 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.423 ms

Other system daemons will have established connections and will be in functional state. 

labroot@lab-router> show system connections | match "192\.168\.1\."

tcp4       0      0  192.168.1.2.53729     192.168.1.1.33081      ESTABLISHED
tcp4       0      0  192.168.1.2.33081     192.168.1.1.53239      ESTABLISHED
tcp4       0      0  192.168.1.2.6022      192.168.1.1.54327      ESTABLISHED
tcp4       0      0  192.168.1.2.5014      192.168.1.1.49843      ESTABLISHED
tcp4       0      0  192.168.1.2.58000     192.168.1.1.58064      ESTABLISHED

Solution

The communication failure between the Host OS and Junos VM occurs over the internal network on the platform over the em1 and virbr0 bridged connection in the __juniper_priivate4__ routing instance. This does not affect forwarding-plane of the device. However, it will interfere with Junos upgrades and will prevent some 'show vmhost' commands from completed.

Junos VM Architecture

Due to communication failure with the Host OS, data collection at the host is not possible.

Steps to take if this issue occurs:

  1. Collect the following outputs:

    • request system information
    • Tarfile of /var/log/*

  2. Contact your JTAC Representative

  3. Power cycle the device.

Note: Please refer to Compressing Troubleshooting Logs from /var/logs to Send to Juniper Networks Technical Support

In addition to providing output from request support information and logs, the data collection below should also be provided to facilitate troubleshooting. These commands execute on the Routing-Engine Shell and provides stats for the internal connection between the Host and VM. 

start shell 
ifconfig
netstat -a
netstat -i em1
netstat -s