For libvirt 1.3.1 in Contrail 3.2 package, the libvirt-bin pre-stop script includes an obsoleted file, which results in an error. This error in the pre-stop script may cause the system to hang while shutting down.

This article describe how to fix the error.

For libvirt 1.3.1 in Contrail 3.2 package, the libvirt-bin pre-stop script includes /usr/lib/libvirt/libvirt-stop-guests . However, this file is obsolete and does not exist under the libvirt folder due to which the pre-stop script error and possible timeout occurs.

root@compute:~# cat /etc/init/libvirt-bin.conf 
description "libvirt daemon"
author "Dustin Kirkland <[email protected]>"

start on stopped rc RUNLEVEL=[2345]
stop on starting rc RUNLEVEL=[016]

expect daemon
respawn

# daemonize
env libvirtd_opts="-d"
# whether libvirtd should run at boot/shutdown
env start_libvirtd="yes"
# by default wait 30 seconds for vms to shut down
env libvirtd_shutdown_timeout=30
# uris for which to shut down vms
env libvirt_uris='qemu:///system lxc:/// xen:///'

pre-start script
        [ -r /etc/default/libvirt-bin ] && . /etc/default/libvirt-bin
        [ ! "x$start_libvirtd" = "xyes" ] && { stop; exit 0; }
        mkdir -p /var/run/libvirt
        # Clean up a pidfile that might be left around
        rm -f /var/run/libvirtd.pid
end script

post-start script
        sockfile=/var/run/libvirt/libvirt-sock
        sockfile_check_retries=5
        while [ ! -S $sockfile ] ; do
                        echo "Waiting for $sockfile - recheck in 2 sec"
                        sleep 2;
                       if ! sockfile_check_retries=`expr $sockfile_check_retries - 1`; then
                                        echo "Giving up waiting for $sockfile."
                                        stop; exit 0
                        fi
        done
        echo "$sockfile ready."
end script

pre-stop script
        if [ -z "$RUNLEVEL" ]; then
                exit 0
        fi

        if [ "$RUNLEVEL" -ne 0 ] && [ "$RUNLEVEL" -ne 1 ] && [ "$RUNLEVEL" -ne 6 ]; then
                exit 0
        fi

        /usr/lib/libvirt/libvirt-stop-guests
end script

# /etc/default/libvirt-bin will be deprecated soon.
# If you used to set $libvirtd_opts in /etc/default/libvirt-bin,
# change the 'exec' line here instead.
script
        [ -r /etc/default/libvirt-bin ] && . /etc/default/libvirt-bin
        exec /usr/sbin/libvirtd $libvirtd_opts
end script

root@compute:~# dpkg -l | grep libvirt
ii  libvirt-bin                          1.3.1-1ubuntu10.1~cloud0+contrail1    amd64        programs for the libvirt library
ii  libvirt0:amd64                       1.3.1-1ubuntu10.1~cloud0+contrail1    amd64        library for interfacing with different virtualization systems
ii  nova-compute-libvirt                 2:13.0.0-0ubuntu2~cloud0.1contrail2   all          OpenStack Compute - compute node libvirt support
ii  python-libvirt                       1.3.1-1ubuntu1~cloud0                 amd64        libvirt Python bindings

root@compute:~# ls  /usr/lib/libvirt/ -alt
total 2808
drwxr-xr-x 76 root root    4096 Oct 18  2018 ..
drwxr-xr-x  4 root root    4096 Oct 18  2018 .
drwxr-xr-x  2 root root    4096 Oct 18  2018 connection-driver
drwxr-xr-x  2 root root    4096 Oct 18  2018 lock-driver
-rwxr-xr-x  1 root root  346712 Jun 15  2016 libvirt_iohelper
-rwxr-xr-x  1 root root  371320 Jun 15  2016 libvirt_leaseshelper
-rwxr-xr-x  1 root root 1538016 Jun 15  2016 libvirt_lxc
-rwxr-xr-x  1 root root  342616 Jun 15  2016 libvirt_parthelper
-rwxr-xr-x  1 root root  230368 Jun 15  2016 virt-aa-helper
-rwxr-xr-x  1 root root   16501 Jun 15  2016 libvirt-guests.sh
root@compute:~#

With the above error in the pre-stop script, if you want to shut down a system with virtual machines, the pre-stop script will fail to stop libvirt-guests . Furthermore, libvirt-bin will stop before libvirt-guests due to which libvirt-guests will not be able to send a shutdown command to the running virtual machines. This will result in some qemu processes being left behind as defunct processes, causing the system to hang.

The /usr/lib/libvirt/libvirt-stop-guests file used to exist in the earlier versions of libvirt (libvirt version 1.2.2 for one), but it has been replaced by /etc/init.d/libvirt-guests in more recent versions. However, the pre-stop script was not updated to reflect this change. 

Note: Use the dpkg -l | grep libvirt command to check the version.

The solution for this error is to update the pre-stop script by replacing /usr/lib/libvirt/libvirt-stop-guests with /usr/lib/libvirt/libvirt-guests.sh stop or /etc/init.d/libvirt-guests stop as shown below:

pre-stop script
        if [ -z "$RUNLEVEL" ]; then
                exit 0
        fi

        if [ "$RUNLEVEL" -ne 0 ] && [ "$RUNLEVEL" -ne 1 ] && [ "$RUNLEVEL" -ne 6 ]; then
                exit 0
        fi

        /usr/lib/libvirt/libvirt-guests.sh stop
end script