Description

This article indicates a specific scenario of Contrail Cloud 13.4 installation failure due to old configuration existence.

Symptoms

# sudo /var/lib/contrail_cloud/scripts/install_contrail_cloud_manager.sh
<<skipping long output>>
TASK [undercloud : Copy contrail env templates] ********************************
Tuesday 21 July 2020  09:20:08 -0700 (0:00:00.757)       0:08:19.025 **********
fatal: [X.X.X.X -> X.X.X.X ]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --out-format=<<CHANGED>>%i %n%L /usr/share/tf-tripleo-heat-templates/ /home/stack/tripleo-heat-templates/", "msg": "rsync: change_dir \"/usr/share/tf-tripleo-heat-templates\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2]\n", "rc": 23}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=105  changed=26   unreachable=0    failed=0    skipped=33   rescued=0    ignored=2   
X.X.X.X                    : ok=74   changed=19   unreachable=0    failed=1    skipped=39   rescued=0    ignored=0   

Tuesday 21 July 2020  09:20:08 -0700 (0:00:00.732)       0:08:19.757 **********
===============================================================================
Wait for undercloud VM to come up from reboot -------------------------- 63.60s
common : Install required packages ------------------------------------- 45.39s
common : Update all packages ------------------------------------------- 42.29s
undercloud : Restart neutron services ---------------------------------- 31.15s
undercloud : Restart nova services ------------------------------------- 21.22s
common : Sync time ----------------------------------------------------- 15.81s
undercloud : Restart ironic services ----------------------------------- 15.78s
common : Sync time ----------------------------------------------------- 15.68s
Wait 15 seconds -------------------------------------------------------- 15.07s
common : Install required packages ------------------------------------- 11.35s
common : Install required packages ------------------------------------- 10.72s
undercloud : Install ironic packages ----------------------------------- 10.36s
common : Install required packages -------------------------------------- 9.20s
common : Update all packages -------------------------------------------- 7.45s
common : Update all packages -------------------------------------------- 7.21s
Configure the undercloud.conf file -------------------------------------- 6.77s
common : Check available subscriptions ---------------------------------- 4.77s
undercloud : Get neutron ctlplane subnet UUID --------------------------- 4.37s
undercloud : Set DNS Nameservers from global['dns'] list ---------------- 4.33s
common : Enable repositories -------------------------------------------- 4.26s
Playbook run took 0 days, 0 hours, 8 minutes, 19 seconds

Playbook install_contrail_cloud_manager.yml Failed!

Solution

If undercloud VM exists from the previous version of Contrail Cloud installations (eg 13.2,13.3) or if it was removed with virsh shutdown/undefine undercloud, since VM image is cached, this issue can be seen.

One other way to verify if this is the problem is by checking the version of contrail-tripleo-heat-templates package in undercloud VM.
Note: Switch to Contrail user first before ssh undercloud from jumphost.

$ rpm -qa contrail-tripleo-heat-templates
contrail-tripleo-heat-templates-20190820065838-6f1cb1b.noarch   <<--Indicate older version of package

In these cases, use "install_contrail_cloud_manager.sh -c" which will unregister undercloud VM from Satellite, unregister node, unregister from any existing subscriptions, undefine networks, delete undercloud VM, Remove VM from /etc/hosts and remove VM definition from .ssh/config. 

# sudo /var/lib/contrail_cloud/scripts/install_contrail_cloud_manager.sh -c
You are going to execute destructive operation. Are you sure (y/N)? y
<<skipping long output>>
Tuesday 21 July 2020  10:24:08 -0700 (0:00:00.034)       0:00:38.007 **********
===============================================================================
create-vm-on-jumphost : Remove unnecessary files ----------------------- 10.48s
common : Unregister from any existing subscriptions --------------------- 7.28s
restart docker ---------------------------------------------------------- 5.84s
Gathering Facts --------------------------------------------------------- 2.09s
Remove unused files ----------------------------------------------------- 1.89s
docker : Install Docker ------------------------------------------------- 1.53s
Setup fact_path --------------------------------------------------------- 1.22s
create-vm-on-jumphost : Destroy domain ---------------------------------- 0.91s
Undefine networks ------------------------------------------------------- 0.76s
create-vm-on-jumphost : Find the VM IP ---------------------------------- 0.72s
Unset net.ipv4.ip_forward ----------------------------------------------- 0.69s
create-vm-on-jumphost : Remove VM from /etc/hosts ----------------------- 0.66s
create-vm-on-jumphost : Remove VM definition from .ssh/config ----------- 0.65s
Destroy networks -------------------------------------------------------- 0.56s
create-vm-on-jumphost : Undefine domain --------------------------------- 0.53s
docker : Remove daemon.json --------------------------------------------- 0.49s
Clean ssh config -------------------------------------------------------- 0.49s
create-vm-on-jumphost : include_tasks ----------------------------------- 0.11s
docker : include_tasks -------------------------------------------------- 0.10s
Get requested state ----------------------------------------------------- 0.10s
Playbook run took 0 days, 0 hours, 0 minutes, 38 seconds
 

Post that run - install_contrail_cloud_manager.sh

sudo /var/lib/contrail_cloud/scripts/install_contrail_cloud_manager.sh
<<skipping long output>>
PLAY RECAP *********************************************************************
127.0.0.1                  : ok=120  changed=48   unreachable=0    failed=0    skipped=19   rescued=0    ignored=2   
X.X.X.X.                   : ok=150  changed=80   unreachable=0    failed=0    skipped=34   rescued=0    ignored=3   

Tuesday 21 July 2020  11:56:34 -0700 (0:00:00.033)       1:31:14.317 **********
===============================================================================
Install undercloud openstack on undercloud VM only if not already installed  1206.96s
registry : Upload docker images --------------------------------------- 785.42s
undercloud : Customize provisioning image ----------------------------- 544.94s
registry : Upload docker images --------------------------------------- 466.76s
registry : Build cc_tempest image ------------------------------------- 440.60s
common : Install required packages ------------------------------------ 431.85s
common : Update all packages ------------------------------------------ 369.89s
registry : Install Contrail Cloud Openstack containers rpm ------------ 166.14s
registry : Upload docker images --------------------------------------- 141.28s
registry : Install Contrail containers rpm ---------------------------- 115.62s
common : Install required packages ------------------------------------- 74.31s
undercloud : Upload overcloud images ----------------------------------- 52.26s
common : Install required packages ------------------------------------- 47.40s
common : Install required packages ------------------------------------- 41.93s
registry : Install Contrail Command containers rpm --------------------- 41.76s
undercloud : Restart neutron services ---------------------------------- 40.35s
create-vm-on-jumphost : Resize partition ------------------------------- 39.77s
Wait for undercloud VM to come up from reboot -------------------------- 30.59s
undercloud : Restart nova services ------------------------------------- 24.10s
undercloud : Create flavors -------------------------------------------- 22.21s
Playbook run took 0 days, 1 hours, 31 minutes, 14 seconds