This article details how to apply a hotfix that is provided to a customer for Contrail processes when there is no Internet access in the undercloud and overcloud nodes.
Note that this hotfix can be for any Contrail process such as contrail-vrouter-agent and contrail-controller-control-control.
Some important points that should be considered before implementing this procedure are as follows:
This procedure is applicable when the undercloud and overcloud nodes do not have Internet connectivity. If there is Internet connectivity, the image can be downloaded directly from hub.juniper.net.
The primary difference between this procedure and the one performed when there is Internet connectivity is the way in which the image becomes accessible to the compute/controller that needs to be patched with the hotfix.
During this procedure, you should always use "docker load" instead of "docker import." When you use "docker import," all the labels and other parameters attached to the image are dropped. This in-turn generates an error while executing the final steps of the procedure.
Updating overcloud_images.yaml is important because it is used for stack update as well. Performing a stack update with unmodified overcloud_images.yaml will result in rollback of the hotfix that you applied.
overcloud_images.yaml
To apply the hotfix for Contrail processes, perform the following:
Get contrail- <Process_Container_Name> . <Version> from hub.juniper.net by using the pull command.
contrail- <Process_Container_Name> . <Version>
pull
root@mgmt-host:~# docker login -u <JUNIPER REGISTRY USERNAME FROM LASTPASS> -p <JUNIPER REGISTRY PASSWORD FROM LASTPASS> hub.juniper.net/pre-release/ . . . . . Login Succeeded root@mgmt-host:~# docker pull hub.juniper.net/ <Folder_Location> /contrail- <Process_Container_Name> : <Version> <Version> : Pulling from <Folder_Location> /contrail- <Process_Container_Name> c9281c141a1b: Pull complete . . . . . 9b1ba50da51d: Pull complete Digest: sha256:171563f3bdac1142071f0fa0c780e89e14fa59eb3cdc051076ffbb1ac955ae5a Status: Downloaded newer image for hub.juniper.net/ <Folder_Location> /contrail- <Process_Container_Name> : <Version> hub.juniper.net/ <Folder_Location> /contrail- <Process_Container_Name> :
Get the docker image ID.
root@mgmt-host:~# docker image ls | grep -e REPO -e vrouter REPOSITORY TAG IMAGE ID CREATED SIZE hub.juniper.net/ <Folder_Location> /contrail- <Process_Container_Name> <Version> 79f0c703a5de 2 weeks ago 2.49GB
Save the docker image to a tar file for transfer to the required nodes.
root@mgmt-host:~# docker image save -o <Tar_File_Save_Location> /contrail- <Process_Container_Name> _ <Version> .tar 79f0c703a5de
Compress the tar file with gzip.
root@mgmt-host:~# ls -cltr <Tar_File_Save_Location> /contrail- <Process_Container_Name> *.tar -rw------- 1 root root 2515657728 Jun 22 15:53 contrail- <Process_Container_Name> _ <Version> .tar root@mgmt-host:~# gzip <Tar_File_Save_Location> /contrail- <Process_Container_Name> _ <Version> .tar root@mgmt-host:~# ls -cltr <Tar_File_Save_Location> /contrail- <Process_Container_Name> *.gz -rw------- 1 root root 787442747 Jun 22 15:56 contrail- <Process_Container_Name> _ <Version> .tar.gz
Transfer the compressed image archive to <Dir_Tar_File_Save_Location> in the RedHat OpenStack Director node and extract the .tar file.
<Dir_Tar_File_Save_Location>
.tar
[stack@XYZ ]$ gunzip <Dir_Tar_File_Save_Location> /contrail- <Process_Container_Name> _ <Version> .tar.gz
Confirm that docker-distribution.service is running on the RHOSP Director node.
docker-distribution.service
[stack@XYZ ~]$ systemctl status docker-distribution.service ● docker-distribution.service - v2 Registry server for Docker Loaded: loaded (/usr/lib/systemd/system/docker-distribution.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2020-02-21 12:02:17 AEDT; 1 years 4 months ago Main PID: 4591 (registry) Tasks: 28 CGroup: /system.slice/docker-distribution.service └─4591 /usr/bin/registry serve /etc/docker-distribution/registry/config.yml
Fetch the IP address details of the repository and registry that are listening on port 8787.
[stack@XYZ ~]$ sudo ss -tlnp | grep registry LISTEN 0 128 <Local_Dir_Repo_IP> :8787 *:* users:(("registry",pid=4591,fd=3)) **OR** [stack@XYZ ~]$ sudo ss -tnlp | grep registry LISTEN 0 128 <Local_Repo_IP> :8787 *:* users:(("registry",pid=25785,fd=3))
The IP address details will be used with the local repositories on the overcloud nodes.
Test whether the docker image catalog is available from the registry:
[stack@XYZ]$ curl -X GET http://Local_Dir_Repo_IP:8787>/v2/_catalog | python -m json.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4901 0 4901 0 0 91203 0 --:--:-- --:--:-- --:--:-- 94250 { "repositories": [ " <Old_Location> /contrail- <Process_Container_Name> ", . . . . . " <Old_Location> /contrail-analytics-query-engine" ] }
Load the contents of the tarball to create a filesystem docker image.
[stack@XYZ ]$ sudo docker image load -i <Dir_Tar_File_Save_Location> /contrail- <Process_Container_Name> _ <Version> .tar 6ad80b248b34: Loading layer [==================================================>] 2.048 kB/2.048 kB . . . . . 1e9e613db6aa: Loading layer [==================================================>] 14.85 k [stack@XYZ ]$ sudo docker image ls | grep -e REPO -e none REPOSITORY TAG IMAGE ID CREATED SIZE <none> <none> 79f0c703a5de 2 weeks ago 2.49 GB
Tag the container with correct information, and then upload to the local repository of the node for local registry access.
[stack@XYZ ]$ sudo docker tag 79f0c703a5de <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> [stack@XYZ ]$ sudo docker image ls | grep -e REPO -e <New_Location> REPOSITORY TAG IMAGE ID CREATED SIZE <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> <Version> 79f0c703a5de 2 weeks ago 2.49 GB [stack@XYZ ]$ sudo docker push <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> The push refers to a repository [ <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> ] 1e9e613db6aa: Pushed . . . . . <Version> : digest: sha256:171563f3bdac1142071f0fa0c780e89e14fa59eb3cdc051076ffbb1ac955ae5a size: 3252
Modify /home/stack/overcloud/deployment/overcloud_images.yaml . This file is used for initial deployment as well a stack update. So if you miss this step, then the next stack update will roll back the hotfix that you applied.
/home/stack/overcloud/deployment/overcloud_images.yaml
[stack@XYZ ]$ cd /home/stack/overcloud/deployment/ [stack@XYZ deployment]$ cp overcloud_images.yaml overcloud_images.yaml_pre-vrouterHF Modify line for DockerContrailVrouterAgentContainerName and DockerContrailVrouterAgentImageName for persistance. [stack@XYZ deployment]$ grep <New_Location> overcloud_images.yaml DockerContrailVrouterAgentContainerName: <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> DockerContrailVrouterAgentImageName: <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> [stack@XYZ deployment]$ diff overcloud_images.yaml overcloud_images.yaml_pre-vrouterHF < DockerContrailVrouterAgentContainerName: <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> --- > DockerContrailVrouterAgentContainerName: <Local_Repo_IP> :8787/ <Old_Location> /contrail- <Process_Container_Name> :1908.47-rhel --- < DockerContrailVrouterAgentImageName: <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> --- > DockerContrailVrouterAgentImageName: <Local_Repo_IP> :8787/ <Old_Location> /contrail- <Process_Container_Name> :1908.47-rhel
Log in to the hypervisor that you wish to update with the hotfix image container. Capture the output of the following commands and save a copy on Notepad.
[stack@XYZ ~]$ ssh [email protected] [heat-admin@ABC-Compute ~]$ sudo su - [root@ABC-Compute ~]# docker image ls | grep -i <Process_Container_Name> REPOSITORY TAG IMAGE ID CREATED SIZE <Local_Repo_IP> :8787/ <Old_Location> /contrail- <Process_Container_Name> 1908.47-rhel af0b73228dba 22 months ago 3.2 GB [root@ABC-Compute ~]# docker ps -a | grep -i <Process_Container_Name> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3040418ab8ad <Local_Repo_IP> :8787/ <Old_Location> /contrail- <Process_Container_Name> :1908.47-rhel "/entrypoint.sh /u..." 9 months ago Up 9 months contrail- <Process_Container_Name> [root@ABC-Compute ~]# curl -X GET http:// <Local_Repo_IP> :8787/v2/_catalog | python -m json.tool | grep grep -i <Process_Container_Name> % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4905 0 4905 0 0 117k 0 --:--:-- --:--:-- --:--:-- 122k " <Old_Location> /contrail- <Process_Container_Name> ", " <New_Location> contrail- <Process_Container_Name> ",
Back up /var/lib/tripleo-config/hashed-docker-container-startup-config-step_5.json .
/var/lib/tripleo-config/hashed-docker-container-startup-config-step_5.json
[root@ABC-Compute ~]# cd /var/lib/tripleo-config/ [root@ABC-Compute tripleo-config]# ls -al hashed-docker-container-startup-config-step_5.json -rw-------. 1 root root 4223 Sep 23 2020 hashed-docker-container-startup-config-step_5.json [root@ABC-Compute tripleo-config]# cp hashed-docker-container-startup-config-step_5.json hashed-docker-container-startup-config-step_5.json_pre-vrouterHF [root@ABC-Compute tripleo-config]# diff hashed-docker-container-startup-config-step_5.json hashed-docker-container-startup-config-step_5.json_pre-vrouterHF
Edit the new tag as shown in the following file, which corresponds to the contrail- <Process_Container_Name> container in /var/lib/tripleo-config/hashed-docker-container-startup-config-step_5.json .
contrail- <Process_Container_Name>
[root@ABC-Compute tripleo-config]# diff hashed-docker-container-startup-config-step_5.json hashed-docker-container-startup-config-step_5.json_pre-vrouterHF < "image": " <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> ", --- > "image": " <Local_Repo_IP> :8787/ <Old_Location> /contrail- <Process_Container_Name> : <Old_Version> ",
Deploy the new contrail- <Process_Container_Name> container .
contrail- <Process_Container_Name> container
[root@ABC-Compute tripleo-config]# step=5 [root@ABC-Compute tripleo-config]# paunch --debug apply --managed-by paunch --config-id tripleo_step$step --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_$step.json stdout: Trying to pull repository <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> ... <Version> : Pulling from <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> c9281c141a1b: Already exists 31114e120ca0: Already exists 7ad41b0756da: Pulling fs layer . . . . . 78c3993e692f: Pull complete 9b1ba50da51d: Pull complete Digest: sha256:171563f3bdac1142071f0fa0c780e89e14fa59eb3cdc051076ffbb1ac955ae5a Status: Downloaded newer image for <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> stderr: stdout: dc1557ec8a48970650e93b141593ef6d9b75a34a3b90d7466ba97bed570cf259 stderr:
Confirm that the new vRouter container is loaded and that contrail-status is functional.
contrail-status
[root@ABC-Compute tripleo-config]# docker ps -a | grep <New_Location> dc1557ec8a48 <Local_Repo_IP> :8787/ <New_Location> /contrail- <Process_Container_Name> : <Version> "/entrypoint.sh /u..." About a minute ago Up About a minute <Process_Container_Name> [root@ABC-Compute tripleo-config]# contrail-status Pod Service Original Name State Id Status vrouter agent contrail- <Process_Container_Name> running dc1557ec8a48 Up About a minute vrouter agent-dpdk contrail- <Process_Container_Name> -dpdk running d2bbc940c6eb Up 9 months vrouter nodemgr contrail-nodemgr running f2b22c17248a Up 9 months vrouter DPDK module is PRESENT == Contrail vrouter == nodemgr: active agent: active