Description

In order to have an increased footprint of the cluster and to make more resources available for unforeseen needs, we sometimes have to scale-out more compute nodes into the contrail cloud cluster. 

Solution

  1. Verify current deployment is healthy and operational by ensuring that your bare metal nodes are not in maintenance mode before you begin scaling out or removing an overcloud node. Eg Use the following command to check this: 
    (undercloud) [stack@undercloud ~]$ openstack baremetal node list
+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name                  | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+
| 9538f7ba-04d8-4114-994a-5b9beab67b64 | control_1             | 813490f1-8bcf-477e-a605-ce16436e4de0 | power on    | active             | False       |
| c5463d94-911b-4465-90a4-c44880ea733c | control_2             | 9c380de8-cb78-46a5-b93b-92b524c26607 | power on    | active             | False       |
| 3a424d6a-19ca-46f1-a195-6b763b0434ff | control_3             | a6ce85f0-e8aa-43fb-b36e-5534e9a6f0fc | power on    | active             | False       |
| 240a7b90-4add-4239-b032-60b5694ce8c4 | contrail-controller_1 | c4dafaf8-6cbe-452e-ad1e-be89d5dabaed | power on    | active             | False       |
| 29458507-eab3-4970-b796-9462e2bcf663 | contrail-controller_2 | edf84e73-5ccf-4235-be5a-f3cca6c40182 | power on    | active             | False       |
| 0f67b07d-910c-4dd5-9dd1-368eb05ff00c | contrail-controller_3 | 717f8bed-7b7e-4491-937e-b4e5179dd40f | power on    | active             | False       |
| 56e55996-5589-4f23-a1b1-670e48591b9e | Compute-1             | 44eeb4bc-420c-42a2-8d9d-6e76335371de | power on    | active             | False       |
| ff47d677-ef5d-4de2-9e4c-c605c4c66d3b | Compute-2             | ee29cc9c-a854-4b77-9f9e-0e90c8433fcd | power on    | active             | False       |
+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+
  2. From the Contrail cloud host(JumpHost), update inventory.yaml and compute-nodes.yaml files in /var/lib/contrail_cloud/config path with new compute node information as contrail user.
    In inventory.yaml, add the new node details 
      - name: "<node-name>"
    pm_addr: "<node-ipmi-address>"
    <<: *hardware1
    mac:
    - "<node-provisioning-interface-mac>"
    Note: hardware1 refers to below details which should already be present in the inventory.yaml file.So, please align above input "hardware1" as per the existing below data in inventory.yaml file. 
        ipmi_hardware1: &hardware1
    pm_type: "ipmi"
    pm_user: "{{ vault['inventory_nodes']['hardware1']['pm_user'] }}"

    pm_password: "{{ vault['inventory_nodes']['hardware1']['pm_password'] }}"
    In compute-nodes.yaml, append the new node details to existing list of compute_nodes_dpdk or compute_nodes_kernel based on type of the node.
    For kernel based compute nodes 
     compute_nodes_kernel:
    - name: <node-name>
      profile: compute-kernel
    For DPDK based compute nodes 
     compute_nodes_dpdk:
    - name: <node-name>
      profile: compute-dpdk
    In the above snippets, compute-kernel and compute-dpdk refers to the hardware profile defined in overcloud-nics.yml and site.yml. These profiles indicate the parameters to be used while deploying the node.
    Note: Moving forward, we will assume we have a DPDK based compute which needs to be added.

  3. Run the inventory-assign.sh script from contrail user and verify it completes successfully. 

      [contrail@cc-jumpost]$ /var/lib/contrail_cloud/scripts/inventory-assign.sh -d
  4. Verify inventory file is loaded and ironic has completed introspection on the newly added nodes. All newly added nodes should show in the output and have Provisioning State as available. 
      (undercloud) [stack@undercloud ~]$ openstack baremetal node list

+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+

| UUID                                 | Name                  | Instance UUID                        | Power State | Provisioning State | Maintenance |

+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+

| 9538f7ba-04d8-4114-994a-5b9beab67b64 | control_1             | 813490f1-8bcf-477e-a605-ce16436e4de0 | power on    | active             | False       |

| c5463d94-911b-4465-90a4-c44880ea733c | control_2             | 9c380de8-cb78-46a5-b93b-92b524c26607 | power on    | active             | False       |

| 3a424d6a-19ca-46f1-a195-6b763b0434ff | control_3             | a6ce85f0-e8aa-43fb-b36e-5534e9a6f0fc | power on    | active             | False       |

| 240a7b90-4add-4239-b032-60b5694ce8c4 | contrail-controller_1 | c4dafaf8-6cbe-452e-ad1e-be89d5dabaed | power on    | active             | False       |

| 29458507-eab3-4970-b796-9462e2bcf663 | contrail-controller_2 | edf84e73-5ccf-4235-be5a-f3cca6c40182 | power on    | active             | False       |

| 0f67b07d-910c-4dd5-9dd1-368eb05ff00c | contrail-controller_3 | 717f8bed-7b7e-4491-937e-b4e5179dd40f | power on    | active             | False       |

| 56e55996-5589-4f23-a1b1-670e48591b9e | Compute-1             | 44eeb4bc-420c-42a2-8d9d-6e76335371de | power on    | active             | False       |

| ff47d677-ef5d-4de2-9e4c-c605c4c66d3b | Compute-2             | ee29cc9c-a854-4b77-9f9e-0e90c8433fcd | power on    | active             | False       |

| ab80e677-ef5d-4ab2-9d4e-c015c4d67xya | New-Compute-3         | None   | power on    | available          | False       |

+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+

  5. Run the compute-nodes-assign.sh script from contrail user and verify it completes successfully.

      [contrail@cc-jumphost]$ /var/lib/contrail_cloud/scripts/compute-nodes-assign.sh -d

  6. Verify compute profile is assigned to baremetal nodes.  #Provide output to these following commands or what is specific to be seen to verify things are good 

      (undercloud) [stack@undercloud ~]$ openstack baremetal node show <node-id> -f json | jq ".properties.capabilities"
"profile:<Selected-compute-Profile-1/2>,cpu_hugepages:true,cpu_txt:true,cpu_vt:true,boot_option:local,cpu_aes:true,cpu_hugepages_1g:true,boot_mode:uefi"

  (undercloud) [stack@undercloud ~]$ openstack overcloud profiles list
+--------------------------------------+-----------------------+-----------------+---------------------------------+-------------------+
| Node UUID                            | Node Name             | Provision State | Current Profile                 | Possible Profiles |
+--------------------------------------+-----------------------+-----------------+---------------------------------+-------------------+
| 9538f7ba-04d8-4114-994a-5b9beab67b64 | control_1             | active          | control                         |                   |
| c5463d94-911b-4465-90a4-c44880ea733c | control_2             | active          | control                         |                   |
| 3a424d6a-19ca-46f1-a195-6b763b0434ff | control_3             | active          | control                         |                   |
| 240a7b90-4add-4239-b032-60b5694ce8c4 | contrail-controller_1 | active          | contrail-controller             |                   |
| 29458507-eab3-4970-b796-9462e2bcf663 | contrail-controller_2 | active          | contrail-controller             |                   |
| 0f67b07d-910c-4dd5-9dd1-368eb05ff00c | contrail-controller_3 | active          | contrail-controller             |                   |
| 56e55996-5589-4f23-a1b1-670e48591b9e | Compute-1             | active          | compute-dpdk                    |                   |
| ff47d677-ef5d-4de2-9e4c-c605c4c66d3b | Compute-2             | active          | compute-dpdk                    |                   |
| ab80e677-ef5d-4ab2-9d4e-c015c4d67xya | <New-Compute>         | available       | compute-dpdk                    |                   |
+--------------------------------------+-----------------------+-----------------+---------------------------------+-------------------+
  7. Run the openstack-deploy.sh script from contrail user and verify it completes successfully. 
      [contrail@cc-jumphost]$ /var/lib/contrail_cloud/scripts/openstack-deploy.sh -d
  8. Verify new compute nodes are deployed in the overcloud along with existing compute nodes.  #if possible add snipped outputs 
      (undercloud) [stack@undercloud ~]$ openstack server list

+--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------------+

| ID                                   | Name                           | Status | Networks               | Image          | Flavor                    |

+--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------------+

| 813490f1-8bcf-477e-a605-ce16436e4de0 | overcloud-controller-0         | ACTIVE | ctlplane=192.168.24.9  | overcloud-full | control                   |

| 9c380de8-cb78-46a5-b93b-92b524c26607 | overcloud-controller-1         | ACTIVE | ctlplane=192.168.24.24 | overcloud-full | control                   |

| a6ce85f0-e8aa-43fb-b36e-5534e9a6f0fc | overcloud-controller-2         | ACTIVE | ctlplane=192.168.24.7  | overcloud-full | control                   |

| c4dafaf8-6cbe-452e-ad1e-be89d5dabaed | overcloud-contrailcontroller-0 | ACTIVE | ctlplane=192.168.24.21 | overcloud-full | contrail-controller       |

| edf84e73-5ccf-4235-be5a-f3cca6c40182 | overcloud-contrailcontroller-1 | ACTIVE | ctlplane=192.168.24.13 | overcloud-full | contrail-controller       |

| 717f8bed-7b7e-4491-937e-b4e5179dd40f | overcloud-contrailcontroller-2 | ACTIVE | ctlplane=192.168.24.19 | overcloud-full | contrail-controller       |

| 225c40ad-2d00-47d2-b8fd-4897ee727aaa | overcloud-contraildpdk-0       | ACTIVE | ctlplane=192.168.24.18 | overcloud-full | compute-dpdk              |

| c4f32036-616a-4152-b902-5687f76355c7 | overcloud-contraildpdk-1       | ACTIVE | ctlplane=192.168.24.11 | overcloud-full | compute-dpdk              |

| <New-Compute-Instance-UUID>    | <New-Compute>         | ACTIVE | ctlplane=192.168.24.15 | overcloud-full | <Selected-compute-Flavor> |

+--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------------+
      
      (overcloud) [stack@undercloud ~]$ openstack hypervisor list

+----+--------------------------------------+-----------------+---------------+-------+

| ID | Hypervisor Hostname                  | Hypervisor Type | Host IP       | State |

+----+--------------------------------------+-----------------+---------------+-------+

|  2 | overcloud-contraildpdk-1.localdomain | QEMU            | 10.1.0.189    | up    |

|  5 | overcloud-contraildpdk-0.localdomain | QEMU            | 10.1.0.47     | up    |

|  6 | <New-Compute>.localdomain | QEMU            | <new-comp-IP> | up    |

+----+--------------------------------------+-----------------+---------------+-------+
      
      (overcloud) [stack@undercloud ~]$ openstack compute service list

+----+------------------+--------------------------------------+-----------------+---------+-------+------------------------------+

| ID | Binary           | Host                                 | Zone            | Status  | State | Updated At                   |

+----+------------------+--------------------------------------+-----------------+---------+-------+------------------------------+

|  2 | nova-scheduler   | overcloud-controller-0.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

|  5 | nova-scheduler   | overcloud-controller-1.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

|  8 | nova-consoleauth | overcloud-controller-0.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 11 | nova-consoleauth | overcloud-controller-1.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 17 | nova-conductor   | overcloud-controller-0.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 26 | nova-scheduler   | overcloud-controller-2.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 29 | nova-conductor   | overcloud-controller-1.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 32 | nova-consoleauth | overcloud-controller-2.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 35 | nova-conductor   | overcloud-controller-2.localdomain   | internal        | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 74 | nova-compute     | overcloud-contraildpdk-1.localdomain | cp1             | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 95 | nova-compute     | overcloud-contraildpdk-0.localdomain | cp0             | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

| 98 | nova-compute     | <New-Compute>.localdomain    | <New-comp-Zone> | enabled | up    | <YYYY-MM-DDTHH:MM:SS.000000> |

+----+------------------+--------------------------------------+-----------------+---------+-------+------------------------------+