Use this article to remove a Ceph storage node from a Ceph cluster. This procedure will demonstrate the removal of a storage node from an environment in the context of Contrail Cloud.
For additional Red Hat guidance and reference, see Removing a Node from the Overcloud and Handling a Node Failure .
In the examples used for this procedure, “storage3” will be the targeted node for removal.
Remove the storage node:
Find the connection between the bare metal server and the overcloud server. The output from the command below shows us that the serer we are looking for is “overcloud8st-cephstorageblue1-0”. This information will be used later in the procedure.
(undercloud) [stack@undercloud ~]$ openstack ccloud nodemap list +---------------------------------+----------------+------------+----------------+ | Name | IP | Hypervisor | Hypervisor IP | +---------------------------------+----------------+------------+----------------+ | overcloud8st-cc-2 | 192.168.213.54 | controler2 | 192.168.213.6 | | overcloud8st-cc-1 | 192.168.213.51 | controler3 | 192.168.213.7 | | overcloud8st-cc-0 | 192.168.213.55 | controler1 | 192.168.213.5 | | overcloud8st-ca-1 | 192.168.213.72 | controler1 | 192.168.213.5 | | overcloud8st-ca-0 | 192.168.213.60 | controler3 | 192.168.213.7 | | overcloud8st-ca-2 | 192.168.213.53 | controler2 | 192.168.213.6 | | overcloud8st-cadb-1 | 192.168.213.71 | controler1 | 192.168.213.5 | | overcloud8st-afxctrl-0 | 192.168.213.69 | controler2 | 192.168.213.6 | | overcloud8st-afxctrl-1 | 192.168.213.52 | controler3 | 192.168.213.7 | | overcloud8st-afxctrl-2 | 192.168.213.58 | controler1 | 192.168.213.5 | | overcloud8st-cadb-0 | 192.168.213.65 | controler2 | 192.168.213.6 | | overcloud8st-ctrl-0 | 192.168.213.73 | controler2 | 192.168.213.6 | | overcloud8st-ctrl-1 | 192.168.213.63 | controler1 | 192.168.213.5 | | overcloud8st-ctrl-2 | 192.168.213.59 | controler3 | 192.168.213.7 | | overcloud8st-cephstorageblue1-0 | 192.168.213.62 | storage3 | 192.168.213.62 | | overcloud8st-compdpdk-0 | 192.168.213.56 | compute1 | 192.168.213.56 | | overcloud8st-cephstorageblue2-0 | 192.168.213.61 | storage2 | 192.168.213.61 | | overcloud8st-cephstorageblue2-1 | 192.168.213.80 | storage1 | 192.168.213.80 | | overcloud8st-cadb-2 | 192.168.213.74 | controler3 | 192.168.213.7 | +---------------------------------+----------------+------------+----------------+
Log in as root user to any of the openstack controllers and verify that the Ceph cluster is healthy:
[root@overcloud8st-ctrl-1 ~]# ceph -s cluster: id: a98b1580-bb97-11ea-9f2b-525400882160 health: HEALTH_OK
Find the OSDs that reside on the server to be removed (overcloud8st-cephstorageblue1-0). We identify osd.2, osd.3, osd.6, and osd.7 from the example below:
[root@overcloud8st-ctrl-1 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 10.91638 root default -3 3.63879 host overcloud8st-cephstorageblue1-0 2 hdd 0.90970 osd.2 up 1.00000 1.00000 3 hdd 0.90970 osd.3 up 1.00000 1.00000 6 hdd 0.90970 osd.6 up 1.00000 1.00000 7 hdd 0.90970 osd.7 up 1.00000 1.00000 -7 3.63879 host overcloud8st-cephstorageblue2-0 1 hdd 0.90970 osd.1 up 1.00000 1.00000 4 hdd 0.90970 osd.4 up 1.00000 1.00000 8 hdd 0.90970 osd.8 up 1.00000 1.00000 10 hdd 0.90970 osd.10 up 1.00000 1.00000 -5 3.63879 host overcloud8st-cephstorageblue2-1 0 hdd 0.90970 osd.0 up 1.00000 1.00000 5 hdd 0.90970 osd.5 up 1.00000 1.00000 9 hdd 0.90970 osd.9 up 1.00000 1.00000 11 hdd 0.90970 osd.11 up 1.00000 1.00000
Remove osd.2, osd.3, osd.6, and osd.7 while still logged in to the openstack controller:
[root@overcloud8st-ctrl-1 ~]# ceph osd out 2 marked out osd.2. [root@overcloud8st-ctrl-1 ~]# ceph osd out 3 marked out osd.3. [root@overcloud8st-ctrl-1 ~]# ceph osd out 6 marked out osd.6. [root@overcloud8st-ctrl-1 ~]# ceph osd out 7 marked out osd.7.
Moving forward, check to see that the cluster health returns to “health_ok” state by using the ceph -s command. Check for cluster health after every operation.
Log in to the Ceph node (overcloud8st-cephstorageblue1-0) to stop the OSD services:
[root@overcloud8st-cephstorageblue1-0 ~]# systemctl stop [email protected] [root@overcloud8st-cephstorageblue1-0 ~]# systemctl stop [email protected] [root@overcloud8st-cephstorageblue1-0 ~]# systemctl stop [email protected] [root@overcloud8st-cephstorageblue1-0 ~]# systemctl stop [email protected]
Log back into the controller and remove further information about the OSDs from overcloud8st-cephstorageblue1-0:
[root@overcloud8st-ctrl-1 ~]# ceph osd crush remove osd.2 removed item id 2 name 'osd.2' from crush map [root@overcloud8st-ctrl-1 ~]# ceph osd crush remove osd.3 removed item id 3 name 'osd.3' from crush map [root@overcloud8st-ctrl-1 ~]# ceph osd crush remove osd.6 removed item id 6 name 'osd.6' from crush map [root@overcloud8st-ctrl-1 ~]# ceph osd crush remove osd.7 removed item id 7 name 'osd.7' from crush map
[root@overcloud8st-ctrl-1 ~]# ceph auth del osd.2 updated [root@overcloud8st-ctrl-1 ~]# ceph auth del osd.3 updated [root@overcloud8st-ctrl-1 ~]# ceph auth del osd.6 updated [root@overcloud8st-ctrl-1 ~]# ceph auth del osd.7 updated
[root@overcloud8st-ctrl-1 ~]# ceph osd rm 2 removed osd.2 [root@overcloud8st-ctrl-1 ~]# ceph osd rm 3 removed osd.3 [root@overcloud8st-ctrl-1 ~]# ceph osd rm 6 removed osd.6 [root@overcloud8st-ctrl-1 ~]# ceph osd rm 7 removed osd.7
[root@overcloud8st-ctrl-1 ~]# ceph osd crush rm overcloud8st-cephstorageblue1-0
From the undercloud VM find the ID of the Ceph storage node:
(undercloud) [stack@undercloud ~]$ openstack server list | grep overcloud8st-cephstorageblue1-0 | 7ee9be4f-efda-4837-a597-a6554027d0c9 | overcloud8st-cephstorageblue1-0 | ACTIVE | ctlplane=192.168.213.62 | overcloud-full | CephStorageBlue1
Initiate a removal using the node ID from the previous step:
(undercloud) [stack@undercloud ~]$ openstack overcloud node delete --stack overcloud 7ee9be4f-efda-4837-a597-a6554027d0c9
Verify that the bare metal node is in a state of power off and available:
(undercloud) [stack@undercloud ~]$ openstack baremetal node list | grep storage3 | 05bbab4b-b968-4d1d-87bc-a26ac335303d | storage3 | None | power off | available | False |
From the jump host as the contrail user mark the storage node with ‘status: deleting’ so the Ceph profile will be removed from it. Do this by adding the ‘status: deleting’ to the storage-nodes.yml file for storage3 and then run the script storage-nodes-assign.sh.
[contrail@5a6s13-node1 contrail_cloud]$ cat config/storage-nodes.yml storage_nodes: - name: storage1 profile: blue2 - name: storage2 profile: blue2 - name: storage3 profile: blue1 status: deleting
[contrail@5a6s13-node1 contrail_cloud]$ ./scripts/storage-nodes-assign.sh
Run openstack-deploy.sh to regenerate the templates to reflect the current state:
[contrail@5a6s13-node1 contrail_cloud]$ ./scripts/openstack-deploy.sh
If the goal is to remove the bare metal node completely, the ‘status: deleting’ can be assigned in the inventory.yml file. Information about the bare metal will be removed from the storage-nodes.yml file:
[contrail@5a6s13-node1 contrail_cloud]$ cat config/inventory.yml ... inventory_nodes: - name: "storage3" pm_addr: "10.84.129.184" status: deleting <<: *common [contrail@5a6s13-node1 contrail_cloud]$ ./scripts/inventory-assign.sh [contrail@5a6s13-node1 contrail_cloud]$ cat config/storage-nodes.yml storage_nodes: - name: storage1 profile: blue2 - name: storage2 profile: blue2</pre>