This article helps with rook-ceph OSD recovery for failure status

[root@rook-ceph-tools-6844974549-7sqd6 /]# ceph osd status

ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE

0 ##### 518M 199G 0 0 2 106 exists,up

1 ##### 0 0 0 0 0 0 autoout,exists

2 ##### 354M 199G 0 0 0 0 exists,up

3 ##### 531M 199G 0 0 1 0 exists,up

[root@rook-ceph-tools-6844974549-7sqd6 /]# ceph osd df

ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS

3 hdd 0.19530 1.00000 200 GiB 532 MiB 39 MiB 2.5 MiB 490 MiB 199 GiB 0.26 1.14 177 up

1 hdd 0.19530 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down

0 hdd 0.19530 1.00000 200 GiB 519 MiB 39 MiB 2.3 MiB 478 MiB 199 GiB 0.25 1.11 177 up

2 hdd 0.19530 1.00000 200 GiB 354 MiB 39 MiB 5.2 MiB 310 MiB 200 GiB 0.17 0.76 177 up

TOTAL 600 GiB 1.4 GiB 117 MiB 10 MiB 1.2 GiB 599 GiB 0.23

MIN/MAX VAR: 0.76/1.14 STDDEV: 0.04

[root@rook-ceph-tools-6844974549-7sqd6 /]# ceph status

cluster:

id: c01d71a9-aa1f-41b5-bee1-524e2138c857

health: HEALTH_WARN

3 slow ops, oldest one blocked for 853 sec, osd.1 has slow ops

services:

mon: 3 daemons, quorum a,f,g (age 6d)

mgr: a(active, since 6d)

mds: 1/1 daemons up, 1 hot standby

osd: 4 osds: 3 up (since 32h), 3 in (since 32h)

rgw: 1 daemon active (1 hosts, 1 zones)

data:

volumes: 1/1 healthy

pools: 11 pools, 177 pgs

objects: 530 objects, 28 MiB

usage: 1.4 GiB used, 599 GiB / 600 GiB avail

pgs: 177 active+clean

io:

client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

We need to follow the steps as documented - Troubleshoot Issues with Ceph and Rook | Paragon Automation (Pathfinder, Planner, Insights) 24.1 | Juniper Networks 

Note :: We need to ensure to have all pods status in running state for rook-ceph and northstar namespace.

Also, this activity requires downtime of approx 40mins to 1hour.

This activity will restart and impact rook-ceph, rook-ceph operator and DCS pod *(Restart impact).'

Working status Output: *(Without issue)

root@prg-01:~# ceph-tools
[root@rook-ceph-tools-6844974549-7sqd6 /]# ceph osd status
ID  HOST           USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE
 0  10.155.20.14   767M   199G      0        0       2      105   exists,up
 1  10.155.20.13   172M   199G      0        0       0        0   exists,up
 2  10.155.20.16   529M   199G      0        0       0        0   exists,up
 3  10.155.20.12   797M   199G      0        0       1        0   exists,up
[root@rook-ceph-tools-6844974549-7sqd6 /]# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE  VAR   PGS  STATUS
 3    hdd  0.19530   1.00000  200 GiB  798 MiB   38 MiB  3.0 MiB  757 MiB  199 GiB  0.39  1.41  147      up
 1    hdd  0.19530   1.00000  200 GiB  173 MiB   35 MiB   11 KiB  138 MiB  200 GiB  0.08  0.30  136      up
 0    hdd  0.19530   1.00000  200 GiB  768 MiB   41 MiB  4.9 MiB  722 MiB  199 GiB  0.37  1.35  129      up
 2    hdd  0.19530   1.00000  200 GiB  530 MiB   26 MiB  3.1 MiB  500 MiB  199 GiB  0.26  0.93  119      up
                       TOTAL  800 GiB  2.2 GiB  139 MiB   11 MiB  2.1 GiB  798 GiB  0.28
MIN/MAX VAR: 0.30/1.41  STDDEV: 0.12
[root@rook-ceph-tools-6844974549-7sqd6 /]# ceph status
  cluster:
    id:     c01d71a9-aa1f-41b5-bee1-524e2138c857
    health: HEALTH_OK

2024-10-31 : Article Created
2026-02-11 : Updated link to documentation