This article explains what all to check when NodeNotReady is seen for the atomdb.

kubectl describe pod -n common atom-db-0 shows the messages:

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Warning NodeNotReady 94d node-controller Node is not ready

Warning NodeNotReady 85d node-controller Node is not ready

Warning NodeNotReady 82d node-controller Node is not ready

Warning NodeNotReady 66d node-controller Node is not ready

Check the output of the patroni list to see the status of the atom-db replicas and leader.

kubectl exec -it -n common atom-db-1 -- bash

root@atom-db-1:/home/postgres# patronictl list

+ Cluster: atom-db (7269413560341123141) ------------+-----+-----------+

| Member | Host | Role | State | TL | Lag in MB |

+-----------+---------------+---------+--------------+-----+-----------+

| atom-db-0 | 10.1.2.3 | Replica | running | 175 | 0 |

| atom-db-1 | 10.3.2.1 | Leader | running | 175 | |

| atom-db-2 | 10.2.3.1 | Replica | start failed | | unknown |

+-----------+---------------+---------+--------------+-----+-----------+

Here, you can see the status of the replica for atomdb-2 is in failed state due to which it goes in NodeNotReady state.

To resolve the issue, You can try to perform a reinit for the replicas by running the command: patronictl reinit atom-db atom-db-2 --force

If the status of the atom-db-2 shows as running now, then that means the replica has been restored.

If not, you need to check why it is in this state.

You can also have the healthcheck-monitoring.sh script installed on the system which will take care of reinitialising the replicas and making sure that no lag is seen in between the replicas and the leader. You can refer to KB104041 [juniper.net] to download the healthcheck-monitoring.sh script.