This article explains the steps to run when the internal self generated kubernetes certificates have expired and need to be renewed.

Some of the symptoms include:

• Unable to login to the UI.
• Pods remaining in "Pending" state on the CLI
• No new pods getting scheduled on the system.
• No pods getting recreated on the system.
• Error messages like the following are seen:

kubectl get pods

Unable to connect to the server x509: certificate has expired or is not yet valid

• kubeadm check certs-expiration shows that the date for the certificates has expired.
• kubectl logs -n kube-system kube-scheduler-<pod-name> shows the following error messages:

Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid

For version 23.1 and below:

To check current certificates expiration date, run the following command on each master node(s):

kubeadm certs check-expiration

- sample output
    

    [check-expiration] Reading configuration from the cluster...
    [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    
    CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
    admin.conf                 Dec 13, 2023 13:20 UTC   328d                                    no
    apiserver                  Dec 13, 2023 13:20 UTC   328d            ca                      no
    apiserver-etcd-client      Dec 13, 2023 13:20 UTC   328d            etcd-ca                 no
    apiserver-kubelet-client   Dec 13, 2023 13:20 UTC   328d            ca                      no
    controller-manager.conf    Dec 13, 2023 13:20 UTC   328d                                    no
    etcd-healthcheck-client    Dec 13, 2023 13:20 UTC   328d            etcd-ca                 no
    etcd-peer                  Dec 13, 2023 13:20 UTC   328d            etcd-ca                 no
    etcd-server                Dec 13, 2023 13:20 UTC   328d            etcd-ca                 no
    front-proxy-client         Dec 13, 2023 13:20 UTC   328d            front-proxy-ca          no
    scheduler.conf             Dec 13, 2023 13:20 UTC   328d                                    no
    
    CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
    ca                      Nov 27, 2032 21:31 UTC   9y              no
    etcd-ca                 Nov 27, 2032 21:31 UTC   9y              no
    front-proxy-ca          Nov 27, 2032 21:31 UTC   9y              no



To renew the certificates, run the following steps on each master node(s)

- Renew certificates
    
    kubeadm certs renew all

    
    - sample output
        
      
        [renew] Reading configuration from the cluster...
        [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
        
        certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
        certificate for serving the Kubernetes API renewed
        certificate the apiserver uses to access etcd renewed
        certificate for the API server to connect to kubelet renewed
        certificate embedded in the kubeconfig file for the controller manager to use renewed
        certificate for liveness probes to healthcheck etcd renewed
        certificate for etcd nodes to communicate with each other renewed
        certificate for serving etcd renewed
        certificate for the front proxy client renewed
        certificate embedded in the kubeconfig file for the scheduler manager to use renewed
        
        Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
 
        
- Check the new expiration date
    

    kubeadm certs check-expiration

    
    - sample output
        
        [check-expiration] Reading configuration from the cluster...
        [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
        
        CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
        admin.conf                 Jan 18, 2024 21:40 UTC   364d                                    no
        apiserver                  Jan 18, 2024 21:40 UTC   364d            ca                      no
        apiserver-etcd-client      Jan 18, 2024 21:40 UTC   364d            etcd-ca                 no
        apiserver-kubelet-client   Jan 18, 2024 21:40 UTC   364d            ca                      no
        controller-manager.conf    Jan 18, 2024 21:40 UTC   364d                                    no
        etcd-healthcheck-client    Jan 18, 2024 21:40 UTC   364d            etcd-ca                 no
        etcd-peer                  Jan 18, 2024 21:40 UTC   364d            etcd-ca                 no
        etcd-server                Jan 18, 2024 21:40 UTC   364d            etcd-ca                 no
        front-proxy-client         Jan 18, 2024 21:40 UTC   364d            front-proxy-ca          no
        scheduler.conf             Jan 18, 2024 21:40 UTC   364d                                    no
        
        CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
        ca                      Nov 27, 2032 21:31 UTC   9y              no
        etcd-ca                 Nov 27, 2032 21:31 UTC   9y              no
        front-proxy-ca          Nov 27, 2032 21:31 UTC   9y              no
        

Restart the following pods to use the new certificates


kubectl delete pod -n kube-system -l component=kube-apiserver
kubectl delete pod -n kube-system -l component=kube-scheduler
kubectl delete pod -n kube-system -l component=kube-controller-manager
kubectl delete pod -n kube-system -l component=etcd

Note: If the certificates have already expired and then the renewal is to be done, then you need to reboot all the master nodes as well. 

• If you want to skip node reboot -> As workaround of rebooting the node, we need to completely remove those pod one by one and restoring it by doing::

• Delete or move kube-controller-manager.yaml and kube-scheduler.yaml from /etc/kubernetes/manifests/

• Wait until pod is gone

• Restore the file

For version 23.2 and above:

Check for the expiry of the certs on the master node with the command:

# for i in /var/lib/rancher/rke2/server/tls/*.crt; do echo $i; openssl x509 -in $i -noout -dates; done

Check for the expiry of the certs on the worker node with the command:

# for i in /var/lib/rancher/rke2/agent/*.crt; do echo $i; openssl x509 -in $i -noout -dates; done

By default RKE2 will automatically renew server and client certificate if it is restarted 3 months before cert renewal date.

To restart RKE2

• on master/control-plane node: systemctl restart rke2-server

• on pure worker node: systemctl restart rke2-agent