This document outlines the detailed steps for upgrading the RedHat operating system in Paragon automation.
Summary Table
Step
Action
Description
1
Backup.
Run backup script and confirm successful backup
2
Store Backup
Copy backup to secure, off-cluster location
3
Health Check
Ensure cluster health before proceeding
4
Identify node name/IP
Determine correct node Identifier
5
Isolate cluster
Optional step to isolate secondary cluster from the primary
6
Cordon Node
Prevent new pods from being scheduled
7
Drain Node
Evict pods safely (except DaemonSets)
8
Check Pending Pods
Identify pods waiting to be rescheduled
9
10
Upgrade node
Upgrade OS on cordoned node
11
Reboot node
Perform a safe node reboot
12
Uncordon node
Uncordon updated node
13
Wait for Rebalancing
Wait for pods to be redistributed
14
15
Repeat steps on master3
Repeat steps 6 – 14 on master3
16
Repeat steps on master2
Repeat steps 6 – 14 on master2
17
Check cluster from master2
Run health_check.sh from master2
18
Cordon Node master1
19
Drain Node master1
20
21
22
23
24
25
26
27
Cluster Check
Overall Cluster Check
28
Repeat steps
Repeat entire process on primary cluster
29
Check geoHA setting
Check geoHA setting on primary cluster once normalized.
Steps:
1. Use data.sh --backup and create a backup of cluster data. (see below for syntax and backup report.)
2. Store backup of data offline (on secondary server outside cluster)
3. On master run health_check.sh script to check nodes/pods once cluster is deemed healthy its ok to proceed.
4. To determine use of nodes or IP run kubectl get nodes (this will output needed in the next few steps.)
5. Not necessary but if customer desires they can Isolate secondary cluster by stopping Federated Exchange Bus
6. On the master node run kubectl cordon <nodename/IP> (start with worker1)
[root@rhel-84-node5 ~]# kubectl cordon #######Node IP Addr########
node/######## cordoned
7. On the master node run kubectl drain <nodename/IP> --ignore-daemonsets --grace-period=0 --force --delete-emptydir-data
8. On master run kubectl get po -A -o wide | grep -v Running (this may produce Pending processes on cordon/drained node, this is evident by <none> for IP and Node in the output.
[root@rhel-84-node5 ~]# kubectl get po -A -o wide | grep -v Running
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
common atom-db-1 0/1 Pending 0 8m8s <none> <none> <none> <none>
common kafka-0 0/2 Pending 0 8m4s <none> <none> <none> <none>
fault telemetrymanager-resync-devices-cron-job-29080325-f9dtk 0/1 Completed 0 16m ######## ######## <none> <none>
fault telemetrymanager-resync-faults-cron-job-29080320-xz9h5 0/1 Completed 0 21m ######### ######## <none> <none>
9. On master run health_check.sh script to check nodes/pods once cluster is deemed healthy its ok to proceed.
10. Perform 'yum -y update && yum -y upgrade' to upgrade the OS from 8.4 to 8.10 depending on repos you may need to run upgrade with –nobest
11. Reboot node
12. Once the node comes back up wait about 5 to 10 min typically on master run kubectl uncordon <IP/Node> .
13. Wait for the cluster to balance itself out for about 15 min
14. Run health_check.sh again to determine that all pods are back up and running.
15. Repeat steps 6 – 14 on master3
16. Repeat steps 6 – 14 on master2
17. Login to master 2 and check if cluster is stabilized run health_check.sh
18. On the master2 node run kubectl cordon <nodename/IP> (Final node upgrade master1)
19. On the master2 node run kubectl drain <nodename/IP> --ignore-daemonsets --grace-period=0 --force --delete-emptydir-data
20. On master2 run kubectl get po -A -o wide | grep -v Running (this may produce Pending processes on cordon/drained node, this is evident by <none> for IP and Node in the output.
fault telemetrymanager-resync-devices-cron-job-29080325-f9dtk 0/1 Completed 0 16m ######### ######### <none> <none>
fault telemetrymanager-resync-faults-cron-job-29080320-xz9h5 0/1 Completed 0 21m ######### ######### <none> <none>
21. On master2 run health_check.sh script to check nodes/pods once cluster is deemed healthy its ok to proceed.
22. Perform 'yum -y update && yum -y upgrade' to upgrade the OS from 8.4 to 8.10 depending on repos you may need to run upgrade with –nobest
23. Reboot updated master1 node.
24. Once the node comes back up wait about 5 to 10 min typically on master2 run kubectl uncordon <IP/Node> .
25. Wait for the cluster to balance itself out for about 15 min
26. Run health_check.sh again to determine that all pods are back up and running.
27. Overall cluster check, pods/node/ui/topo/data
28. Repeat same process on primary cluster
29. Once Primary cluster is back check geoHA configuration
kubectl exec -it -n northstar rabbitmq-0 -- rabbitmqctl list_parameters
Login into one of the rabbitmq pod “kubectl exec -it -n northstar rabbitmq-0 bash ”
execute “for i in 0 1 2; do rabbitmqctl federation_status -n rabbit@rabbitmq-$i.rabbitmq-headless.northstar.svc.cluster.local; done”
Node drain output
[root@rhel-84-node5 ~] # kubectl drain ####Node IP Addr##### --ignore-daemonsets --grace-period=0 --force --delete-emptydir-data
node/######### already cordoned
WARNING: ignoring DaemonSet-managed Pods: common/local-volume-provisioner-pgdzw, healthbot/configmanager-4k766, healthbot/ingest-snmp-proxy-pc2sx, healthbot/node-exporter-sw5nk, healthbot/udf-farm-88ndb, kube-system/calico-node-4jfdn, kube-system/docker-registry-docker-registry-proxy-m2rhb, kube-system/kube-proxy-h4kx7, metallb-system/speaker-jhrhj, rook-ceph/csi-cephfsplugin-dvrzp, rook-ceph/csi-rbdplugin-bw2br
evicting pod ambassador/ambassador-859dd596fc-6nm57
evicting pod ems/dmonproxy-547dbcdff-nh6hm
evicting pod kube-system/reloader-reloader-6cd6899bd8-9dq4s
evicting pod rook-ceph/rook-ceph-osd-prepare-172.25.152.27-gq6rb
evicting pod ambassador/ambassador-agent-679579885c-lhkg2
evicting pod healthbot/hbmon-5b89df8966-6dcc2
evicting pod healthbot/kube-state-metrics-87477b87c-6j2lq
evicting pod northstar/bmp-744cb887d5-qs6lg
evicting pod healthbot/influxdb-172-25-152-27-7fdcfbf9b4-7kxjk
evicting pod common/atom-db-1
evicting pod healthbot/inference-engine-5868b8b899-hltrz
evicting pod common/ambassador-auth-jwt-c8c964487-tzrmt
evicting pod northstar/ns-anuta-proxy-5577f7b667-8mjp9
evicting pod northstar/ns-mladapter-6789cb8dc8-796bl
evicting pod northstar/rabbitmq-0
evicting pod metallb-system/controller-7cccff9cfc-kmc9r
evicting pod common/kafka-0
evicting pod northstar/dcscheduler-787f8cf665-vbwtn
evicting pod ems/dcs-559d5679f7-wdvcc
evicting pod northstar/ns-anycastgroup-6f4694c78f-xr2gj
evicting pod northstar/redis-master-68fdffbbc-47nsx
evicting pod rook-ceph/rook-ceph-mon-e-5546f5c497-qwgcn
evicting pod northstar/ns-cmgd-79c4cb8dd5-jnrbc
evicting pod northstar/ns-srt-7c586bbb96-v7tlc
evicting pod northstar/ns-web-84c7f56bcd-cvmcb
evicting pod rook-ceph/rook-ceph-osd-2-66987f6cc6-z2jwd
evicting pod healthbot/rfr-redis-0
evicting pod northstar/ns-celeryscheduler-5c8b95845c-9nq6l
evicting pod rook-ceph/rook-ceph-crashcollector-172.25.152.27-68fdb568d-m4gvq
evicting pod rook-ceph/csi-cephfsplugin-provisioner-5bc5b556f5-4frd2
evicting pod healthbot/license-client-8f684646d-9wsx2
evicting pod healthbot/rfs-redis-746bfcbbc7-wh948
evicting pod northstar/ns-celeryworker-0
evicting pod northstar/ns-pcviewer-bfcbdbc88-rk25v
evicting pod northstar/ns-lsp-intent-6466bdf696-rjtvc
evicting pod northstar/ns-dbutils-6f6f59c9c7-dptgd
evicting pod northstar/ns-ipe-665dbcf856-hpp8q
evicting pod northstar/ns-epe-planner-9f99f7475-789lq
evicting pod northstar/license-util-799cdfb5c5-dcm5k
evicting pod northstar/ns-dpadapter-58b8d996c4-kjt2v
evicting pod northstar/ns-configserver-67ff799bdb-72hdr
evicting pod northstar/ns-pceserver-f8888dc58-6rtzx
pod/ns-srt-7c586bbb96-v7tlc evicted
pod/rook-ceph-mon-e-5546f5c497-qwgcn evicted
pod/redis-master-68fdffbbc-47nsx evicted
pod/ns-cmgd-79c4cb8dd5-jnrbc evicted
pod/rfr-redis-0 evicted
pod/csi-cephfsplugin-provisioner-5bc5b556f5-4frd2 evicted
pod/rook-ceph-osd-2-66987f6cc6-z2jwd evicted
pod/atom-db-1 evicted
pod/ns-web-84c7f56bcd-cvmcb evicted
pod/ns-celeryscheduler-5c8b95845c-9nq6l evicted
pod/rook-ceph-crashcollector-172.25.152.27-68fdb568d-m4gvq evicted
pod/ambassador-agent-679579885c-lhkg2 evicted
pod/license-client-8f684646d-9wsx2 evicted
pod/rfs-redis-746bfcbbc7-wh948 evicted
I0416 12:13:03.153979 576820 request.go:668] Waited for 1.000448598s due to client-side throttling, not priority and fairness, request: POST:https://#########:6443/api/v1/namespaces/ems/pods/dmonproxy-547dbcdff-nh6hm/eviction
pod/dmonproxy-547dbcdff-nh6hm evicted
pod/ambassador-859dd596fc-6nm57 evicted
pod/rook-ceph-osd-prepare-172.25.152.27-gq6rb evicted
pod/ns-celeryworker-0 evicted
pod/ambassador-auth-jwt-c8c964487-tzrmt evicted
pod/ns-pcviewer-bfcbdbc88-rk25v evicted
pod/kube-state-metrics-87477b87c-6j2lq evicted
pod/inference-engine-5868b8b899-hltrz evicted
pod/ns-mladapter-6789cb8dc8-796bl evicted
pod/bmp-744cb887d5-qs6lg evicted
pod/hbmon-5b89df8966-6dcc2 evicted
pod/influxdb-172-25-152-27-7fdcfbf9b4-7kxjk evicted
pod/ns-anuta-proxy-5577f7b667-8mjp9 evicted
pod/ns-lsp-intent-6466bdf696-rjtvc evicted
pod/dcscheduler-787f8cf665-vbwtn evicted
pod/controller-7cccff9cfc-kmc9r evicted
pod/kafka-0 evicted
pod/dcs-559d5679f7-wdvcc evicted
pod/ns-dbutils-6f6f59c9c7-dptgd evicted
pod/reloader-reloader-6cd6899bd8-9dq4s evicted
pod/ns-anycastgroup-6f4694c78f-xr2gj evicted
pod/ns-ipe-665dbcf856-hpp8q evicted
pod/ns-epe-planner-9f99f7475-789lq evicted
pod/license-util-799cdfb5c5-dcm5k evicted
pod/ns-dpadapter-58b8d996c4-kjt2v evicted
pod/ns-configserver-67ff799bdb-72hdr evicted
pod/rabbitmq-0 evicted
pod/ns-pceserver-f8888dc58-6rtzx evicted
node/######### evicted
Backup script Output
[root@rhel-84-node1 ~]# data.sh --backup
Backing up mgd-b7bb4d5c9-8kt9v
Defaulted container "mgd" out of: mgd, init (init), sync (init)
Job(name: db-backup-paa-2025-04-16) creation successful for backup, please track using "kubectl -n common get jobs"
Performing backup please wait.
job.batch/db-backup-paa-2025-04-16 condition met
=====================Backup Report======================
Name: db-backup-paa-2025-04-16
Namespace: common
Selector: controller-uid=2df5343b-2219-4cf3-8a41-b3e5d61e28f5
Labels: apps=db-backup
common=db-backup
id=paa-2025-04-16
Annotations: <none>
Parallelism: 1
Completions: 1
Start Time: Wed, 16 Apr 2025 11:39:26 -0400
Completed At: Wed, 16 Apr 2025 11:39:46 -0400
Duration: 20s
Pods Statuses: 0 Running / 1 Succeeded / 0 Failed
Pod Template:
Labels: app=db-backup
controller-uid=2df5343b-2219-4cf3-8a41-b3e5d61e28f5
job-name=db-backup-paa-2025-04-16
Service Account: db-backup
Containers:
db-backup:
Image: localhost:5000/eng-registry.juniper.net/northstar-scm/northstar-containers/ns_dbinit:release-23-1-ge572e4b914
Port: <none>
Host Port: <none>
Command:
/bin/sh
Args:
-c
exec /entrypoint.sh --backup /paa-2025-04-16
Environment:
PG_HOST: atom-db.common
PG_PORT: 5432
PG_ADMIN_USER: <set to the key 'username' in secret 'atom.atom-db.credentials'> Optional: false
PG_ADMIN_PASS: <set to the key 'password' in secret 'atom.atom-db.credentials'> Optional: false
Mounts:
/opt/northstar/data/backup from postgres-backup (rw)
Volumes:
postgres-backup:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: db-backup-pvc
ReadOnly: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 21s job-controller Created pod: db-backup-paa-2025-04-16-5th4p
Normal Completed 1s job-controller Job completed
======================================================================Running EMS Backup.
=======================Get Backup file location======================
Name: local-pv-ce62658
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/provisioned-by: local-volume-provisioner-172.25.152.20-a34fb91c-353c-42cb-8794-55ab2db13400
Finalizers: [kubernetes.io/pv-protection]
StorageClass: local-storage
Status: Bound
Claim: common/db-backup-pvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 119Gi
Node Affinity:
Required Terms:
Term 0: kubernetes.io/hostname in [#########]
Message:
Source:
Type: LocalVolume (a persistent volume backed by local storage on a node)
Path: /export/local-volumes/pv3 < Location of the backed up files.
Events: <none>
=====================================================================
Running Pathfinder Kubernetes Config Backup.
Saving ns-anuta-rest secret
Saving ns-anuta-rest configmaps ============================Backup Completed==========================
For any issue please contact JTAC support.