Description

This KB describes procedure to clear mnesia db for rabbitmq in tf-operator style contrail deployments. This procedure differs from the traditional procedure used in Ansible deployments or RHOSP style deployments.

Symptoms

There can be multiple reasons for clearing mnesia db which can be due to problems in other components where if any microservice is not consuming data as it should be due to which queue for rabbitmq can get filled up and get stuck in some cases.

[dcn contrail@cc-jumphost:~]$ kubectl -n tf exec rabbitmq1-rabbitmq-statefulset-0 -- sh -c '. /etc/rabbitmq/rabbitmq-common.env ; rabbitmqctl list_queues --silent --vhost $(rabbitmqctl list_vhosts --silent)' | awk '$2 != "0"'
vnc_config.controller0-k8s-cc.internalapi.dcn.opencontrail.net-8082-0  17
schema_transformer.controller2-k8s-cc.internalapi.dcn.opencontrail.net 33
-contrail-cloud-k8s-kube_manager.controller2-k8s-ca.internalapi.dcn.opencontrail.net   93
-contrail-cloud-k8s-kube_manager.controller1-k8s-ca.internalapi.dcn.opencontrail.net   124
svc_monitor.controller0-k8s-cc.internalapi.dcn.opencontrail.net        3

Solution

Below are pods for contrail microservices deployed via tf-operator style deployment

[dcn contrail@cc-jumphost:~]$ kubectl get pods -n tf
NAME                                           READY   STATUS    RESTARTS   AGE
analytics1-analytics-statefulset-0             4/4     Running   4          55d
analytics1-analytics-statefulset-1             4/4     Running   4          55d
analytics1-analytics-statefulset-2             4/4     Running   8          55d
analyticsalarm1-analyticsalarm-statefulset-0   4/4     Running   4          55d
analyticsalarm1-analyticsalarm-statefulset-1   4/4     Running   4          55d
analyticsalarm1-analyticsalarm-statefulset-2   4/4     Running   8          55d
analyticsdb1-cassandra-statefulset-0           3/3     Running   3          55d
analyticsdb1-cassandra-statefulset-1           3/3     Running   6          55d
analyticsdb1-cassandra-statefulset-2           3/3     Running   3          55d
analyticssnmp1-analyticssnmp-statefulset-0     4/4     Running   8          55d
analyticssnmp1-analyticssnmp-statefulset-1     4/4     Running   4          55d
analyticssnmp1-analyticssnmp-statefulset-2     4/4     Running   4          55d
config1-config-statefulset-0                   7/7     Running   0          52m
config1-config-statefulset-1                   7/7     Running   0          52m
config1-config-statefulset-2                   7/7     Running   0          53m
configdb1-cassandra-statefulset-0              3/3     Running   7          55d
configdb1-cassandra-statefulset-1              3/3     Running   9          55d
configdb1-cassandra-statefulset-2              3/3     Running   9          55d
control1-control-statefulset-0                 5/5     Running   15         55d
control1-control-statefulset-1                 5/5     Running   11         55d
control1-control-statefulset-2                 5/5     Running   15         55d
kubemanager1-kubemanager-statefulset-0         1/1     Running   0          23s
kubemanager1-kubemanager-statefulset-1         1/1     Running   0          26s
kubemanager1-kubemanager-statefulset-2         1/1     Running   0          31s
queryengine1-queryengine-statefulset-0         1/1     Running   2          55d
queryengine1-queryengine-statefulset-1         1/1     Running   1          55d
queryengine1-queryengine-statefulset-2         1/1     Running   1          55d
rabbitmq1-rabbitmq-statefulset-0               1/1     Running   0          55m
rabbitmq1-rabbitmq-statefulset-1               1/1     Running   0          55m
rabbitmq1-rabbitmq-statefulset-2               1/1     Running   0          55m
redis1-redis-statefulset-0                     2/2     Running   2          55d
redis1-redis-statefulset-1                     2/2     Running   4          55d
redis1-redis-statefulset-2                     2/2     Running   2          55d
tf-operator-646fb557f-r4r4w                    1/1     Running   2          55d
vrouter1-vrouter-daemonset-9s4xb               3/3     Running   9          55d
vrouter1-vrouter-daemonset-f6v97               3/3     Running   6          55d
vrouter1-vrouter-daemonset-fnb8s               3/3     Running   3          55d
vrouter1-vrouter-daemonset-nrk9t               3/3     Running   6          55d
vrouter1-vrouter-daemonset-pksrt               3/3     Running   9          55d
vrouter1-vrouter-daemonset-z927z               3/3     Running   3          55d
webui1-webui-statefulset-0                     2/2     Running   2          12d
webui1-webui-statefulset-1                     2/2     Running   4          12d
webui1-webui-statefulset-2                     2/2     Running   2          12d
zookeeper1-zookeeper-statefulset-0             1/1     Running   3          55d
zookeeper1-zookeeper-statefulset-1             1/1     Running   3          55d
zookeeper1-zookeeper-statefulset-2             1/1     Running   2          55d

NOTE: Mnesia is a distributed database that RabbitMQ uses to store information about users, exchanges, queues, and bindings. DB cannot be deleted without stopping containers running rabbitmq which actively use mnesia db. In tf-operator style deployment, pods cannot be killed or stopped.

In order to stop rabbitmq we need to stop tf-operator and kubemanager pods and then proceed to rabbitmq pod.

tf-operator pod is configured as deployment

[dcn contrail@cc-jumphost:~]$ kubectl get deployments.apps -n tf
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
tf-operator   1/1     1            1           56d

Scale down the deployment for tf-operator to 0 this would kill tf-operator pod

[dcn contrail@cc-jumphost:~]$ kubectl -n tf scale deployments tf-operator --replicas=0
deployment.apps/tf-operator scaled

Once tf-operator pod is terminated we can verify via getting all pods and looking for tf-operator, here pod is not present which means its already in terminated state

[dcn contrail@cc-jumphost:~]$ kubectl get pods -n tf
NAME                                           READY   STATUS    RESTARTS   AGE
analytics1-analytics-statefulset-0             4/4     Running   4          56d
analytics1-analytics-statefulset-1             4/4     Running   4          56d
analytics1-analytics-statefulset-2             4/4     Running   8          56d
analyticsalarm1-analyticsalarm-statefulset-0   4/4     Running   4          56d
analyticsalarm1-analyticsalarm-statefulset-1   4/4     Running   4          56d
analyticsalarm1-analyticsalarm-statefulset-2   4/4     Running   8          56d
analyticsdb1-cassandra-statefulset-0           3/3     Running   3          56d
analyticsdb1-cassandra-statefulset-1           3/3     Running   6          56d
analyticsdb1-cassandra-statefulset-2           3/3     Running   3          56d
analyticssnmp1-analyticssnmp-statefulset-0     4/4     Running   8          56d
analyticssnmp1-analyticssnmp-statefulset-1     4/4     Running   4          56d
analyticssnmp1-analyticssnmp-statefulset-2     4/4     Running   4          56d
config1-config-statefulset-0                   7/7     Running   0          15h
config1-config-statefulset-1                   7/7     Running   0          15h
config1-config-statefulset-2                   7/7     Running   0          15h
configdb1-cassandra-statefulset-0              3/3     Running   7          56d
configdb1-cassandra-statefulset-1              3/3     Running   9          56d
configdb1-cassandra-statefulset-2              3/3     Running   9          56d
control1-control-statefulset-0                 5/5     Running   15         56d
control1-control-statefulset-1                 5/5     Running   11         56d
control1-control-statefulset-2                 5/5     Running   15         56d
kubemanager1-kubemanager-statefulset-0         1/1     Running   0          23h
kubemanager1-kubemanager-statefulset-1         1/1     Running   0          23h
kubemanager1-kubemanager-statefulset-2         1/1     Running   0          23h
queryengine1-queryengine-statefulset-0         1/1     Running   2          56d
queryengine1-queryengine-statefulset-1         1/1     Running   1          56d
queryengine1-queryengine-statefulset-2         1/1     Running   1          56d
rabbitmq1-rabbitmq-statefulset-0               1/1     Running   0          24h
rabbitmq1-rabbitmq-statefulset-1               1/1     Running   0          24h
rabbitmq1-rabbitmq-statefulset-2               1/1     Running   0          24h
redis1-redis-statefulset-0                     2/2     Running   2          56d
redis1-redis-statefulset-1                     2/2     Running   4          56d
redis1-redis-statefulset-2                     2/2     Running   2          56d
vrouter1-vrouter-daemonset-9s4xb               3/3     Running   9          56d
vrouter1-vrouter-daemonset-f6v97               3/3     Running   6          56d
vrouter1-vrouter-daemonset-fnb8s               3/3     Running   3          56d
vrouter1-vrouter-daemonset-nrk9t               3/3     Running   6          56d
vrouter1-vrouter-daemonset-pksrt               3/3     Running   9          56d
vrouter1-vrouter-daemonset-z927z               3/3     Running   3          56d
webui1-webui-statefulset-0                     2/2     Running   2          13d
webui1-webui-statefulset-1                     2/2     Running   4          13d
webui1-webui-statefulset-2                     2/2     Running   2          13d
zookeeper1-zookeeper-statefulset-0             1/1     Running   3          56d
zookeeper1-zookeeper-statefulset-1             1/1     Running   3          56d
zookeeper1-zookeeper-statefulset-2             1/1     Running   2          56d

Scale down the kubeManager pods to 0 as shown below, kubeManager is deployed as a statefulset, kubeManager pod can be seen as terminating state in below snippets

[dcn contrail@cc-jumphost:~]$ kubectl -n tf scale statefulset/kubemanager1-kubemanager-statefulset --replicas=0
statefulset.apps/kubemanager1-kubemanager-statefulset scaled

Below two kubeManager pods are in terminating state and one of pod is already in terminated so its not present in list

[dcn contrail@cc-jumphost:~]$ kubectl get pods -n tf
NAME                                           READY   STATUS        RESTARTS   AGE
analytics1-analytics-statefulset-0             4/4     Running       4          56d
analytics1-analytics-statefulset-1             4/4     Running       4          56d
analytics1-analytics-statefulset-2             4/4     Running       8          56d
analyticsalarm1-analyticsalarm-statefulset-0   4/4     Running       4          56d
analyticsalarm1-analyticsalarm-statefulset-1   4/4     Running       4          56d
analyticsalarm1-analyticsalarm-statefulset-2   4/4     Running       8          56d
analyticsdb1-cassandra-statefulset-0           3/3     Running       3          56d
analyticsdb1-cassandra-statefulset-1           3/3     Running       6          56d
analyticsdb1-cassandra-statefulset-2           3/3     Running       3          56d
analyticssnmp1-analyticssnmp-statefulset-0     4/4     Running       8          56d
analyticssnmp1-analyticssnmp-statefulset-1     4/4     Running       4          56d
analyticssnmp1-analyticssnmp-statefulset-2     4/4     Running       4          56d
config1-config-statefulset-0                   7/7     Running       0          15h
config1-config-statefulset-1                   7/7     Running       0          15h
config1-config-statefulset-2                   7/7     Running       0          15h
configdb1-cassandra-statefulset-0              3/3     Running       7          56d
configdb1-cassandra-statefulset-1              3/3     Running       9          56d
configdb1-cassandra-statefulset-2              3/3     Running       9          56d
control1-control-statefulset-0                 5/5     Running       15         56d
control1-control-statefulset-1                 5/5     Running       11         56d
control1-control-statefulset-2                 5/5     Running       15         56d
kubemanager1-kubemanager-statefulset-1         1/1     Terminating   0          23h
kubemanager1-kubemanager-statefulset-2         1/1     Terminating   0          23h
queryengine1-queryengine-statefulset-0         1/1     Running       2          56d
queryengine1-queryengine-statefulset-1         1/1     Running       1          56d
queryengine1-queryengine-statefulset-2         1/1     Running       1          56d
rabbitmq1-rabbitmq-statefulset-0               1/1     Running       0          24h
rabbitmq1-rabbitmq-statefulset-1               1/1     Running       0          24h
rabbitmq1-rabbitmq-statefulset-2               1/1     Running       0          24h
redis1-redis-statefulset-0                     2/2     Running       2          56d
redis1-redis-statefulset-1                     2/2     Running       4          56d
redis1-redis-statefulset-2                     2/2     Running       2          56d
vrouter1-vrouter-daemonset-9s4xb               3/3     Running       9          56d
vrouter1-vrouter-daemonset-f6v97               3/3     Running       6          56d
vrouter1-vrouter-daemonset-fnb8s               3/3     Running       3          56d
vrouter1-vrouter-daemonset-nrk9t               3/3     Running       6          56d
vrouter1-vrouter-daemonset-pksrt               3/3     Running       9          56d
vrouter1-vrouter-daemonset-z927z               3/3     Running       3          56d
webui1-webui-statefulset-0                     2/2     Running       2          13d
webui1-webui-statefulset-1                     2/2     Running       4          13d
webui1-webui-statefulset-2                     2/2     Running       2          13d
zookeeper1-zookeeper-statefulset-0             1/1     Running       3          56d
zookeeper1-zookeeper-statefulset-1             1/1     Running       3          56d
zookeeper1-zookeeper-statefulset-2             1/1     Running       2          56d
[dcn contrail@cc-jumphost:~]$ 

Verify the mounts for rabbitmq pod, rabbitmq-data is the directory which stores mnesia db which is mounted on /var/lib/rabbitmq

[dcn contrail@cc-jumphost:~]$ kubectl describe  pods -n tf rabbitmq1-rabbitmq-statefulset-0 
Name:                 rabbitmq1-rabbitmq-statefulset-0
Namespace:            tf
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 controller1-k8s-cc.internalapi/100.78.168.22
Start Time:           Wed, 10 Apr 2024 10:37:19 +0200
Labels:               controller-revision-hash=rabbitmq1-rabbitmq-statefulset-569b9cc868
                      rabbitmq=rabbitmq1
                      statefulset.kubernetes.io/pod-name=rabbitmq1-rabbitmq-statefulset-0
                      tf_manager=rabbitmq
Annotations:          hostname: controller1-k8s-cc.internalapi.dcn.opencontrail.net
                      kubectl.kubernetes.io/restartedAt: 2024-04-10T10:36:16+02:00
Status:               Running
IP:                   100.78.168.22
IPs:
  IP:           100.78.168.22
Controlled By:  StatefulSet/rabbitmq1-rabbitmq-statefulset
Containers:
  rabbitmq:
    Container ID:  cri-o://de82207d91cbd7e3aac700ffb887d72670dca709d33d550140da887ade4994fd
    Image:         100.78.148.100:9798/contrail-nightly/contrail-external-rabbitmq:21.4.L2.329
    Image ID:      100.78.148.100:9798/contrail-nightly/contrail-external-rabbitmq@sha256:8e132e036fcb7c468877a2126c999873e15eae76460cf462475189708c9de0ff
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      /etc/contrailconfigmaps/run-rabbitmq.sh
    State:          Running
      Started:      Wed, 10 Apr 2024 10:37:19 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      POD_IP:      (v1:status.podIP)
      POD_NAME:   rabbitmq1-rabbitmq-statefulset-0 (v1:metadata.name)
      NODE_TYPE:  config-database
      LOG_LEVEL:  SYS_DEBUG
    Mounts:
      /etc/certificates from rabbitmq1-secret-certificates (ro)
      /etc/contrailconfigmaps from rabbitmq1-rabbitmq-volume (rw)
      /etc/hosts from etc-hosts (ro)
      /etc/localtime from etc-localtime (ro)
      /etc/resolv.conf from etc-resolv (ro)
      /etc/ssl/certs/kubernetes from ca-certs (ro)
      /etc/timezone from etc-timezone (ro)
      /var/crashes from var-crashes (rw)
      /var/lib/rabbitmq from rabbitmq-data (rw)
      /var/log/rabbitmq from rabbitmq-logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l46gm (ro)
      
Scale down the rabbitmq stateful set

[dcn contrail@cc-jumphost:~]$ kubectl -n tf scale statefulset rabbitmq1-rabbitmq-statefulset --replicas=0
statefulset.apps/rabbitmq1-rabbitmq-statefulset scaled

rabbitmq pods can be seen as terminating state in below outputs

[dcn contrail@cc-jumphost:~]$ kubectl get pods -n tf
NAME                                           READY   STATUS        RESTARTS   AGE
analytics1-analytics-statefulset-0             4/4     Running       4          56d
analytics1-analytics-statefulset-1             4/4     Running       4          56d
analytics1-analytics-statefulset-2             4/4     Running       8          56d
analyticsalarm1-analyticsalarm-statefulset-0   4/4     Running       4          56d
analyticsalarm1-analyticsalarm-statefulset-1   4/4     Running       4          56d
analyticsalarm1-analyticsalarm-statefulset-2   4/4     Running       8          56d
analyticsdb1-cassandra-statefulset-0           3/3     Running       3          56d
analyticsdb1-cassandra-statefulset-1           3/3     Running       6          56d
analyticsdb1-cassandra-statefulset-2           3/3     Running       3          56d
analyticssnmp1-analyticssnmp-statefulset-0     4/4     Running       8          56d
analyticssnmp1-analyticssnmp-statefulset-1     4/4     Running       4          56d
analyticssnmp1-analyticssnmp-statefulset-2     4/4     Running       4          56d
config1-config-statefulset-0                   7/7     Running       0          16h
config1-config-statefulset-1                   7/7     Running       0          16h
config1-config-statefulset-2                   7/7     Running       0          16h
configdb1-cassandra-statefulset-0              3/3     Running       7          56d
configdb1-cassandra-statefulset-1              3/3     Running       9          56d
configdb1-cassandra-statefulset-2              3/3     Running       9          56d
control1-control-statefulset-0                 5/5     Running       15         56d
control1-control-statefulset-1                 5/5     Running       11         56d
control1-control-statefulset-2                 5/5     Running       15         56d
queryengine1-queryengine-statefulset-0         1/1     Running       2          56d
queryengine1-queryengine-statefulset-1         1/1     Running       1          56d
queryengine1-queryengine-statefulset-2         1/1     Running       1          56d
rabbitmq1-rabbitmq-statefulset-0               1/1     Terminating   0          25h
rabbitmq1-rabbitmq-statefulset-1               1/1     Terminating   0          25h
rabbitmq1-rabbitmq-statefulset-2               1/1     Terminating   0          25h
redis1-redis-statefulset-0                     2/2     Running       2          56d
redis1-redis-statefulset-1                     2/2     Running       4          56d
redis1-redis-statefulset-2                     2/2     Running       2          56d
vrouter1-vrouter-daemonset-9s4xb               3/3     Running       9          56d
vrouter1-vrouter-daemonset-f6v97               3/3     Running       6          56d
vrouter1-vrouter-daemonset-fnb8s               3/3     Running       3          56d
vrouter1-vrouter-daemonset-nrk9t               3/3     Running       6          56d
vrouter1-vrouter-daemonset-pksrt               3/3     Running       9          56d
vrouter1-vrouter-daemonset-z927z               3/3     Running       3          56d
webui1-webui-statefulset-0                     2/2     Running       2          13d
webui1-webui-statefulset-1                     2/2     Running       4          13d
webui1-webui-statefulset-2                     2/2     Running       2          13d
zookeeper1-zookeeper-statefulset-0             1/1     Running       3          56d
zookeeper1-zookeeper-statefulset-1             1/1     Running       3          56d
zookeeper1-zookeeper-statefulset-2             1/1     Running       2          56d      

Once rabbitmq pods are terminated, login to worker nodes where pods are hosted and navigate to rabbitmq-data directory which is mounted on /var/lib/contrail/rabbitmq/

Pod hosted on worker nodes in cluster can be checked via below command

[dcn contrail@cc-jumphost:~]$ kubectl get pods -ntf -o wide | grep rabbitmq
rabbitmq1-rabbitmq-statefulset-0               1/1     Running   0          38m   100.78.168.22   controller1-k8s-cc.internalapi  <none>           <none>
rabbitmq1-rabbitmq-statefulset-1               1/1     Running   0          38m   100.78.168.21   controller0-k8s-cc.internalapi   <none>           <none>
rabbitmq1-rabbitmq-statefulset-2               1/1     Running   0          39m   100.78.168.23   controller2-k8s-cc.internalapi   <none>           <none>

Delete contents of mnesia on all worker nodes which are hosting rabbitmq pods

controller0
[root@controller0-k8s-cc contrail]# cd cd /var/lib/contrail/rabbitmq/
[root@controller0-k8s-cc rabbitmq]# ls -la
total 4
drwxr-xr-x. 3 polkitd unbound  42 Feb 14 15:21 .
drwxr-xr-x. 9 root    root    121 Feb 14 15:19 ..
-rw-------. 1 polkitd unbound  37 Apr 10 10:37 .erlang.cookie
drwxr-xr-x. 4 polkitd unbound 260 Apr 10 10:38 mnesia
[root@controller0-k8s-cc rabbitmq]# ls -la mnesia/
total 8
drwxr-xr-x. 4 polkitd unbound  260 Apr 10 10:38 .
drwxr-xr-x. 3 polkitd unbound   42 Feb 14 15:21 ..
drwxr-xr-x. 5 polkitd unbound 4096 Apr 11 11:41 [email protected]
-rw-r--r--. 1 polkitd unbound  194 Apr 10 10:38 rabbit@controller0-k8s-cc.internalapi.dcn.opencontrail.net-feature_flags
drwxr-xr-x. 2 polkitd unbound    6 Apr 10 10:38 rabbit@controller0-k8s-cc.internalapi.dcn.opencontrail.net-plugins-expand
[root@controller0-k8s-cc rabbitmq]# rm -rf mnesia/*
[root@controller0-k8s-cc rabbitmq]# ls -la mnesia/
total 0
drwxr-xr-x. 2 polkitd unbound  6 Apr 11 11:49 .
drwxr-xr-x. 3 polkitd unbound 42 Feb 14 15:21 ..


controller1
[contrail@controller1-k8s-cc ~]$ cd /var/lib/contrail/rabbitmq/
[contrail@controller1-k8s-cc rabbitmq]$ ls -la
total 4
drwxr-xr-x. 3 polkitd unbound  42 Feb 14 15:21 .
drwxr-xr-x. 9 root    root    121 Feb 14 15:19 ..
-rw-------. 1 polkitd unbound  37 Apr 10 10:37 .erlang.cookie
drwxr-xr-x. 4 polkitd unbound 260 Apr 10 10:37 mnesia
[contrail@controller1-k8s-cc rabbitmq]$ ls -la mnesia/
total 8
drwxr-xr-x. 4 polkitd unbound  260 Apr 10 10:37 .
drwxr-xr-x. 3 polkitd unbound   42 Feb 14 15:21 ..
drwxr-xr-x. 5 polkitd unbound 4096 Apr 11 11:40 [email protected]
-rw-r--r--. 1 polkitd unbound  194 Apr 10 10:37 rabbit@controller1-k8s-cc.internalapi.dcn.opencontrail.net-feature_flags
drwxr-xr-x. 2 polkitd unbound    6 Apr 10 10:37 rabbit@controller1-k8s-cc.internalapi.dcn.opencontrail.net-plugins-expand
[root@controller1-k8s-cc rabbitmq]# rm -rf mnesia/*
[root@controller1-k8s-cc rabbitmq]# ls -la mnesia/
total 0
drwxr-xr-x. 2 polkitd unbound  6 Apr 11 11:50 .
drwxr-xr-x. 3 polkitd unbound 42 Feb 14 15:21 ..


controller2
[root@controller2-k8s-cc contrail]# cd /var/lib/contrail/rabbitmq/
[root@controller2-k8s-cc rabbitmq]# ls -la
total 4
drwxr-xr-x. 3 polkitd unbound  42 Feb 14 15:21 .
drwxr-xr-x. 9 root    root    121 Feb 14 15:19 ..
-rw-------. 1 polkitd unbound  37 Apr 10 10:36 .erlang.cookie
drwxr-xr-x. 4 polkitd unbound 260 Apr 10 10:37 mnesia
[root@controller2-k8s-cc rabbitmq]# ls -la mnesia/
total 8
drwxr-xr-x. 4 polkitd unbound  260 Apr 10 10:37 .
drwxr-xr-x. 3 polkitd unbound   42 Feb 14 15:21 ..
drwxr-xr-x. 5 polkitd unbound 4096 Apr 11 11:48 [email protected]
-rw-r--r--. 1 polkitd unbound  170 Apr 10 10:37 rabbit@controller2-k8s-cc.internalapi.dcn.opencontrail.net-feature_flags
drwxr-xr-x. 2 polkitd unbound    6 Apr 10 10:37 rabbit@controller2-k8s-cc.internalapi.dcn.opencontrail.net-plugins-expand
[root@controller2-k8s-cc rabbitmq]# rm -rf mnesia/*
[root@controller2-k8s-cc rabbitmq]# 
[root@controller2-k8s-cc rabbitmq]# 
[root@controller2-k8s-cc rabbitmq]# 
[root@controller2-k8s-cc rabbitmq]# ls -la mnesia/
total 0
drwxr-xr-x. 2 polkitd unbound  6 Apr 11 11:50 .
drwxr-xr-x. 3 polkitd unbound 42 Feb 14 15:21 ..
[root@controller2-k8s-cc rabbitmq]# exit
exit
[contrail@controller2-k8s-cc ~]$ 

After mnesia directory is cleared on all nodes, tf-operator needs to be scaled back

[dcn contrail@cc-jumphost:~]$ kubectl -n tf scale deployment tf-operator --replicas=1
deployment.apps/tf-operator scaled

[dcn contrail@cc-jumphost:~]$ kubectl get pods -n tf
NAME                                           READY   STATUS    RESTARTS   AGE
analytics1-analytics-statefulset-0             4/4     Running   4          56d
analytics1-analytics-statefulset-1             4/4     Running   4          56d
analytics1-analytics-statefulset-2             4/4     Running   8          56d
analyticsalarm1-analyticsalarm-statefulset-0   4/4     Running   4          56d
analyticsalarm1-analyticsalarm-statefulset-1   4/4     Running   4          56d
analyticsalarm1-analyticsalarm-statefulset-2   4/4     Running   8          56d
analyticsdb1-cassandra-statefulset-0           3/3     Running   3          56d
analyticsdb1-cassandra-statefulset-1           3/3     Running   6          56d
analyticsdb1-cassandra-statefulset-2           3/3     Running   3          56d
analyticssnmp1-analyticssnmp-statefulset-0     4/4     Running   8          56d
analyticssnmp1-analyticssnmp-statefulset-1     4/4     Running   4          56d
analyticssnmp1-analyticssnmp-statefulset-2     4/4     Running   4          56d
config1-config-statefulset-0                   7/7     Running   0          16h
config1-config-statefulset-1                   7/7     Running   0          16h
config1-config-statefulset-2                   7/7     Running   0          16h
configdb1-cassandra-statefulset-0              3/3     Running   7          56d
configdb1-cassandra-statefulset-1              3/3     Running   9          56d
configdb1-cassandra-statefulset-2              3/3     Running   9          56d
control1-control-statefulset-0                 5/5     Running   15         56d
control1-control-statefulset-1                 5/5     Running   11         56d
control1-control-statefulset-2                 5/5     Running   15         56d
queryengine1-queryengine-statefulset-0         1/1     Running   2          56d
queryengine1-queryengine-statefulset-1         1/1     Running   1          56d
queryengine1-queryengine-statefulset-2         1/1     Running   1          56d
redis1-redis-statefulset-0                     2/2     Running   2          56d
redis1-redis-statefulset-1                     2/2     Running   4          56d
redis1-redis-statefulset-2                     2/2     Running   2          56d
tf-operator-646fb557f-mtb6z                    1/1     Running   0          9s
vrouter1-vrouter-daemonset-9s4xb               3/3     Running   9          56d
vrouter1-vrouter-daemonset-f6v97               3/3     Running   6          56d
vrouter1-vrouter-daemonset-fnb8s               3/3     Running   3          56d
vrouter1-vrouter-daemonset-nrk9t               3/3     Running   6          56d
vrouter1-vrouter-daemonset-pksrt               3/3     Running   9          56d
vrouter1-vrouter-daemonset-z927z               3/3     Running   3          56d
webui1-webui-statefulset-0                     2/2     Running   2          13d
webui1-webui-statefulset-1                     2/2     Running   4          13d
webui1-webui-statefulset-2                     2/2     Running   2          13d
zookeeper1-zookeeper-statefulset-0             1/1     Running   3          56d
zookeeper1-zookeeper-statefulset-1             1/1     Running   3          56d
zookeeper1-zookeeper-statefulset-2             1/1     Running   2          56d

tf-operator is scaled back, further it will automatically  scale kubemanager pods and rabbitmq pods

[dcn contrail@cc-jumphost:~]$ kubectl get pods -n tf
NAME                                           READY   STATUS    RESTARTS   AGE
analytics1-analytics-statefulset-0             4/4     Running   4          56d
analytics1-analytics-statefulset-1             4/4     Running   4          56d
analytics1-analytics-statefulset-2             4/4     Running   8          56d
analyticsalarm1-analyticsalarm-statefulset-0   4/4     Running   4          56d
analyticsalarm1-analyticsalarm-statefulset-1   4/4     Running   4          56d
analyticsalarm1-analyticsalarm-statefulset-2   4/4     Running   8          56d
analyticsdb1-cassandra-statefulset-0           3/3     Running   3          56d
analyticsdb1-cassandra-statefulset-1           3/3     Running   6          56d
analyticsdb1-cassandra-statefulset-2           3/3     Running   3          56d
analyticssnmp1-analyticssnmp-statefulset-0     4/4     Running   8          56d
analyticssnmp1-analyticssnmp-statefulset-1     4/4     Running   4          56d
analyticssnmp1-analyticssnmp-statefulset-2     4/4     Running   4          56d
config1-config-statefulset-0                   7/7     Running   0          16h
config1-config-statefulset-1                   7/7     Running   0          16h
config1-config-statefulset-2                   7/7     Running   0          16h
configdb1-cassandra-statefulset-0              3/3     Running   7          56d
configdb1-cassandra-statefulset-1              3/3     Running   9          56d
configdb1-cassandra-statefulset-2              3/3     Running   9          56d
control1-control-statefulset-0                 5/5     Running   15         56d
control1-control-statefulset-1                 5/5     Running   11         56d
control1-control-statefulset-2                 5/5     Running   15         56d
kubemanager1-kubemanager-statefulset-0         1/1     Running   0          3s
kubemanager1-kubemanager-statefulset-1         1/1     Running   0          3s
kubemanager1-kubemanager-statefulset-2         1/1     Running   0          3s
queryengine1-queryengine-statefulset-0         1/1     Running   2          56d
queryengine1-queryengine-statefulset-1         1/1     Running   1          56d
queryengine1-queryengine-statefulset-2         1/1     Running   1          56d
rabbitmq1-rabbitmq-statefulset-0               1/1     Running   0          3s
rabbitmq1-rabbitmq-statefulset-1               1/1     Running   0          3s
rabbitmq1-rabbitmq-statefulset-2               1/1     Running   0          3s
redis1-redis-statefulset-0                     2/2     Running   2          56d
redis1-redis-statefulset-1                     2/2     Running   4          56d
redis1-redis-statefulset-2                     2/2     Running   2          56d
tf-operator-646fb557f-mtb6z                    1/1     Running   0          24s
vrouter1-vrouter-daemonset-9s4xb               3/3     Running   9          56d
vrouter1-vrouter-daemonset-f6v97               3/3     Running   6          56d
vrouter1-vrouter-daemonset-fnb8s               3/3     Running   3          56d
vrouter1-vrouter-daemonset-nrk9t               3/3     Running   6          56d
vrouter1-vrouter-daemonset-pksrt               3/3     Running   9          56d
vrouter1-vrouter-daemonset-z927z               3/3     Running   3          56d
webui1-webui-statefulset-0                     2/2     Running   2          13d
webui1-webui-statefulset-1                     2/2     Running   4          13d
webui1-webui-statefulset-2                     2/2     Running   2          13d
zookeeper1-zookeeper-statefulset-0             1/1     Running   3          56d
zookeeper1-zookeeper-statefulset-1             1/1     Running   3          56d
zookeeper1-zookeeper-statefulset-2             1/1     Running   2          56d

 

Modification History

KB Checked