This KB article explains rabbitmq issues arriving post reboot of bare metal servers in CSO HA installations
Post server reboots, Component health script sometimes may indicates rabbitmq as unhealthy(highlighted in red below) root@startupserver1:/Contrail_Service_Orchestration_5.1.1# ./components_health.sh INFO Updating the mine and syncing the grains INFO ************************************************************************ INFO HEALTH CHECK FOR INFRASTRUCTURE COMPONENTS STARTED IN CENTRAL ENVIRONMENT INFO ************************************************************************ INFO Health Check for Infrastructure Component Saltstack Started INFO The Infrastructure Component Saltstack is Healthy INFO Health Check for Infrastructure Component Cassandra Started INFO The Infrastructure Component Cassandra is Healthy INFO Health Check for Infrastructure Component Mariadb Started INFO The Infrastructure Component Mariadb is Healthy INFO Health Check for Infrastructure Component Swift Started INFO Attempt: 1 - Retrying Health Check for Component Swift INFO The Infrastructure Component Swift is Healthy INFO Health Check for Infrastructure Component Redis Started INFO The Infrastructure Component Redis is Healthy INFO Health Check for Infrastructure Component Arangodb Started INFO The Infrastructure Component Arangodb is Healthy INFO Health Check for Infrastructure Component Keystone Started INFO The Infrastructure Component Keystone is Healthy INFO Health Check for Infrastructure Component Elasticsearch Started INFO The Infrastructure Component Elasticsearch is Healthy INFO Health Check for Infrastructure Component Elk_Elasticsearch Started INFO The Infrastructure Component Elk_Elasticsearch is Healthy INFO Health Check for Infrastructure Component Icinga Started INFO The Infrastructure Component Icinga is Healthy INFO Health Check for Infrastructure Component Rabbitmq Started INFO Attempt: 1 - Retrying Health Check for Component Rabbitmq INFO Attempt: 2 - Retrying Health Check for Component Rabbitmq ERROR The Infra Component : Rabbitmq is Unhealthy INFO Health Check for Infrastructure Component Etcd Started INFO The Infrastructure Component Etcd is Healthy INFO Health Check for Infrastructure Component Rsyslog Started INFO The Infrastructure Component Rsyslog is Healthy INFO Health Check for Infrastructure Component Kubernetes Started INFO The Infrastructure Component Kubernetes is Healthy INFO Health Check for Infrastructure Component Elk_Logstash Started INFO The Infrastructure Component Elk_Logstash is Healthy INFO Health Check for Infrastructure Component Elk_Kibana Started INFO The Infrastructure Component Elk_Kibana is Healthy INFO Health Check for Infrastructure Component Zookeeper Started INFO The Infrastructure Component Zookeeper is Healthy INFO Health Check for Infrastructure Component Contrail_Analytics Started INFO The Infrastructure Component Contrail_Analytics is Healthy INFO Overall result: INFO The following Infrastructure Components are Healthy: INFO ['Saltstack', 'Cassandra', 'Mariadb', 'Swift', 'Redis', 'Arangodb', 'Keystone', 'Elasticsearch', 'Elk_Elasticsearch', 'Icinga', 'Etcd', 'Rsyslog', 'Kubernetes', 'Elk_Logstash', 'Elk_Kibana', 'Zookeeper', 'Contrail_Analytics'] ERROR The following Infrastructure Components are Unhealthy: ['Rabbitmq'] Please Check the logs: =========================== Info Logs Path: /Contrail_Service_Orchestration_5.1.1/logs/health_check.log
root@startupserver1:/Contrail_Service_Orchestration_5.1.1# ./components_health.sh INFO Updating the mine and syncing the grains INFO ************************************************************************ INFO HEALTH CHECK FOR INFRASTRUCTURE COMPONENTS STARTED IN CENTRAL ENVIRONMENT INFO ************************************************************************
INFO Health Check for Infrastructure Component Saltstack Started INFO The Infrastructure Component Saltstack is Healthy
INFO Health Check for Infrastructure Component Cassandra Started INFO The Infrastructure Component Cassandra is Healthy
INFO Health Check for Infrastructure Component Mariadb Started INFO The Infrastructure Component Mariadb is Healthy
INFO Health Check for Infrastructure Component Swift Started INFO Attempt: 1 - Retrying Health Check for Component Swift INFO The Infrastructure Component Swift is Healthy
INFO Health Check for Infrastructure Component Redis Started INFO The Infrastructure Component Redis is Healthy
INFO Health Check for Infrastructure Component Arangodb Started INFO The Infrastructure Component Arangodb is Healthy
INFO Health Check for Infrastructure Component Keystone Started INFO The Infrastructure Component Keystone is Healthy
INFO Health Check for Infrastructure Component Elasticsearch Started INFO The Infrastructure Component Elasticsearch is Healthy
INFO Health Check for Infrastructure Component Elk_Elasticsearch Started INFO The Infrastructure Component Elk_Elasticsearch is Healthy
INFO Health Check for Infrastructure Component Icinga Started INFO The Infrastructure Component Icinga is Healthy
INFO Health Check for Infrastructure Component Rabbitmq Started INFO Attempt: 1 - Retrying Health Check for Component Rabbitmq INFO Attempt: 2 - Retrying Health Check for Component Rabbitmq ERROR The Infra Component : Rabbitmq is Unhealthy
INFO Health Check for Infrastructure Component Etcd Started INFO The Infrastructure Component Etcd is Healthy
INFO Health Check for Infrastructure Component Rsyslog Started INFO The Infrastructure Component Rsyslog is Healthy
INFO Health Check for Infrastructure Component Kubernetes Started INFO The Infrastructure Component Kubernetes is Healthy
INFO Health Check for Infrastructure Component Elk_Logstash Started INFO The Infrastructure Component Elk_Logstash is Healthy
INFO Health Check for Infrastructure Component Elk_Kibana Started INFO The Infrastructure Component Elk_Kibana is Healthy
INFO Health Check for Infrastructure Component Zookeeper Started INFO The Infrastructure Component Zookeeper is Healthy
INFO Health Check for Infrastructure Component Contrail_Analytics Started INFO The Infrastructure Component Contrail_Analytics is Healthy
INFO Overall result: INFO The following Infrastructure Components are Healthy: INFO ['Saltstack', 'Cassandra', 'Mariadb', 'Swift', 'Redis', 'Arangodb', 'Keystone', 'Elasticsearch', 'Elk_Elasticsearch', 'Icinga', 'Etcd', 'Rsyslog', 'Kubernetes', 'Elk_Logstash', 'Elk_Kibana', 'Zookeeper', 'Contrail_Analytics'] ERROR The following Infrastructure Components are Unhealthy: ['Rabbitmq'] Please Check the logs: =========================== Info Logs Path: /Contrail_Service_Orchestration_5.1.1/logs/health_check.log
The cause of this issue is rabbitmq cluster may not be formed correctly post reboot. Logging into rabbitmq webui using : http://<>:15672 , (username: cspmq, password can be obtained from command ./python.sh deploy_manager/utils/decrypt_password.py ) issue can be verified by clicking on overview tab below. As we can see, one of the rabbitmq node in the HA cluster is not running, which triggers the "unhealthy" message in health check.
Rabbitmq can be recovered by running recovery script, which is located in the install directory of Contrail_Service_Orchestration_5.1.1 root@startupserver1:/Contrail_Service_Orchestration_5.1.1# ./recovery.sh *************** This tool assists you recover your CSO setup. *************** Following components can be recovered 1: mariadb 2: vrr 3: icinga 4: rabbitmq Specify one of the component to recover (In Number) : 4 INFO Started recovering rabbitmq component at 2020-03-04 21:31:02.537449 ... INFO Rabbitmq failure recovery is initiated, it will take couple of mintues... INFO Rabbitmq recovery started. Please wait... INFO Completed recovering rabbitmq component at 2020-03-04 21:48:11.882178 . INFO Time taken to recover 0:17:09.344729 Verification can be done by running components_health.sh script again after recover step taken. Note : In case of rabbitmq failure, error is printed as "INFO Unable to Recover Rabbitmq" , this will be enhanced in future and it will print failure as ERROR instead of INFO