Article Detail

Description

This article covers the specific scenario of a components health check failing in the installer VM and a workaround for it.

Note: This article is applicable only to CSO version 4.0.2 or earlier.

Symptoms

The following error is reported during a components health check:

root@cso-installer-vm:~/CSO-4.0.2/Contrail_Service_Orchestration_4.0.2# ./components_health.sh
Syncing salt data, this will take a while..

************************************************************************

HEALTH CHECK FOR INFRASTRUCTURE COMPONENTS STARTED IN CENTRAL ENVIRONMENT

************************************************************************

INFO     Health Check for Infrastructure Component Cassandra Started
ERROR    No minions matched the target. Please check the minion running status.

Please Check the logs:
===========================
Console Logs Path: /root/CSO-4.0.2/Contrail_Service_Orchestration_4.0.2/logs/health_check_console.log
Error Logs Path: /root/CSO-4.0.2-/Contrail_Service_Orchestration_4.0.2/logs/health_check_error.log
Info Logs Path: /root/CSO-4.0.2/Contrail_Service_Orchestration_4.0.2/logs/health_check.log

Solution

This is due to a bug in the salt version ( 2017.7.5) used in CSO 4.0.2, due to which the data that is cached for minions by the salt primary in a long-running setup is lost.

The following commands need to be run in the installer VM.

Run the salt-key command to check whether all the VMs are listed in the output.

# salt-key
Accepted Keys:
csp-central-infravm.R82ZN0.central
csp-central-k8mastervm.R82ZN0.central
csp-central-msvm.R82ZN0.central
csp-contrailanalytics-1.R82ZN0.central
csp-regional-sblb.KNVKBT.regional
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Initiate the salt '*' test.ping command from the installer VM to run a simple ping. A value of true indicates that the test was successful.

# salt '*' test.ping
csp-regional-sblb.KNVKBT.regional:
    True
csp-central-infravm.R82ZN0.central:
    True
csp-central-k8mastervm.R82ZN0.central:
    True
csp-contrailanalytics-1.R82ZN0.central:
    True
csp-central-msvm.R82ZN0.central:
    True

Run the salt '*' deployutils.get_role_ips 'cassandra' command to list the IP addresses assigned to the VM/role.

In the following output, no IP address assignment is seen for the VMs.

# salt '*' deployutils.get_role_ips 'cassandra'
csp-central-k8mastervm.R82ZN0.central:
csp-central-infravm.R82ZN0.central:
csp-regional-sblb.KNVKBT.regional:
csp-central-msvm.R82ZN0.central:
csp-contrailanalytics-1.R82ZN0.central:

There is a known workaround for this problem for CSO 4.0.2 and earlier versions, which is to run the following command:

salt '*' cmd.run "service salt-minion restart"

The above command restarts the minions and makes the salt primary fetch data from these minions again to keep in the local cache.

# salt '*' cmd.run "service salt-minion restart"
csp-regional-sblb.KNVKBT.regional:
    salt-minion stop/waiting
    salt-minion start/running, process 22374
csp-central-infravm.R82ZN0.central:
    salt-minion stop/waiting
    salt-minion start/running, process 28947
csp-central-k8mastervm.R82ZN0.central:
    salt-minion stop/waiting
    salt-minion start/running, process 16572
csp-contrailanalytics-1.R82ZN0.central:
    salt-minion stop/waiting
    salt-minion start/running, process 4634
csp-central-msvm.R82ZN0.central:
    salt-minion stop/waiting
    salt-minion start/running, process 28515

Note: This issue is not seen in CSO 4.1.1 versions because the components health check script first checks the salt primary cache and if the cache data is missing, it syncs the minion data before running the components health check, which prevents the specified error. Refer to Salt 2017.7.5 Release Notes for more information.

Run the components health script again after executing the above workaround and verify the output:

root@cso-installer-vm:~/CSO-4.0.2/Contrail_Service_Orchestration_4.0.2# ./components_health.sh
Syncing salt data, this will take a while..

************************************************************************
HEALTH CHECK FOR INFRASTRUCTURE COMPONENTS STARTED IN CENTRAL ENVIRONMENT
************************************************************************

INFO     Health Check for Infrastructure Component Cassandra Started
INFO     The Infrastructure Component Cassandra is Healthy

INFO     Health Check for Infrastructure Component ElasticSearch Started
INFO     The Infrastructure Component ElasticSearch is Healthy

INFO     Health Check for Infrastructure Component Etcd Started
INFO     The Infrastructure Component Etcd is Healthy

INFO     Health Check for Infrastructure Component MariaDb Started
INFO     The Infrastructure Component MariaDb is Healthy

INFO     Health Check for Infrastructure Component RabbitMQ Started
INFO     The Infrastructure Component RabbitMQ is Healthy

INFO     Health Check for Infrastructure Component ZooKeeper Started
INFO     The Infrastructure Component ZooKeeper is Healthy

INFO     Health Check for Infrastructure Component Redis Started
INFO     The Infrastructure Component Redis is Healthy

INFO     Health Check for Infrastructure Component ArangoDb Started
INFO     The Infrastructure Component ArangoDb is Healthy

INFO     Health Check for Infrastructure Component Sim_Cluster Started
INFO     The Infrastructure Component Sim_Cluster is Healthy

INFO     Health Check for Infrastructure Component Elk_Logstash Started
INFO     The Infrastructure Component Elk_Logstash is Healthy

INFO     Health Check for Infrastructure Component Elk_Kibana Started
INFO     The Infrastructure Component Elk_Kibana is Healthy

INFO     Health Check for Infrastructure Component Keystone Started
INFO     The Infrastructure Component Keystone is Healthy

INFO     Health Check for Infrastructure Component Swift Started
INFO     The Infrastructure Component Swift is Healthy

INFO     Health Check for Infrastructure Component Kubernetes Started
INFO     The Infrastructure Component Kubernetes is Healthy

INFO     Health Check for Infrastructure Component Contrail_Analytics Started
INFO     The Infrastructure Component Contrail_Analytics is Healthy

Overall result:

         The following Infrastructure Components are Healthy:
                 ['Cassandra', 'ElasticSearch', 'Etcd', 'MariaDb', 'RabbitMQ', 'ZooKeeper', 'Redis', 'ArangoDb', 'Sim_Cluster', 'Elk_Logstash', 'Elk_Kibana', 'Keystone', 'Swift', 'Kubernetes', 'Contrail_Analytics']

HEALTH CHECK NOT REQUIRED FOR COLLOCATED REGION

Please check the logs:
Console Logs Path: /root/CSO-4.0.2/Contrail_Service_Orchestration_4.0.2/logs/health_check_console.log
Error Logs Path: /root/CSO-4.0.2/Contrail_Service_Orchestration_4.0.2/logs/health_check_error.log
Info Logs Path: /root/CSO-4.0.2/Contrail_Service_Orchestration_4.0.2/logs/health_check.log

INFO     =============== Health Check start time: 2019-07-23 17:51:52.029308 ==============
INFO     =============== Health Check end time: 2019-07-23 17:56:06.896536   ==============
INFO     =============== Health Check duration: 0:04:14.867228   ==============

Expected output (where X.X.X.X is the IP address)

# salt '*' deployutils.get_role_ips 'cassandra'
csp-regional-sblb.KNVKBT.regional:
csp-central-k8mastervm.R82ZN0.central:
    - X.X.X.X
csp-central-infravm.R82ZN0.central:
    - X.X.X.X
csp-contrailanalytics-1.R82ZN0.central:
    - X.X.X.X
csp-central-msvm.R82ZN0.central:
    - X.X.X.X

# salt '*' deployutils.get_role_ips 'haproxy_confd'
csp-regional-sblb.KNVKBT.regional:
csp-contrailanalytics-1.R82ZN0.central:
    - X.X.X.X
csp-central-k8mastervm.R82ZN0.central:
    - X.X.X.X
csp-central-infravm.R82ZN0.central:
    - X.X.X.X
csp-central-msvm.R82ZN0.central:
    - X.X.X.X

Modification History

2021-03-23: Updated the article terminology to align with Juniper's Inclusion & Diversity initiatives