Salt or Saltstack is an Open Source configuration management and orchestration tool, which is designed to allow users to explicitly target and issue commands to multiple machines directly. Saltstack is used for provisioning Juniper Contrail Service Orchestration (CSO) software.
This article lists the Salt commands that are used for checking the health and reachability of the database nodes while troubleshooting.
The commands are as follows:
The general Salt command to check the reachability of CSO VMs from the Salt primary server is given below. Log in to the installer VM and run the command.
<snip> From installer VM
root@host:~/# salt ‘*’ test.ping csp-regional-sblb.DB7RFF.regional: True csp-contrailanalytics-1.8V1O2D.central: True csp-central-msvm.8V1O2D.central: True csp-regional-k8mastervm.DB7RFF.regional: True csp-central-infravm.8V1O2D.central: False csp-regional-msvm.DB7RFF.regional: False csp-regional-infravm.DB7RFF.regional: True csp-central-k8mastervm.8V1O2D.central: True
Note: If the status of a VM is False, you must log in to the VM and restart the Salt Minion. The command to restart a Salt Minion is “ service salt-minion restart .”
service salt-minion restart
To check the reachability of the Elasticsearch and Cassandra node, use the command salt -C 'G@roles: <NODE> ' test.ping .
salt -C 'G@roles: <NODE> ' test.ping
<Snip> From installer node
salt -C 'G@roles:elasticsearch' test.ping” ip-12-74-2-100.elasticsearch-0.CWED8J.central: True ip-12-74-2-132.elasticsearch-2.CWED8J.central: True ip-12-74-2-116.elasticsearch-1.CWED8J.central: True salt -C 'G@roles: cassandra' test.ping”
salt -C 'G@roles:elasticsearch' test.ping”
ip-12-74-2-100.elasticsearch-0.CWED8J.central: True ip-12-74-2-132.elasticsearch-2.CWED8J.central: True ip-12-74-2-116.elasticsearch-1.CWED8J.central: True salt -C 'G@roles: cassandra' test.ping”
To determine free disk space in the Elasticsearch and Cassandra nodes, use salt -C 'G@roles: <NODE> ' cmd.run 'df -h /mnt/data '.
salt -C 'G@roles: <NODE> ' cmd.run 'df -h /mnt/data
<snip> From installer node
salt -C 'G@roles:elasticsearch' cmd.run 'df -h /mnt/data' ip-12-74-2-100.elasticsearch-0.CWED8J.central: Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 493G 266G 202G 57% /mnt/data ip-12-74-2-116.elasticsearch-1.CWED8J.central: Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 493G 265G 203G 57% /mnt/data ip-12-74-2-132.elasticsearch-2.CWED8J.central: Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 493G 266G 202G 57% /mnt/data salt -C 'G@roles:cassandra' cmd.run 'df -h /mnt/data'
salt -C 'G@roles:elasticsearch' cmd.run 'df -h /mnt/data'
ip-12-74-2-100.elasticsearch-0.CWED8J.central: Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 493G 266G 202G 57% /mnt/data ip-12-74-2-116.elasticsearch-1.CWED8J.central: Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 493G 265G 203G 57% /mnt/data ip-12-74-2-132.elasticsearch-2.CWED8J.central: Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 493G 266G 202G 57% /mnt/data
salt -C 'G@roles:cassandra' cmd.run 'df -h /mnt/data'
To determine free memory space in the Elasticsearch and Cassandra nodes, use salt -C 'G@roles: <NODE> ' cmd.run 'free -mh' .
salt -C 'G@roles: <NODE> ' cmd.run 'free -mh'
<snip>
salt -C 'G@roles:cassandra' cmd.run 'free -mh' ip-12-74-2-117.cassandra-1.CWED8J.central: total used free shared buff/cache available Mem: 30G 26G 219M 336M 4.4G 3.9G Swap: 0B 0B 0B ip-12-74-2-101.cassandra-0.CWED8J.central: total used free shared buff/cache available Mem: 30G 25G 221M 328M 5.2G 4.7G Swap: 0B 0B 0B ip-12-74-2-133.cassandra-2.CWED8J.central: total used free shared buff/cache available Mem: 30G 25G 219M 328M 5.4G 4.9G Swap: 0B 0B 0B
To check the debug logs in the Elasticsearch and Cassandra nodes, use salt -C 'G@roles: <NODE> ' cmd.run "cat /var/log/ <node> /debug.log | tail -20" .
salt -C 'G@roles: <NODE> ' cmd.run "cat /var/log/ <node> /debug.log | tail -20"
<Snip>
salt -C 'G@roles:cassandra' cmd.run "ls /var/log/cassandra/debug.log" ip-12-74-2-117.cassandra-1.CWED8J.central: /var/log/cassandra/debug.log ip-12-74-2-133.cassandra-2.CWED8J.central: /var/log/cassandra/debug.log ip-12-74-2-101.cassandra-0.CWED8J.central: /var/log/cassandra/debug.log
salt -C 'G@roles:cassandra' cmd.run "cat /var/log/cassandra/debug.log | tail -20" ip-12-74-2-117.cassandra-1.CWED8J.central: DEBUG [MemtableFlushWriter:18174] 2021-10-13 11:25:22,315 ColumnFamilyStore.java:1229 - Flushed to [BigTableReader(path='/mnt/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-5297-big-Data.db')] (1 sstables, 5.273KiB), biggest 5.273KiB, smallest 5.273KiB DEBUG [CompactionExecutor:192523] 2021-10-13 11:25:22,315 CompactionTask.java:155 - Compacting (41024db0-2c18-11ec-8cab-0955211797b6) [/mnt/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-5297-big-Data.db:level=0, /mnt/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-5294-big-Data.db:level=0, /mnt/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-5296-big-Data.db:level=0, /mnt/data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-5295-big-Data.db:level=0, ] DEBUG [PerDiskMemtableFlushWriter_0:18175] 2021-10-13 11:25:22,320 Memtable.java:456 - Writing Memtable-size_estimates@1328434465(526.962KiB serialized bytes, 52632 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)] DEBUG [PerDiskMemtableFlushWriter_0:18175] 2021-10-13 11:25:22,327 Memtable.java:485 - Completed flushing /mnt/data/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/md-11267-big-Data.db (361.751KiB) for commitlog position CommitLogPosition(segmentId=1604387576092, position=9841833)