Description

When going into net_setup.py , under HA Setting > Check cluster status, output displays FQDN for hostname.  Additionally, the cluster appears in an unstable state.

Symptoms

Cluster status check is showing two sets of hostnames for each node in the cluster.  The second set is displaying FQDN.  Additionally, even though priority is set for each node, one of the nodes does not have a priority, and the cluster seems unstable.

General HA status

NorthStar HA Status

Host Name	    IPv4		Role			 Version
host-1	            192.168.1.5		Infrastructure		 5.1.0-20191210_220522_bb37a329b_64.x86_64
host-2	            192.168.1.7		Active,Coordinator	 5.1.0-20191210_220522_bb37a329b_64.x86_64
host-1.example.com		        Standby		         5.1.0-20191210_220522_bb37a329b_64.x86_64
host-3	            192.168.1.9		Standby			 5.1.0-20191210_220522_bb37a329b_64.x86_64
host-2.example.com		        Infrastructure	         5.1.0-20191210_220522_bb37a329b_64.x86_64
host-3.example.com		        Infrastructure	         5.1.0-20191210_220522_bb37a329b_64.x86_64

ZooKeeper cluster status:

Host Name	    IPv4		Mode			 Version
host-1	            192.168.1.5		follower		 3.5.4-beta-7f51e5b68cf2f80176ff944a9ebd2abbc65e7320, built on 05/11/2018 16
host-3	            192.168.1.9		follower		 3.5.4-beta-7f51e5b68cf2f80176ff944a9ebd2abbc65e7320, built on 05/11/2018 16
host-2	            192.168.1.7		leader			 3.5.4-beta-7f51e5b68cf2f80176ff944a9ebd2abbc65e7320, built on 05/11/2018 16

Solution

When setting up a cluster, it is recommended to use only hostname instead of using any fully qualified domain name (FQDN), like host-1.example.com.  Using FQDN can cause cluster instabilities.

There are 5 places to check for FQDN on NorthStar:

  1. /opt/northstar/data/ha_setup.json
  2. /etc/hosts
  3. /etc/sysconfig/network
  4. hostnamectl
  5. /etc/hostname

If an FQDN exists in any one of these files, edit the file and remove the domain portion of the hostname.  In this example, instead of host-1.example.com, modify it to read only host-1.  Once this is modified, restart northstar by issuing the command:

service northstar restart

Repeat the same procedure for all nodes in the cluster.

Modification History

2021-08-24: minor non-technical edits