Description

How to check if SSR deployed in HA is healthy.

Solution

Below are the few commands that can check if router in HA is healthy, This needs to be executed in both nodes.

It is assumed that 128T service is already running, you may verify it with "systemctl status 128T" and proceed with further steps.

Use "show system connectivity" to check the connectivity status to conductor as well peer node.

PCLI> show system connnectivity
Retrieving system connectivity...
============= ======================== ===========
 Local Node  Remote Node       State
============= ======================== ===========
 ha-128t-99a  ha-128t-99b.ha-128t-99  connected
 ha-128t-99a  node1.128t-con1     connected

 

Use "show system processes" to check the status of process, If running as primary it is expected to have "Y" and in secondary node "N"

PCLI> show system processes
Retrieving system processes...
======================== =========================== ========= ========= =======
 Node           Process           Status  Primary  Role
======================== =========================== ========= ========= =======
 ha-128t-99a.ha-128t-99  accessManager        running       combo
 ha-128t-99a.ha-128t-99  analyticsReporter      running       combo
 ha-128t-99a.ha-128t-99  applicationFrameworkManager  running       combo
 ha-128t-99a.ha-128t-99  configDirector       running  Y     combo
 ha-128t-99a.ha-128t-99  conflux           running       combo
 ha-128t-99a.ha-128t-99  databaseQueryCoordinator  running       combo
 ha-128t-99a.ha-128t-99  dnsManager         running  Y     combo
 ha-128t-99a.ha-128t-99  dynamicPeerUpdateManager  running  Y     combo
 ha-128t-99a.ha-128t-99  highway           running       combo
 ha-128t-99a.ha-128t-99  nodeMonitor         running       combo
 ha-128t-99a.ha-128t-99  redisServerManager     running  Y     combo
 ha-128t-99a.ha-128t-99  routingManager       running  Y     combo
 ha-128t-99a.ha-128t-99  secureCommunicationManager  running       combo
 ha-128t-99a.ha-128t-99  securityKeyManager     running  Y     combo
 ha-128t-99a.ha-128t-99  snmpTrapAgent        running       combo
 ha-128t-99a.ha-128t-99  stateMonitor        running       combo
 ha-128t-99a.ha-128t-99  systemServicesCoordinator  running       combo

 

Other option is to run "128status.sh", which will do basic health check of node status, zookeeper health, init files and peer HA node connectivity

shell# 128status.sh
---------------------------
128status.sh version 0.0.6:
---------------------------
check_rpm_signed:       PASS 128T-6.1.4-23.r2.el7 signed with d1e8bb27f891f0bd
check_time_jumps:       PASS 03:30:45 UTC, 03:30:45 UTC   
check_global_init:      PASS Valid json format /etc/128technology/global.init
check_local_init:       PASS Valid json format /etc/128technology/local.init
check_json_env_schema:    WARN Cannot be performed on 128T > 4.1.5 (jsonschema)
check_procmgr_list_init:   PASS Valid json format /etc/128technology/processManagerProcessList.json
check_listen_ports:      PASS Check Listen Ports: 930, 6380, 12222, 2181
check_zed_status:       PASS Starting PDS in redundant mode
check_zk_ruok:        PASS Zookeeper 4 letter RUOK
check_zk_status:       PASS Connections: 19, Mode: follower
check_zk_nodes:        PASS All 10 required zk nodes exist
check_zk_node_proc_config:  PASS 36/36 /state/node/<node>/processes/<process> paths have config
 :                 Node ha-128t-99a; type=control
check_nonzk_ruok:       PASS All FLW 17 processes returned IMOK     
check_zk_connections:     PASS All FLW 17 processes are ZK connected   
check_free_memory:      PASS 1387/7763MB Free Memory
check_critical_files:     PASS owner+perms for 13/13 files
check_ssh_to_peer:      PASS Success connecting to peer (10.254.254.2)
check_ssh_to_conductor:    PASS 1/1 conductors are reachable
==================================
All 18 tests:         PASS  

Modification History

2024-07-30 : Article Created