How to get status of all the services running on Session Smart Router and Conductor

Description

Below are the commands (utilities) available to check 128T Conductor process status.

Solution

[CONDUCTOR]

On the Conductor CLI run the below two commands to get process status [output taken from v5.1]:

[t128@conductor ~]$ sudo systemctl status 128T
● 128T.service - 128T service
   Loaded: loaded (/usr/lib/systemd/system/128T.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-05-18 16:55:58 UTC; 1 day 1h ago
 Main PID: 2009 (processManager)
    Tasks: 345
   Memory: 1.5G
   CGroup: /system.slice/128T.service
           ├─ 2009 /usr/bin/processManager
           ├─ 2049 /usr/bin/secureCommunicationManager --startup-log-level Info
           ├─ 2054 /usr/bin/java -Dlog4j.configuration=file:///etc/128technology/log4j.properties -Dzookeeper.root.logger=ERROR -Dzookeeper.log.dir=/var/log/128technology -Dznode.container.checkIntervalMs=10000 -cp /usr/lib64/*:/us...
           ├─ 2056 /usr/bin/persistentDataManager --startup-log-level Info
           ├─ 2105 /usr/sbin/openssh-fips/sshd -p 930 -o ListenAddress=0.0.0.0 -D -o ClientAliveInterval=5 -o ClientAliveCountMax=4 -f /etc/128technology/ssh/scm_sshd_config
           ├─ 2108 /usr/bin/openssh-fips/ssh -L 127.0.0.2:12222:127.0.0.1:12222 127.0.0.1 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=1 -o ServerAliveCountMax=9 -i /etc/128technology/ssh/pdc_ssh_ke...
           ├─ 2115 /usr/bin/openssh-fips/ssh -L 127.0.0.2:15555:127.0.0.1:15555 127.0.0.1 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=1 -o ServerAliveCountMax=9 -i /etc/128technology/ssh/pdc_ssh_ke...
           ├─ 2123 sshd: root
           ├─ 2124 sshd: root
           ├─ 2305 /usr/bin/securityKeyManager --startup-log-level Info
           ├─ 2307 /usr/bin/snmpTrapAgent --startup-log-level Info
           ├─ 2309 /usr/bin/influxd -config /etc/128technology/influxdb/influxdb.conf
           ├─ 2383 /usr/bin/systemServicesCoordinator --startup-log-level Info
           ├─ 2386 /usr/bin/analyticsReporter --startup-log-level Info
           ├─ 2388 /usr/bin/databaseQueryCoordinator --startup-log-level Info
           ├─ 2390 /usr/bin/conflux --startup-log-level Info
           ├─ 2392 /usr/sbin/openssh-fips/sshd -D -f /etc/128technology/ssh/pdm_sshd_config
           ├─ 2394 /usr/bin/stateMonitor --startup-log-level Info
           ├─ 2396 /usr/bin/nodeMonitor --startup-log-level Info
           ├─ 2649 /usr/bin/accessManager --startup-log-level Info
           ├─ 2651 /usr/sbin/rsyslogd -n -f /etc/128technology/rsyslog.conf
           ├─ 2653 /usr/bin/automatedProvisioner --startup-log-level Info
           ├─ 2655 /usr/bin/dynamicPeerUpdateManager --startup-log-level Info
           ├─ 2657 /usr/sbin/squid -N -f /etc/128technology/squid.conf
           ├─ 2659 /usr/bin/HTTPFileServer --path /var/www/repos --hostname 127.0.0.1 --port 5143
           ├─ 2661 /usr/bin/dnsManager --startup-log-level Info
           ├─ 2663 /usr/bin/stepRepository --startup-log-level Info
           ├─ 2712 (logfile-daemon) /var/log/squid/access.log
           ├─ 3093 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3098 python3.6 /usr/lib/128technology/par/event_publisher.par
           ├─ 3120 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3124 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3125 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3126 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3127 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3128 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3129 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3130 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3131 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3132 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3133 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3134 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3136 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3137 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3138 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 3139 python3.6 /usr/lib/128technology/par/salt_master.par --user=root --log-file=/var/log/128technology/saltMaster.log
           ├─ 9364 sshd: root
           ├─ 9366 sshd: root
           ├─18475 sshd: root
           ├─18476 sshd: root
           ├─19285 sshd: root
           ├─19286 sshd: root
           ├─19363 sshd: root
           ├─19365 sshd: root
           ├─20133 sshd: root
           └─20134 sshd: root

May 19 18:27:03 poc-conductor processManager[2009]: 128T Salt Master: [ERROR   ] Authentication attempt from 78be457c-e5de-496d-b0b9-2709e4dfbb01 failed, the public keys did not match. This may be an attempt to comp...he Salt cluster.
May 19 18:29:20 poc-conductor automatedProvisioner[2653]: [ERROR   ] Authentication attempt from 78be457c-e5de-496d-b0b9-2709e4dfbb01 failed, the public keys did not match. This may be an attempt to compromise the Salt cluster.
May 19 18:29:20 poc-conductor processManager[2009]: Automated Provisioner: 128T Salt Master: [ERROR   ] Authentication attempt from 78be457c-e5de-496d-b0b9-2709e4dfbb01 failed, the public keys did not match. This ma...he Salt cluster.
May 19 18:29:20 poc-conductor processManager[2009]: Automated Provisioner:
May 19 18:29:20 poc-conductor processManager[2009]: 128T Salt Master: [ERROR   ] Authentication attempt from 78be457c-e5de-496d-b0b9-2709e4dfbb01 failed, the public keys did not match. This may be an attempt to comp...he Salt cluster.
May 19 18:31:41 poc-conductor automatedProvisioner[2653]: [ERROR   ] Authentication attempt from 78be457c-e5de-496d-b0b9-2709e4dfbb01 failed, the public keys did not match. This may be an attempt to compromise the Salt cluster.
May 19 18:31:41 poc-conductor processManager[2009]: 128T Salt Master: [ERROR   ] Authentication attempt from 78be457c-e5de-496d-b0b9-2709e4dfbb01 failed, the public keys did not match. This may be an attempt to comp...he Salt cluster.
May 19 18:31:41 poc-conductor processManager[2009]: Automated Provisioner: 128T Salt Master: [ERROR   ] Authentication attempt from 78be457c-e5de-496d-b0b9-2709e4dfbb01 failed, the public keys did not match. This ma...he Salt cluster.
May 19 18:31:41 poc-conductor processManager[2009]: Automated Provisioner:
May 19 18:32:31 poc-conductor rsyslogd[2651]: imjournal: journal reloaded... [v8.24.0-52.el7 try http://www.rsyslog.com/e/0 ]

Hint: Some lines were ellipsized, use -l to show in full.

[t128@conductor ~]$ sudo 128status.sh
---------------------------
128status.sh version 0.0.6:
---------------------------
check_rpm_signed:             PASS 128T-5.1.0-1.el7 signed with 725a97071b515174
check_time_jumps:             PASS 18:35:38 UTC, 18:35:38 UTC             
check_global_init:            PASS Valid json format /etc/128technology/global.init
check_local_init:             PASS Valid json format /etc/128technology/local.init
check_json_env_schema:        WARN Cannot be performed on 128T > 4.1.5 (jsonschema)
check_procmgr_list_init:      PASS Valid json format /etc/128technology/processManagerProcessList.json
check_listen_ports:           PASS Check Listen Ports: 930, 12222, 830, 2181
check_zed_status:             WARN UNMATCHED LOG: Starting PDS with configured servers, ZED inactive
check_zk_ruok:                PASS Zookeeper 4 letter RUOK
check_zk_status:              PASS Connections: 17, Mode: standalone (ZED match)
check_zk_nodes:               PASS All 10 required zk nodes exist
check_zk_node_proc_config:    PASS 16/16 /state/node/<node>/processes/<process> paths have config
 :                                 Node conductor1; type=conductor
check_nonzk_ruok:             PASS All FLW 15 processes returned IMOK       
check_zk_connections:         PASS All FLW 15 processes are ZK connected    
check_free_memory:            PASS 5596/7812MB Free Memory
check_critical_files:         PASS owner+perms for 13/13 files
check_ssh_to_peer:            WARN No peer IP in global.init!
check_ssh_to_conductor:       WARN Skipping; node is a conductor.
==================================
All 18 tests:                 PASS

[ROUTER]

On 128T router CLI run the below two commands to get process status [output taken from v5.1]:

[t128@router ~]$  sudo systemctl status 128T
● 128T.service - 128T service
   Loaded: loaded (/usr/lib/systemd/system/128T.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-05-18 23:10:59 UTC; 19h ago
 Main PID: 1621 (processManager)
    Tasks: 416
   Memory: 1.0G
   CGroup: /system.slice/128T.service
           ├─1621 /usr/bin/processManager
           ├─1661 /usr/bin/secureCommunicationManager --startup-log-level Info
           ├─1664 /usr/bin/java -Dlog4j.configuration=file:///etc/128technology/log4j.properties -Dzookeeper.root.logger=ERROR -Dzookeeper.log.dir=/var/log/128technology -Dznode.container.checkIntervalMs=10000 -cp /usr/lib64/*:/usr...
           ├─1667 /usr/bin/persistentDataManager --startup-log-level Info
           ├─1720 /usr/sbin/openssh-fips/sshd -p 930 -o ListenAddress=0.0.0.0 -D -o ClientAliveInterval=1 -o ClientAliveCountMax=9 -f /etc/128technology/ssh/scm_sshd_config
           ├─1723 /usr/bin/openssh-fips/ssh -L 127.0.0.2:2620:127.0.0.1:2620 127.0.0.1 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=1 -o ServerAliveCountMax=9 -i /etc/128technology/ssh/pdc_ssh_key -...
           ├─1725 /usr/bin/openssh-fips/ssh -L 127.0.0.1:4391:127.0.0.1:6379 127.0.0.1 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=1 -o ServerAliveCountMax=9 -i /etc/128technology/ssh/pdc_ssh_key -...
           ├─1727 /usr/bin/openssh-fips/ssh -L 127.0.0.2:6379:127.0.0.1:6379 127.0.0.1 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=1 -o ServerAliveCountMax=9 -i /etc/128technology/ssh/pdc_ssh_key -...
           ├─1730 /usr/bin/openssh-fips/ssh -L 127.0.0.2:12222:127.0.0.1:12222 127.0.0.1 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=1 -o ServerAliveCountMax=9 -i /etc/128technology/ssh/pdc_ssh_key...
           ├─1733 /usr/bin/openssh-fips/ssh -L 127.0.0.2:13333:127.0.0.1:13333 127.0.0.1 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=1 -o ServerAliveCountMax=9 -i /etc/128technology/ssh/pdc_ssh_key...
           ├─1734 sshd: root
           ├─1735 sshd: root
           ├─1736 sshd: root
           ├─1738 /usr/bin/openssh-fips/ssh -L 127.0.0.2:14444:127.0.0.1:14444 127.0.0.1 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=1 -o ServerAliveCountMax=9 -i /etc/128technology/ssh/pdc_ssh_key...
           ├─1741 sshd: root
           ├─1744 sshd: root
           ├─1745 sshd: root
           ├─1787 /usr/bin/openssh-fips/ssh -L 127.0.1.2:15555:127.0.0.1:15555 10.85.216.140 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=5 -o ServerAliveCountMax=4 -i /etc/128technology/ssh/pdc_ssh...
           ├─1791 /usr/bin/openssh-fips/ssh -L 127.0.1.2:12222:127.0.0.1:12222 10.85.216.140 -p 930 -N -o StrictHostKeyChecking=no -o UseRoaming=no -o ServerAliveInterval=5 -o ServerAliveCountMax=4 -i /etc/128technology/ssh/pdc_ssh...
           ├─1924 /usr/bin/securityKeyManager --startup-log-level Info
           ├─1926 /usr/bin/snmpTrapAgent --startup-log-level Info
           ├─2031 /usr/bin/influxd -config /etc/128technology/influxdb/influxdb.conf
           ├─2325 /usr/bin/systemServicesCoordinator --startup-log-level Info
           ├─2342 /usr/bin/analyticsReporter --startup-log-level Info
           ├─2344 /usr/bin/databaseQueryCoordinator --startup-log-level Info
           ├─2348 /usr/bin/conflux --startup-log-level Info
           ├─2353 /usr/bin/routingManager --startup-log-level Info
           ├─2368 /usr/sbin/openssh-fips/sshd -D -f /etc/128technology/ssh/pdm_sshd_config
           ├─2388 /usr/bin/highway --startup-log-level Info
           ├─2391 /usr/bin/stateMonitor --startup-log-level Info
           ├─2399 /usr/bin/nodeMonitor --startup-log-level Info
           ├─2531 /usr/bin/redisServerManager --startup-log-level Info
           ├─2543 /usr/bin/redis-server 127.0.0.1:6379
           ├─2547 /usr/bin/accessManager --startup-log-level Info
           ├─2556 /usr/sbin/rsyslogd -n -f /etc/128technology/rsyslog.conf
           ├─2562 /usr/bin/applicationFrameworkManager --startup-log-level Info
           ├─2565 /usr/bin/dynamicPeerUpdateManager --startup-log-level Info
           ├─2573 /usr/bin/dnsManager --startup-log-level Info
           ├─3476 /usr/sbin/zebra -f /etc/128technology/routing/zebra.conf -M fpm
           ├─3491 /usr/sbin/bgpd -f /etc/128technology/routing/bgpd.conf
           └─3504 /usr/sbin/ospfd -f /etc/128technology/routing/ospfd.conf

May 19 18:02:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:05:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:08:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:11:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:14:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:17:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:23:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:26:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:29:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports
May 19 18:32:26 base-router routingManager[2353]: Path SLA report scheduler processed 2 reports

[t128@router ~]$ sudo 128status.sh
---------------------------
128status.sh version 0.0.6:
---------------------------
check_rpm_signed:             PASS 128T-5.1.0-1.el7 signed with 725a97071b515174
check_time_jumps:             PASS 18:36:51 UTC, 18:36:51 UTC             
check_global_init:            PASS Valid json format /etc/128technology/global.init
check_local_init:             PASS Valid json format /etc/128technology/local.init
check_json_env_schema:        WARN Cannot be performed on 128T > 4.1.5 (jsonschema)
check_procmgr_list_init:      PASS Valid json format /etc/128technology/processManagerProcessList.json
check_listen_ports:           PASS Check Listen Ports: 930, 12222, 830, 2181
check_zed_status:             WARN UNMATCHED LOG: Starting PDS with configured servers, ZED inactive
check_zk_ruok:                PASS Zookeeper 4 letter RUOK
check_zk_status:              PASS Connections: 19, Mode: standalone (ZED match)
check_zk_nodes:               PASS All 10 required zk nodes exist
check_zk_node_proc_config:    PASS 18/18 /state/node/<node>/processes/<process> paths have config
 :                                 Node node1; type=control
check_nonzk_ruok:             PASS All FLW 18 processes returned IMOK        
check_zk_connections:         PASS All FLW 18 processes are ZK connected     
check_free_memory:            PASS 3810/7812MB Free Memory
check_critical_files:         PASS owner+perms for 13/13 files
check_ssh_to_peer:            WARN No peer IP in global.init!
check_ssh_to_conductor:       PASS 1/1 conductors are reachable
==================================
All 18 tests:                 PASS