Article Detail

Description

After upgrading to NorthStar version 4.2.0 on CentOS/RHEL 7.x, some processes such as Cassandra and Health Monitor may fail to start, and some of the symbolic links under /opt/northstar/thirdparty/jdk may be seen to be incorrect.

This article gives the cause for this issue and the corrective step that can be taken to resolve the problem.

Symptoms

After upgrade, some of the symbolic links under /opt/northstar/thirdparty/jdk are incorrect, as shown in the following output:

[root@SERVER ~]# supervisorctl status
analytics:elasticsearch          RUNNING   pid 9254, uptime 0:11:32
analytics:esauthproxy            FATAL     can't find command '/opt/northstar/thirdparty/node-v6.12.2-linux-x64/bin/node'
analytics:logstash               RUNNING   pid 9255, uptime 0:11:32
collector:worker1                RUNNING   pid 9256, uptime 0:11:32
collector_main:es_publisher      RUNNING   pid 9265, uptime 0:11:32
collector_main:task_scheduler    RUNNING   pid 11987, uptime 0:02:55
infra:cassandra                  FATAL     Exited too quickly (process log may have details)
infra:ha_agent                   RUNNING   pid 12457, uptime 0:01:29
infra:healthmonitor              FATAL     Exited too quickly (process log may have details)
infra:license_monitor            RUNNING   pid 9261, uptime 0:11:32
infra:prunedb                    RUNNING   pid 9257, uptime 0:11:32
infra:rabbitmq                   RUNNING   pid 9259, uptime 0:11:32
infra:redis_server               RUNNING   pid 9263, uptime 0:11:32
infra:web                        RUNNING   pid 12498, uptime 0:01:22
infra:zookeeper                  RUNNING   pid 9258, uptime 0:11:32
junos:junosvm                    RUNNING   pid 9253, uptime 0:11:32
listener1:listener1_00           RUNNING   pid 9252, uptime 0:11:32
netconf:netconfd                 RUNNING   pid 12174, uptime 0:02:20
northstar:mladapter              RUNNING   pid 11613, uptime 0:03:40
northstar:npat                   RUNNING   pid 11615, uptime 0:03:40
northstar:pceserver              RUNNING   pid 11614, uptime 0:03:40
northstar:scheduler              FATAL     Exited too quickly (process log may have details)
northstar:toposerver             RUNNING   pid 10821, uptime 0:07:24
northstar_pcs:PCServer           RUNNING   pid 10928, uptime 0:06:46
northstar_pcs:PCViewer           RUNNING   pid 10927, uptime 0:06:46
northstar_pcs:configServer       RUNNING   pid 10929, uptime 0:06:46
[root@SERVER ~]#

[root@SERVER ~]# cd /opt/northstar/thirdparty/jdk
lrwxrwxrwx. 1 pcs pcs 34 Nov 22 18:09 bin -> /usr/lib/jvm/jre-1.8.0-openjdk/bin
drwxr-xr-x. 2 pcs pcs 26 Nov 22 18:09 jre
lrwxrwxrwx. 1 pcs pcs 34 Nov 22 18:09 lib -> /usr/lib/jvm/jre-1.8.0-openjdk/lib
[root@SERVER ~]#

Solution

To resolve this problem, the symbolic links must be corrected along with any permissions, and then the affected processes must be restarted.

[root@SERVER ~]# cd /opt/northstar/thirdparty/jdk/
[root@SERVER ~]# rm -f lib bin
[root@SERVER ~]# ln -s /usr/lib/jvm/jre/bin .
[root@SERVER ~]# ln -s /usr/lib/jvm/jre/lib .
[root@SERVER ~]# chown -h pcs:pcs bin
[root@SERVER ~]# chown -h pcs:pcs lib

[root@SERVER ~]# cd jre/
[root@SERVER ~]# rm -f lib bin
[root@SERVER ~]# ln -s /usr/lib/jvm/jre/bin .
[root@SERVER ~]# ln -s /usr/lib/jvm/jre/lib .
[root@SERVER ~]# chown -h pcs:pcs bin
[root@SERVER ~]# chown -h pcs:pcs lib

[root@SERVER ~]# ls -l /opt/northstar/thirdparty/jdk/
lrwxrwxrwx. 1 pcs  pcs 21 Nov 26 10:01 bin -> /usr/lib/jvm/jre/bin/
drwxr-xr-x. 2 pcs  pcs  26 Nov 22 18:09 jre
lrwxrwxrwx. 1 pcs pcs 21 Nov 26 10:02 lib -> /usr/lib/jvm/jre/lib/
[root@SERVER ~]#

After the processes are manually restarted, all the affected processes start running normally.

[root@SERVER ~]# supervisorctl status
analytics:elasticsearch          RUNNING   pid 9254, uptime 0:11:32
analytics:esauthproxy            RUNNING   pid 9255, uptime 0:11:32  
analytics:logstash               RUNNING   pid 9256, uptime 0:11:32
collector:worker1                RUNNING   pid 9257, uptime 0:11:32
collector_main:es_publisher      RUNNING   pid 9268, uptime 0:11:32
collector_main:task_scheduler    RUNNING   pid 12460, uptime 0:02:55
infra:cassandra                  RUNNING   pid 9259, uptime 0:02:55
infra:ha_agent                   RUNNING   pid 12457, uptime 0:01:29
infra:healthmonitor              RUNNING   pid 12465, uptime 0:02:55  
infra:license_monitor            RUNNING   pid 9261, uptime 0:11:32
infra:prunedb                    RUNNING   pid 9277, uptime 0:11:32
infra:rabbitmq                   RUNNING   pid 9279, uptime 0:11:32
infra:redis_server               RUNNING   pid 9263, uptime 0:11:32
infra:web                        RUNNING   pid 12532, uptime 0:01:22
infra:zookeeper                  RUNNING   pid 9258, uptime 0:11:32
junos:junosvm                    RUNNING   pid 9253, uptime 0:11:32
listener1:listener1_00           RUNNING   pid 9252, uptime 0:11:32
netconf:netconfd                 RUNNING   pid 12174, uptime 0:02:20
northstar:mladapter              RUNNING   pid 11613, uptime 0:03:40
northstar:npat                   RUNNING   pid 11615, uptime 0:03:40
northstar:pceserver              RUNNING   pid 11614, uptime 0:03:40
northstar:scheduler              RUNNING   pid 12472, uptime 0:02:55  
northstar:toposerver             RUNNING   pid 10821, uptime 0:07:24
northstar_pcs:PCServer           RUNNING   pid 10928, uptime 0:06:46
northstar_pcs:PCViewer           RUNNING   pid 10927, uptime 0:06:46
northstar_pcs:configServer       RUNNING   pid 10929, uptime 0:06:46
[root@SERVER ~]#

Modification History

2020-05-19: Added that the issue is applicable only to CentOS/RHEL 7.x installations in Summary and article