This article applies to a scenario with RHOSP cluster and Contrail Networking 2011.L2. It is only applicable if a stack update is performed with the stable/train version parent ID 232535ec.It does not apply when a new cluster is deployed using the same template version parent ID 232535ec. This version has the files "deployment/contrail/contrail-config-database.yaml" and "deployment/contrail/contrail-analytics-database.yaml" updated with "user: root" permissions.
After performing a stack update, when the command, 'contrail-status' is executed on config/analytics database, the node manager gets stuck in initializing state.
[root@lxvcw1jd1001 ~]# contrail-status Pod Service Original Name Original Version State Id Status database cassandra contrail-external-cassandra ubi-queens-2011-372 running 7537f29a09c8 Up 3 days ago database nodemgr contrail-nodemgr ubi-queens-2011-372 running 4f313d00ec90 Up 6 weeks ago database provisioner contrail-provisioner ubi-queens-2011-372 running 97de3b99a638 Up 6 weeks ago database query-engine contrail-analytics-query-engine ubi-queens-2011-372 running c7964033aad8 Up 6 weeks ago == Contrail database == nodemgr: initializing (Cassandra state detected DOWN. Disk space for DB not retrievable. ) query-engine: active cassandra: active
Also, in the podman logs for the node manager container references issues with the root user:
CRITICAL:root:Error: unable to find user root: no matching entries in passwd file ERROR:contrail-database-nodemgr:Failed to get database usage: Result: 1
However, the root user is present in the passwd file within this container:
(database-nodemgr)[root@lxvcw1jd1001 /]$ cat /etc/passwd | grep root root:x:0:0:root:/root:/bin/bash
Rolling Restart of nodemanager service on all 3 controllers at 1 min intervals resolves the issue and no errors are seen then.Restart can be done as below:
[root@overcloud-contrailcontroller-0 heat-admin]# docker ps | grep -i database_nodemgr 971dd46b0fd4 192.168.24.1:8787/contrail/contrail-nodemgr:2011.L2.372 "/entrypoint.sh /b..." 4 months ago Up 2 months contrail_analytics_database_nodemgr 33136efe81eb 192.168.24.1:8787/contrail/contrail-nodemgr:2011.L2.372 "/entrypoint.sh /b..." 4 months ago Up 2 months contrail_config_database_nodemgr [root@overcloud-contrailcontroller-0 heat-admin]#
Identify the name of the nodemanager which has the issues. For example, if it is Analytics Database nodemanager, then restart the analytics nodemanager as below:
podman restart contrail_analytics_database_nodemgr