Contrail uses IP addr for reaching different components internally in cluster. Internal IP ADD are passed as ENV variables while containers are created. Since certs generated by IPA using hostname not IP addresses, certificate error occurs when components try to communicate with ip addresses while SSL enabled. (example: ACK alarms feature sends ack request to analytics API as --> {host='172.31.21.22', port=5995} but since hostnames are used in certificates SSL transaction fails with SSL Certificate Error).

Each alarm has a base64 encoded "token" field: and we see below alarm with signature as config missing or incorrect.


[root@contrail-ctrl03 ~]# curl -H "X-Auth-Token: $OS_TOKEN" https://contrail-anly03.tungstenfabric.io:8081/analytics/alarms | jq '. | {"config-database-node"}' { "config-database-node": [ { "name": "contrail-ctrl02.tungstenfabric.io", "value": { "UVEAlarms": { "alarms": [ { "severity": 1, "alarm_rules": { "or_list": [ { "and_list": [ { "match": [ { "json_operand1_value": "null", "json_variables": {} } ], "condition": { "operation": "==", "operand1": "ContrailConfig", "variables": [], "operand2": { "json_value": "null" } } } ] } ] }, "timestamp": 1669307475002715, "ack": true, "token": "eyJ0aW1lc3RhbXAiOiAxNjY5MzA3NDc1MDAyNzE1LCAiaHR0cF9wb3J0IjogNTk5NSwgImhvc3RfaXAiOiAiMTcyLjMxLjIxLjIyIn0=", "type": "default-global-system-config:system-defined-conf-incorrect", "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect." },

 

Further decoding token, we can see below data

{"timestamp": 1669307475002715, "http_port": 5995, "host_ip": "172.31.21.22"}

Further trying to acknowledge the alarm, the contrail analytics-api/console.log shows below error:

Socket Connection error : HTTPSConnectionPool(host='172.31.21.22', port=5995): Max retries exceeded with url: /Snh_SandeshAlarmAckRequest?table=ObjectConfigDatabaseInfo&type=default-global-system-config%3Asystem-defined-conf-incorrect&name=contrail-ctrl03.tungstenfabric.io&timestamp=1669307474952796 (Caused by SSLError(CertificateError("hostname '172.31.21.22' doesn't match either of 'contrail-anly03.tungstenfabric.io', 'contrail-anly03.ctlplane.tungstenfabric.io', 'contrail-anly03.internalapi.tungstenfabric.io', 'contrail-anly03.tenant.tungstenfabric.io'",),))

So problem is that an IP address is used instead of the hostname in the base64 "token" inside the alarm and IP address is not mentioned in the alarmgen certificate, therefore the SSL connection cannot be validated when trying to acknowledge the alarm.

Below workaround can be done to bypass SSL and restore status of a broken contrail-cluster

INTROSPECT_SSL_INSECURE=True

Modify /common.sh in all the contrail_analytics_api containers, which disables SSL certificate checking.

RHOSP environment can use below mentioned patch for patching contrail heat templates that adds the node's internal_api IP address to the SANs on the certificate in certmonger.

--- /tf-tripleo-heat-templates-stable-train/deployment/contrail/contrail-certmonger-user.j2.yaml 2022-11-03 23:40:02
+++ /tf-tripleo-heat-templates-stable-train/deployment/contrail/contrail-certmonger-user.j2 copy.yaml 2022-11-25 21:18:36
@@ -101,11 +101,12 @@
         hostname: "%{hiera('fqdn_canonical')}"
         principal: "contrail/%{hiera('fqdn_canonical')}"
         dnsnames:
-         repeat:
-          template:
-           - "%{hiera('fqdn_NAME')}"
-          for_each:
-           NAME: {get_attr: [DnsNamesInCert, value]}
+         list_concat:
+          - - "%{hiera('internal_api')}"
+          - repeat:
+            template: "%{hiera('fqdn_NAME')}"
+            for_each:
+             NAME: {get_attr: [DnsNamesInCert, value]}
       - {}
 
    metadata_settings:
 
----------