Contrail users using a Contrail Networking (CN 1908) with RedHat OpenStack (RHOSP 13) setup may encounter frequent RabbitMQ timeout errors related to "AMQP connection getting closed."
This article explains how to determine the cause for these errors and to clear them.
RabbitMQ logs show frequent error messages related to "AMQP connection getting closed" after timeout.
=INFO REPORT==== 30-Aug-2021::16:36:14 === accepting AMQP connection <0.4068.64> (172.16.18.13:46710 -> 172.16.18.13:5673) =INFO REPORT==== 30-Aug-2021::16:36:14 === connection <0.4068.64> (172.16.18.13:46710 -> 172.16.18.13:5673): user 'guest' authenticated and granted access to vhost '/' =ERROR REPORT==== 30-Aug-2021::16:36:53 === closing AMQP connection <0.4068.64> (172.16.18.13:46710 -> 172.16.18.13:5673):
The "svc_monitor" queue shows a high number of unacknowledged messages.
[root@DPDK-Compute1 ~]# sudo docker exec -it contrail_config_RabbitMQ RabbitMQctl list_queues Listing queues svc_monitor.DPDK-Compute1.example.com 3913310 . . [root@DPDK-Compute1 ~]#
One of the RabbitMQ channels also reports a high number of unacknowledged messages.
[root@DPDK-Compute1 ~]# sudo docker exec -it contrail_config_RabbitMQ RabbitMQctl list_channels Listing channels . . <[email protected]> guest 1 9100 [root@DPDK-Compute1 ~]#
To troubleshoot, perform the following steps:
Check the RabbitMQ cluster status. As shown below, the result shows a stable cluster with no alarms:
[root@DPDK-Compute1 ~] # sudo docker exec -it contrail_config_RabbitMQ RabbitMQctl cluster_status Cluster status of node contrail@DPDK-Compute1 [{nodes,[{disc,[contrail@DPDK-Compute1,contrail@DPDK-Compute2, contrail@DPDK-Compute3]}]}, {running_nodes,[contrail@DPDK-Compute3,contrail@DPDK-Compute2, contrail@DPDK-Compute1]}, {cluster_name,<<"[email protected]">>}, {partitions,[]}, {alarms,[{contrail@DPDK-Compute3,[]}, {contrail@DPDK-Compute2,[]}, {contrail@DPDK-Compute1,[]}]}] [root@DPDK-Compute1 ~]#
Check contrail-status on all three controllers. As shown, all services are seen to be running smoothly:
contrail-status
[root@DPDK-Compute1 ~]# contrail-status -xd . . == Contrail control == control: active nodemgr: active named: active dns: active == Contrail config-database == nodemgr: active zookeeper: active RabbitMQ: active cassandra: active == Contrail config == svc-monitor: active nodemgr: active device-manager: active api: active schema: active == Contrail webui == web: active job: active [root@DPDK-Compute1 ~]#
To determine the actual cause, check the Service monitor logs:
vmi = <svc_monitor.config_db.VirtualMachineInterfaceSM object> vmi.aaps = [{'address_mode': 'active-standby', 'ip': {'ip_prefix': '203.47.180.3', 'ip_prefix_len': 32}}, {'address_mode': 'active-standby', 'ip': {'ip_prefix': '2001:8006:3532:206::3', 'ip_prefix_len': 128}}] idx = 0 aaps = [{'address_mode': 'active-standby', 'ip': {'ip_prefix': '203.47.180.3', 'ip_prefix_len': 32}, 'mac': None}, {'address_mode': 'active-standby', 'ip': {'ip_prefix': '2001:8006:3532:206::3', 'ip_prefix_len': 128}, 'mac': None}] <type 'exceptions.KeyError'>: 'mac'
As shown above, the logs indicate some python errors ( <type 'exceptions.KeyError'> : 'mac' ) related to MAC-ADDR when new Allowed Address Pairs (AAPS) were being created. A closer review shows that "vmi.aaps" is missing a "mac" key, which is unusual.
<type 'exceptions.KeyError'> : 'mac'
Restart the Service monitor (svc_monitor) service and check whether the reset has resolved the abnormal unacknowledged messages in the "svc_monitor" RabbitMQ queue and the Python errors reported by "svc_monitor".
Note: It is recommended that you plan to restart svc_monitor in a production setup during a Maintenance Window after confirming with the customer.
[root@DPDK-Compute1 ~]# docker restart svc-monitor . . [root@DPDK-Compute1 ~]#
The svc_monitor service has now been restarted.
After the restart has completed successfully, check the status of the RabbitMQ errors and the Python errors reported in the logs. In this case:
The RabbitMQ "closing AMQP connection" errors in RabbitMQ.log have stopped The svc_monitor logs also stopped reporting Python errors while creating Allowed Address Pairs It is assumed that the RabbitMQ queue was also reset, because it shows a normal count of messages, for example 0 messages
The RabbitMQ "closing AMQP connection" errors in RabbitMQ.log have stopped
RabbitMQ.log
The svc_monitor logs also stopped reporting Python errors while creating Allowed Address Pairs
It is assumed that the RabbitMQ queue was also reset, because it shows a normal count of messages, for example 0 messages
[root@DPDK-Compute1 ~]# docker exec -it contrail_config_RabbitMQ RabbitMQctl list_queues Listing queues svc_monitor.DPDK-Compute3.example.com 0 . . [root@DPDK-Compute1 ~]#
All other channels are also seen to report a normal count of messages
[root@DPDK-Compute1 ~]# docker exec -it contrail_config_RabbitMQ RabbitMQctl list_channels Listing channels <[email protected]> guest 1 0 <[email protected]> guest 1 0 . . <[email protected]> guest 1 0 <[email protected]> guest 1 0 [root@DPDK-Compute1 ~]#
Note: If the above steps do not resolve the issue, contact Support to investigate the problem further.