In the Contrail Cloud environment, it is observed that the 'rabbitmqctl list_queues' command always get stuck at following last line:
'rabbitmqctl list_queues'
[root@overcloud-cc-0 ~]# docker exec -it contrail_config_rabbitmq rabbitmqctl list_queues Timeout: 60.0 seconds ... Listing queues for vhost / ... name messages ...... {:badrpc, {:timeout, 60.0, "Some queue(s) are unresponsive, use list_unresponsive_queues command."}} <---
Since RabbitMQ release 3.7.4, the rabbitmq CLI tools parallelize the querying of information from cluster members. In order to receive stream data back, the CLI instance binds and registers itself on an available port (default between 35672 and 35682, inclusive). If these ports are firewalled off, then rabbitmqctl commands such as 'list_queues' will hang waiting for data from remote cluster members. That is why "timeout" is seen in the above example. The issue is fixed in higher releases.
'list_queues'
This issue is seen in Contrail Cloud 13.2.1 with Contrail Networking 1912. The rabbitmq version used in this release is 3.7.20, which can be verified with the following command:
root@overcloudsd9-cc-2:/# rabbitmqctl status Status of node contrail@overclouds-cc-2 ... [{pid,581}, {running_applications, [{rabbitmq_management,"RabbitMQ Management Console","3.7.20"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.20"}, {amqp_client,"RabbitMQ AMQP Client","3.7.20"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.20"}, {rabbit,"RabbitMQ","3.7.20"}, {rabbit_common, "Modules shared by rabbitmq-server and rabbitmq-erlang-client", "3.7.20"}, {cowboy,"Small, fast, modern HTTP server.","2.6.1"},
A workaround is to ensure ports 35672 - 35682 are open in all three controllers. Enforce this with the 'iptables' command:
'iptables'
iptables -I INPUT 1 -p tcp --match multiport --dports 35672:35682 -j ACCEPT