Description

RabbitMQ queue names are built from the hostname. Hence, changing the hostname can affect the RabbitMQ cluster.  This article explains what to  check in RabbitMQ cluster after such changes.

Symptoms

Contrail RabbitMQ subscriber queue names are created from the subscriber hostname and bound to vnc_config exchange to get messages published by contrail-api. Relevant subscribers' queue is deleted when subscribers' service is started while connecting to the RabbitMQ. If the hostname is changed, the relevant queue name is changed and will not match the previous queue name. Therefore, old queues will not be deleted. They will continue to get messages as it is still bound to the exchange. As result, the number of messages will grow. Over time time, it can result in slow messages propagating from the publisher to the subscriber.

RabbitMQ subscribers are all other config services, control node, and some analytics services. One of the symptoms can be missing system route target for a routing instance when a new virtual network is created, or control node missing new updates which can be reflected in missing networking connectivity for workloads in virtual networks created during the issue time.

Solution

A known behavior in 4.x, 19xx and 20xx releases is that we can end up having ghost queues in some corner cases caused by /etc/hosts changes. 

A queue name is obtained from /etc/hosts file by the following code instructions when a service is started and while trying to connect to RabbitMQ:

q_name = 'vnc_config.%s-%s' % (socket.getfqdn(host_ip), listen_port)
>>> import socket
>>> socket.getfqdn("10.0.0.14")
'overcloud-contrailcontroller-2.tenant.localdomain'

In a corner case, when the /etc/hosts file is changed by modifying or adding another domain name, the system can end up with a new queue after restarting the service. That can happen if a new domain name is changed or inserted before the existing one (for the same IP).

Subscriber, the relevant restarted service, will be listening to the “New Queue”. The “Old Queue” will not be deleted after the service restarts and it will not have any subscribers. On the other side, it is still bound to the vnc_config exchange and will continue getting messages from the exchange.

Queues can be listed by using rabbitmqctl list_queues command or by using RabbitMQ Management Plugin. Here is an example with resulted queues after changing the name to fully qualified domain name:

icmlw-p1-jcontvm-001a:contrail-snmp-collector:0.icmlw-p1-jcontvm-001a 284141
icmlw-p1-jcontvm-001b:contrail-snmp-collector:0.icmlw-p1-jcontvm-001b 284143
icmlw-p1-jcontvm-001c:contrail-snmp-collector:0.icmlw-p1-jcontvm-001c 284147
icmlw-p1-jcontvm-001a.itpc.uk.pri.o2.com:contrail-snmp-collector:0.icmlw-p1-jcontvm-001a.itpc.uk.pri.o2.com 0
icmlw-p1-jcontvm-001b.itpc.uk.pri.o2.com:contrail-snmp-collector:0.icmlw-p1-jcontvm-001b.itpc.uk.pri.o2.com 0
icmlw-p1-jcontvm-001c.itpc.uk.pri.o2.com:contrail-snmp-collector:0.icmlw-p1-jcontvm-001c.itpc.uk.pri.o2.com 0

This old queue must be deleted manually to prevent large numbers of messages building up and affecting the RabbitMQ cluster.

In the example above, all old queues 'icmlw-p1-jcontvm-001x:contrail-snmp-collector:0.icmlw-p1-jcontvm-001x' must be manually deleted. The RabbitMQ queue can be deleted either by using rabbitmqctl delete_queue command or from a RabbitMQ Management Plugin.

Related Information