This article explains the issue seen with respect to rabbitmq queues in Contrail version 1910 as well as in Contrail cloud 13.2 (which is also using release 1910).
== Contrail control == control: initializing (Number of connections:2, Expected:3 Missing: Database:Cassandra, IFMap Server End-Of-RIB not computed, No BGP configuration for self) nodemgr: active named: active dns: active
== Contrail config-database == nodemgr: active zookeeper: active rabbitmq: active cassandra: active
== Contrail config == svc-monitor: backup nodemgr: active device-manager: backup api: active schema: backup
== Contrail webui == web: active job: active
== Contrail analytics == nodemgr: active api: active collector: initializing (Number of connections:6, Expected:7 Missing: Database:Cassandra)
== Contrail analytics-alarm == nodemgr: active kafka: active alarm-gen: active
== Contrail analytics-snmp == snmp-collector: active nodemgr: active topology: active
RabbitMq client tries to delete the 'old' queue at restart, but only if it is not empty and not in use. In some cases, the queue was not empty because it does not get deleted. Earlier, with exclusive queue flag, it can't be used further because it was locked by previous connection. After changing the flag to non-exclusive, DeclareQueue function gives an error if flags are changed on the queue. This happens in the case where some config gets pushed in the queue during restart of control node.
With 1911 fix - https://review.opencontrail.org/c/Juniper/contrail-common/+/54877 , the code will "always" delete old queue even if there are messages sitting in it. With this change, old queue will always get deleted.
Check container errorlogs - control_control_1 (contrail controller) and analytics_collector_1 (contrail collector) and find matching matching as below:
Logs can be checked using 'docker inspect control_control_1 | grep log ' or 'docker logs control_control_1'
'docker inspect control_control_1 | grep log
'docker logs control_control_1'
control_control_1.log indicates below: RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-control.overcloudo4z-cc-1.tenant.cc13-testbed-1.juniper.net' in vhost '/' not empty RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-control.overcloudo4z-cc-1.tenant.cc13-testbed-1.juniper.net' in vhost '/' not empty
control_control_1.log indicates below:
RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-control.overcloudo4z-cc-1.tenant.cc13-testbed-1.juniper.net' in vhost '/' not empty RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-control.overcloudo4z-cc-1.tenant.cc13-testbed-1.juniper.net' in vhost '/' not empty
'docker inspect
analytics_collector_1
| grep log'
'docker logs
analytics_collector_1'
analytics_collector.log indicates below: RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-collector.overcloudo4z-ca-1.cc13-testbed-1.juniper.net' in vhost '/' not empty RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-collector.overcloudo4z-ca-1.cc13-testbed-1.juniper.net' in vhost '/' not empty
analytics_collector.log indicates below:
RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-collector.overcloudo4z-ca-1.cc13-testbed-1.juniper.net' in vhost '/' not empty RabbitMQ SM: Caught exception while connecting to RabbitMQ: <ip>:5673 : channel error: 406: AMQP_QUEUE_DELETE_METHOD caused: PRECONDITION_FAILED - queue 'contrail-collector.overcloudo4z-ca-1.cc13-testbed-1.juniper.net' in vhost '/' not empty
Permanently, this issue will be fixed in Contrail release 1911.
As workaround, the following can be implemented:
Note: Stop control_control_1 container and analytics_collector_1 container from all nodes using command 'docker stop <container name> ' before implementing the workaround.
'docker stop <container name> '
docker exec -it contrail_config_rabbitmq bash (Note: Container name may vary as per environment eg config_database_rabbitmq_1)
Purge those queues which are seen in control_control_1.log as well as analytics_collector.log manually using the command, 'rabbitmqctl purge_queue <queue name> '
'rabbitmqctl purge_queue <queue name> '
rabbitmqctl list_queues | grep collector rabbitmqctl purge_queue contrail-collector.overcloudo4z-ca-0.cc13-testbed-1.juniper.net
Repeat rabbitmqctl purge for all the queues seen in control_control_1.log and analytics_collector.log
Restart contrail-controller and analytics_collector_1 using the command, ' docker restart <container name> '
docker restart <container name> '
docker restart control_control_1 docker restart analytics_collector_1
2019-12-07: Added note to stop containers before implementing workaround.