Description

Contrail  Rabbitmq external containers were continuously restarting.

Symptoms

Contrail Rabbitmq external containers continuously restart as the mnesia db looks to be corrupted.

Solution

Perform mnesia db cleanup on ALL the contrail controllers, use the bellow steps:

  1. Stop Contrail Config API instances / containers on all the contrail controllers— this is important to clear the internal Config API queues and cache.
  2. Stop RabbitMQ containers on all the contrail nodes.
  3. Locate the mnesia _data directory:
    1. Run docker ps and identify the RabbitMQ container (usually named external_rabbitmq). Do not pick any kolla RabbitMQ container (that’s OpenStack and out of scope).
    2. Run docker inspect <rabbitmq_container_id> | grep "_data" to find the volume path, e.g. /var/lib/docker/volumes/<volume_id>/_data.
    3. cd into that path; you will find the _mnesia directory — this is the mnesia DB.
  4. Move the mnesia DB on all three nodes to /tmp (do not delete; moving provides a backup).
  5. Start RabbitMQ containers on all the contrail controller nodes.
  6. Start the Contrail Config API instances / containers on all the contrail controllers.
  7. Wait for 10-15 mins and the issue should be resolved.

Modification History

2025-11-03 : Article Created