Contrail uses the messaging services of rabbitmq when coordinating operations and status information among several of its services. Any inconsistencies within a rabbitmq cluster may render the different services of Contrail in a non-working state. rabbitmqctl is a command-line tool that is frequently used for managing and troubleshooting a rabbitmq cluster of nodes. It can be used to perform several diagnostic and data-collection actions by connecting to the target rabbitMQ node on a dedicated CLI communication port.
rabbitmqctl
If any of the Contrail services fails due to its dependencies on rabbitmq, it is typically reflected in the output of the contrail-status command or in the Contrail logs of the corresponding service that is failing. In such cases, it is advisable to take a look at the status of the rabbitmq cluster and verify the health of the different components of rabbitmq.
contrail-status
This article lists some of the useful rabbitmqctl commands that you should keep handy when troubleshooting issues in Contrail that are related to rabbitmq.
Some useful rabbitmqctl commands are given below:
#rabbitmqctl cluster_status
This command displays all the nodes in the cluster grouped by node type, together with the currently running nodes. This command is useful in determining the overall health of the rabbitmq cluster.
If any of the nodes are down, then cluster_status output will show nodedown in the alarm section:
cluster_status
nodedown
To print the status information for a particular node, use the rabbitmqctl status command. This command will display the status of the rabbitmq server/broker node on which the command is run:
rabbitmqctl status
#rabbitmqctl list_queues
This is one of the rabbitmq topology introspect commands that is useful to see the list of available queues. By default, this command returns a list of queues with their corresponding name and queue depth. Queue depth indicates the sum of ready and unacknowledged messages in a queue.
#rabbitmqctl list_connections
This command lists the state of TCP connections between each of the rabbitmq servers/broker nodes and the clients. The clients in this case will be different Contrail services that are using the messaging services of rabbitmq.
This is a cluster wide command, which means running this command on any nodes gets the status across the entire cluster.
Example
# rabbitmqctl list_connections Listing connections guest 172.18.101.102 53910 running guest 172.18.101.101 48842 running guest 172.18.101.101 48844 running guest 172.18.101.101 48845 running guest 172.18.101.101 48846 running guest 172.18.101.101 48847 running guest 172.18.101.101 48849 running guest 172.18.101.101 48850 running guest 172.18.101.101 48851 running guest 172.18.101.101 48852 running guest 172.18.101.101 48854 running guest 172.18.101.101 48853 running ---------------Output Skipped------------------
In the above output, guest is the username of the rabbitmq user the client is using to connect to the server. IP addresses are the IP addresses of the rabbitmq servers along with the TCP port numbers on the servers the clients are connecting to. The last column of the output indicates the state of the TCP connections. The connection state could be one of the following:
In the above output, guest is the username of the rabbitmq user the client is using to connect to the server.
guest
IP addresses are the IP addresses of the rabbitmq servers along with the TCP port numbers on the servers the clients are connecting to.
The last column of the output indicates the state of the TCP connections. The connection state could be one of the following:
#rabbitmqctl list_bindings
In rabbitmq, messages are not published directly to a queue. Instead, the producer sends messages to an exchange. An exchange accepts messages from the producer application and routes them to the appropriate message queues. A binding is a "link" that is set up to bind a queue to an exchange. This command output lists all the available queues along with their linked exchanges. If an exchange does not exist for a queue, then the messages sent to that queue will not be received by the subscribers of that queue.
In rabbitmq, messages are not published directly to a queue. Instead, the producer sends messages to an exchange. An exchange accepts messages from the producer application and routes them to the appropriate message queues. A binding is a "link" that is set up to bind a queue to an exchange.
This command output lists all the available queues along with their linked exchanges. If an exchange does not exist for a queue, then the messages sent to that queue will not be received by the subscribers of that queue.
#rabbitmqctl list_channels
A channel is a virtual connection inside a TCP connection. When a Contrail service is publishing or consuming messages from a queue, it is done over a channel. Thus, it is important at the time of troubleshooting to validate if an active channel exists between the client and the broker/server. There can be multiple channels established over a single TCP connection.
A channel is a virtual connection inside a TCP connection. When a Contrail service is publishing or consuming messages from a queue, it is done over a channel. Thus, it is important at the time of troubleshooting to validate if an active channel exists between the client and the broker/server.
There can be multiple channels established over a single TCP connection.
In the above output:
1st column: Is the rabbitmq PID of the process associated with the connection 2nd column: Is the username associated with the channel 3rd column: Is the number of consumers retrieving messages via the channel 4th column: Indicates unacknowledged messages, which are the number of messages delivered via this channel but not yet acknowledged Each channel can also be identified based on the TCP connection it resides in; we can view this by running the above command with the "name" option.
1st column: Is the rabbitmq PID of the process associated with the connection
2nd column: Is the username associated with the channel
3rd column: Is the number of consumers retrieving messages via the channel
4th column: Indicates unacknowledged messages, which are the number of messages delivered via this channel but not yet acknowledged
Each channel can also be identified based on the TCP connection it resides in; we can view this by running the above command with the "name" option.
#rabbitmqctl list_consumers
This command lists all the subscriptions to a queue´s message stream.
Each printed line is separated by tab characters and shows the following: The name of the queue subscribed to The PID of the channel process via which the subscription was created and is managed The consumer tag which uniquely identifies the subscription within a channel A boolean indicating whether acknowledgements are expected for messages delivered to this consumer
Each printed line is separated by tab characters and shows the following:
The name of the queue subscribed to
The PID of the channel process via which the subscription was created and is managed
The consumer tag which uniquely identifies the subscription within a channel
A boolean indicating whether acknowledgements are expected for messages delivered to this consumer
#rabbitmqctl eval "rabbit_diagnostics:maybe_stuck()".
This command periodically samples stack traces of all rabbitmq PIDs ("lightweight threads" also called "Erlang Processes") on the node and reports the processes for which stack trace samples are identical. Identical samples may indicate that the process is not making any progress. However, it is not necessarily an indication of a problem.
This command periodically samples stack traces of all rabbitmq PIDs ("lightweight threads" also called "Erlang Processes") on the node and reports the processes for which stack trace samples are identical.
Identical samples may indicate that the process is not making any progress. However, it is not necessarily an indication of a problem.
2020-07-03: Fixed typo for command 6 (added rabbitmqctl list_consumers)
2020-07-14: Fixed Step 7, added . at the end of the command