This article describes an issue related to Contrail Analytics where the Analytics API goes into init mode after the Kafka partition is reported to be down. While the Analytics API remains in init state, other dependent components in Contrail also moves into init state, having observed the API to be down.
Users may encounter instances when the Contrail Analytics API goes into initializing state following an "UvePartitions:UVE-Aggregation[Partitions:X] connection down" error. When this happens, the Analytics API will not get restored even after you restart supervisor-analytics. Further you will observe the same behavior across all Contrail controllers.
Note: This issue was observed on Contrail 3.x and the resolution mentioned has been tested on the same version.
root@controller-1:~#contrail-status == Contrail Analytics == supervisor-analytics: active contrail-alarm-gen active contrail-analytics-api initializing (UvePartitions:UVE-Aggregation[Partitions:26] connection down) contrail-analytics-nodemgr active contrail-collector active contrail-query-engine active contrail-snmp-collector active contrail-topology active
The issue is caused by negative leader election in Kafka for one or some of topics due to which Kafka partition will be reported as down. To identify the issue, the following procedure was used:
Zookeeper status showed the leader using the below command on contrail controllers:
root@controller-1:~# netstat -plantu | grep 2181 | grep LISTEN tcp6 0 0 :::2181 :::* LISTEN 1468/java root@controller-1:~# echo stat | nc 127.0.0.1 2181 Zookeeper version: 3.4.5--1, built on 06/10/2013 17:26 GMT <clipped> Latency min/avg/max: 0/0/14 Received: 5391695 Sent: 5391715 Connections: 13 Outstanding: 0 Zxid: 0x8700000631 Mode: leader Node count: 2159 root@cont102:~#
Executed the commands below using leader zookeeper IP, which shows partitions reporting negative leader due to these partitions being marked as down in Kafka:
root@controller-1:~# for each in $(echo -uve-{0..29});do /usr/share/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --describe --topic $each;done Topic:-uve-0 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-0 Partition: 0 Leader: 26 Replicas: 26,27 Isr: 26,27 Topic:-uve-1 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-1 Partition: 0 Leader: -1 Replicas: 25,26 Isr: 26 Topic:-uve-2 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-2 Partition: 0 Leader: -1 Replicas: 25,26 Isr: Topic:-uve-3 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-3 Partition: 0 Leader: 26 Replicas: 27,26 Isr: 26 Topic:-uve-4 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-4 Partition: 0 Leader: -1 Replicas: 25,27 Isr: Topic:-uve-5 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-5 Partition: 0 Leader: -1 Replicas: 25,26 Isr: Topic:-uve-6 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-6 Partition: 0 Leader: -1 Replicas: 26,25 Isr: Topic:-uve-7 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-7 Partition: 0 Leader: -1 Replicas: 25,26 Isr: Topic:-uve-8 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-8 Partition: 0 Leader: 26 Replicas: 26,27 Isr: 26,27 Topic:-uve-9 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-9 Partition: 0 Leader: 25 Replicas: 27,25 Isr: 25 Topic:-uve-10 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-10 Partition: 0 Leader: -1 Replicas: 26,25 Isr: Topic:-uve-11 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-11 Partition: 0 Leader: 25 Replicas: 27,25 Isr: 25 Topic:-uve-12 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-12 Partition: 0 Leader: 26 Replicas: 26,27 Isr: 26 Topic:-uve-13 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-13 Partition: 0 Leader: -1 Replicas: 26,25 Isr: Topic:-uve-14 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-14 Partition: 0 Leader: -1 Replicas: 25,26 Isr: Topic:-uve-15 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-15 Partition: 0 Leader: -1 Replicas: 25,26 Isr: Topic:-uve-16 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-16 Partition: 0 Leader: 26 Replicas: 27,26 Isr: 26 Topic:-uve-17 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-17 Partition: 0 Leader: 26 Replicas: 26,27 Isr: 26,27 Topic:-uve-18 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-18 Partition: 0 Leader: -1 Replicas: 26,25 Isr: Topic:-uve-19 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-19 Partition: 0 Leader: -1 Replicas: 25,26 Isr: Topic:-uve-20 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-20 Partition: 0 Leader: 25 Replicas: 27,25 Isr: 25 Topic:-uve-21 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-21 Partition: 0 Leader: 26 Replicas: 26,27 Isr: 26,27 Topic:-uve-22 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-22 Partition: 0 Leader: -1 Replicas: 25,26 Isr: 26 Topic:-uve-23 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-23 Partition: 0 Leader: -1 Replicas: 26,25 Isr: Topic:-uve-24 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-24 Partition: 0 Leader: 25 Replicas: 27,25 Isr: 25 Topic:-uve-25 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-25 Partition: 0 Leader: -1 Replicas: 25,26 Isr: Topic:-uve-26 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-26 Partition: 0 Leader: 26 Replicas: 26,27 Isr: 26,27 Topic:-uve-27 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-27 Partition: 0 Leader: 25 Replicas: 27,25 Isr: 25 Topic:-uve-28 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-28 Partition: 0 Leader: -1 Replicas: 25,27 Isr: 27 Topic:-uve-29 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-29 Partition: 0 Leader: -1 Replicas: 26,25 Isr:
Once it is confirmed there are no leaders with the below command which describes each topics in kafka:
for each in $(echo -uve-{0..29});do /usr/share/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --describe --topic $each;done
Stop Kafka on all controller nodes
root@controller1:~# root@controller1:~# service kafka stop kafka: stopped root@controller1:~#
Delete consumers, brokers and workers zknodes from zookeeper.
/usr/share/zookeeper/bin/zkCli.sh execute rmr /consumers execute rmr /workers (if workers available) execute rmr /brokers root@controller1:~# /usr/share/zookeeper/bin/zkCli.sh Connecting to localhost:2181 <clipped> 2022-06-08 15:57:03,820 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1207] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0xff803b9acee20095, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] rmr /consumers [zk: localhost:2181(CONNECTED) 1] rmr /workers [zk: localhost:2181(CONNECTED) 2] rmr /brokers [zk: localhost:2181(CONNECTED) 3] service kafka start on all nodes root@controller1:~# service kafka start kafka: started root@controller1:~#
Confirm that there is a leader of each topic.
root@controller1:~# for each in $(echo -uve-{0..29});do /usr/share/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --describe --topic $each;done Topic:-uve-0 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-0 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-1 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-1 Partition: 0 Leader: 1 Replicas: 0,1 Isr: 1,0 Topic:-uve-2 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-2 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-3 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-3 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-4 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-4 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic:-uve-5 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-5 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic:-uve-6 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-6 Partition: 0 Leader: 2 Replicas: 2,1 Isr: 1,2 Topic:-uve-7 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-7 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-8 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-8 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-9 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-9 Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0 Topic:-uve-10 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-10 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-11 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-11 Partition: 0 Leader: 0 Replicas: 0,2 Isr: 2,0 Topic:-uve-12 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-12 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic:-uve-13 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-13 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-14 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-14 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic:-uve-15 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-15 Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0 Topic:-uve-16 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-16 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic:-uve-17 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-17 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic:-uve-18 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-18 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-19 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-19 Partition: 0 Leader: 2 Replicas: 2,1 Isr: 1,2 Topic:-uve-20 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-20 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic:-uve-21 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-21 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic:-uve-22 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-22 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-23 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-23 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic:-uve-24 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-24 Partition: 0 Leader: 2 Replicas: 2,1 Isr: 1,2 Topic:-uve-25 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-25 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-26 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-26 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic:-uve-27 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-27 Partition: 0 Leader: 2 Replicas: 2,1 Isr: 1,2 Topic:-uve-28 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-28 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic:-uve-29 PartitionCount:1 ReplicationFactor:2 Configs: Topic: -uve-29 Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0
Restart supervisor-analytics
root@controller1:~# service supervisor-analytics restart supervisor-analytics stop/waiting supervisor-analytics start/running, process 24294 root@controller1:~#
2022-08-16: Initial publication.