Description

Contrail uses the Apache ZooKeeper process to maintain synchronization of Contrail configuration, analytics and database running on the different instances of controllers. This article provides a list of helpful points related to Contrail and the Apache ZooKeeper process that can be used for troubleshooting.

Symptoms

 Provide troubleshooting tips on the Apache ZooKeeper process for Contrail users.

Solution

 

ZooKeeper helps in :

  1. Election: Services like service monitor, device monitor, contrail schema are active-backup. ZooKeeper helps in election of active service. Apart from these three services, all other Contrail services are active-active.
  2. Locking: ZooKeeper locks resources to avoid data inconsistency.
  3. Allocation of resources: ZooKeeper helps allocate resources like IP addresses. In case of concurrent requests from multiple services for same resource, it makes sure the resources are uniquely utilized.

Helpful points:

  • Ports 2888 and 3888 are the default ports. This info is stored in /etc/zookeeper/conf/zoo.cfg . The configuration should be consistent across all controller nodes. Make sure the process is listening on that port on all controller nodes:

    #netstat –lnp | grep 2181

  • Each controller instance gets a unique ZooKeeper ID stored in /etc/zookeeper/conf/myid . Make sure it is unique on every controller.

  • Only one controller instance is in leader mode. All others are in follower mode. To check which controller is in what mode, run the following commands on the controller CLI:

    Telnet to port 2180 to connect to ZooKeeper:

    root@ccpe-8:~# telnet localhost 2181 
    Trying ::1...
    Connected to localhost.
    Escape character is '^]'.

    When the connection is established, enter the command "stat":

    stat                                    
    ZooKeeper version: 3.4.5--1, built on 06/10/2013 17:26 GMT
    Clients:
    /10.85.29.9:38238[1](queued=0,recved=570834,sent=570834)
    /10.85.29.9:38248[1](queued=0,recved=570846,sent=570846)
    /0:0:0:0:0:0:0:1:48017[0](queued=0,recved=1,sent=0)

    Latency min/avg/max: 0/0/3
    Received: 1141692
    Sent: 1141691
    Connections: 3
    Outstanding: 0
    Zxid: 0x600000062
    Mode: leader
    Node count: 341
    Connection closed by foreign host.
    root@ccpe-8:~#

  • ZooKeeper also depends on Java. Current versions of Contrail (2.X and 3.0) support JDK version1.7.0_79.

    #java -version
    java version "1.7.0_79"
    OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
    OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

  • ZooKeeper logs: ZooKeeper server logs are located at /var/log/ZooKeeper/ZooKeeper.log on the controller nodes. The Contrail API process uses ZooKeeper; the logs are located at /var/log/contrail/api-0-zk.log on the controller node.

Related Information