Description

Due to a software defect in 2.x and 3.x releases, contrail discovery service may be stuck in initializing state if too many TCP connections to port 9160 have been opened and used up socket quota allocated by host operating system. This article describes such failure, its possible cause and the way to recover from such failure state. There are a few JTAC cases sharing the same trigger, to name a few, 2017-0926-0688, 2018-0104-0636, and 2018-0303-0029.

Symptoms

If we observe below discovery behaviors at the same time, it means a software defect has been triggered.

​1. contrail-discovery service is stuck in initializing state, saying connection down

root@aio3260:~# contrail-status -d | grep discovery
contrail-discovery:0          i nitializing (Collector, Discovery:Collector[Subscribe - Connection Error] connection down) ​ pid 10654, uptime 13 days, 15:42:31

2. Cassandra DB is up and running
root@aio3260:/var/log/contrail# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  10.174.8.4  105.17 MB  256          ?       e34f60c3-ab08-436f-9412-12ce766c41ee  rack1

​3. Below error trace is logged in /var/log/contrail/contrail-discovery-0-stdout.log​

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/baseserver.py", line 140, in _do_read
    args = self.do_read()
  File "/usr/lib/python2.7/dist-packages/gevent/server.py", line 93, in do_read
    client_socket, address = self.socket.accept()
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 307, in accept
    client_socket, address = sock.accept()
error: [Errno 24] Too many open files​

​4. lsof check on established TCP connections from discovery process (get pid from contrail-status -d command) to port 9160 (Cassandra DB thrift port) returns a large number,
root@aio3260:~# contrail-status -d | grep discovery
contrail-discovery:0          initializing ( Collector, Discovery:Collector[Subscribe - Connection Error] connection down )pid 10654 , uptime 13 days, 15:42:31    

root@aio3260:~# lsof -a -p 10654 |grep -c 9160
10311
 

Solution

A possible trigger is when Cassandra DB is temporarily down/not responding, discovery service is still trying to connect to Cassandra continuously. Therefore, large number of TCP sockets(files) will be consumed in such case in a short time. After Cassandra DB recovers, since all system allocated sockets have been used up, discovery service is unable to get any available socket to use. This issue has been tracked under ​launchpad id 1727060 (https://bugs.launchpad.net/juniperopenstack/%2Bbug/1727060), which has been fixed in more recent releases such as 3.0.3.6, 3.2.7.0, and onward. 

If an upgrade to a software release with the aforementioned launchpad item fix is not possible, we can do the following to try to recover the discovery server on config nodes,

​1. Make sure Cassandra DB is up and running via nodetool status command. If not, do service contrail-database start to bring it up;

​2. Bounce discovery service by service contrail-discovery restart command to clean all existing discovery TCP connections to port 9160.
root@aio3260:~# service contrail-discovery restart
contrail-discovery:0: stopped
contrail-discovery:0: started

Related Information