Often it may be seen that several TCP connection trials between Contrail processes (such as web-ui and redis server) are causing de facto DoS (Denial-of-Service) for some Contrail analytics processes such as analytics-api, alarm-gen, and contrail-collector.

This article describes the symptoms of this issue, a possible workaround, and the software fix that is available.

Often when contrail-status is run, multiple analytics processes have been observed to fail or to be stuck in initializing state:

# sudo contrail-status -d                                                                           
== Contrail Analytics ==
supervisor-analytics:         active
contrail-alarm-gen            active              pid 10527, uptime 7:50:29             
contrail-analytics-api        active              pid 10526, uptime 7:50:29             
contrail-analytics-nodemgr    active              pid 10521, uptime 7:50:29             
contrail-collector            failed              Exited too quickly (process log may have details)
contrail-query-engine         initializing (Collector connection down)pid 10525, uptime 7:50:29             
contrail-snmp-collector       active              pid 10522, uptime 7:50:29             
contrail-topology             active              pid 15013, uptime 4:02:56             

​Typically in the ​contrail-api log files, a " max number of clients reached " exception is reported for the current redis clients.

01/04/2019 12:23:03 PM [contrail-analytics-api]: [172.16.0.17:6379] AlarmGen 0,12 Exception ResponseError in uve stream proc. Arguments:
('max number of clients reached',) : traceback Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 353, in _run
pb.subscribe('AGPARTPUB:%s:%d' % (inst, part))
File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1696, in subscribe
return self.execute_command('SUBSCRIBE', *channels)
File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1643, in execute_command
connection.send_command(*args)
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 310, in send_command
self.send_packed_command(self.pack_command(*args))
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 292, in send_packed_command
self.connect()
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 238, in connect
self.on_connect()
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 274, in on_connect
if nativestr(self.read_response()) != 'OK':
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 320, in read_response
raise response
ResponseError: max number of clients reached

The following redis-cli command indicates that connection quota has been exhausted.

# redis-cli -h 127.0.0.1 -p 6379 info
ERR max number of clients reached
# redis-cli -h 127.0.0.1 -p 6379 --stat
ERROR: ERR max number of clients reached
# redis-cli -h 127.0.0.1 -p 6379 monitor
(error) ERR max number of clients reached
Error: Connection reset by peer

​In general, the redis server allows up to 3,984 simultaneous connections, which can be checked by using the redis-cli command below:

​redis-cli config get maxclients
1) "maxclients"
2) "3984"

​However, in Contrail releases prior to release 3.2.14.0, tcp-keepalive is not set by default in the redis configuration file nor is the redis connection timeout value. This is, however, not a good practice to avoid zombie connections from taking up resources.

cat /etc/redis/redis.conf | egrep -A15 "TCP keepalive"
# TCP keepalive.
#
# If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence
# of communication. This is useful for two reasons:
#
# 1) Detect dead peers.
# 2) Take the connection alive from the point of view of network
#    equipment in the middle.
#
# On Linux, the specified value (in seconds) is the period used to send ACKs.
# Note that to close the connection the double of the time is needed.
# On other kernels the period depends on the kernel configuration.
#
# A reasonable value for this option is 60 seconds.
tcp-keepalive 0 

cat /etc/redis/redis.conf | egrep  -C1 "timeout"
# Close the connection after a client is idle for N seconds (0 to disable)
timeout 0

JCB-218835, which was opened to track this particular issue, was addressed in 3.2.14.0+ wherein a software fix was added in the provision.py file, which sets the above two parameters during cluster provisioning.

Meanwhile, a workaround is to explicitly set the aforementioned tcp-keepalive and timeout values to say 300 seconds and 43200 seconds (12 hours), respectively, and then restart the redis server to make these changes effective.