Description

By default, the redis-server does not enable the TCP KA (keepalive) and idle timeout.  This article explains how to turn on TCP KA (keepalive) and idle timeout, and provides a demo of the related behaviors that occur on Contrail 3.2.x and 3.0.x releases.

Symptoms

By default, the redis-server only allows up to 3984 max connections:

# redis-cli config get maxclients
  1. "maxclients"
  2. "3984"

While in the field production environment, this limited redis-server tcp connection number could be exhausted due to some DDoS attacks, network device bounce or malfunction.  The existing redis-server tcp connections could not be teared down due to the TCP KA and idle timeout features not enabled there. 

root@aio140:~# # cat /etc/redis/redis.conf | egrep -i "tcp-keepalive|timeout"
timeout 0
tcp-keepalive 0

Solution

Modify the tcp-keepalive and timeout setting in the file /etc/redis/redis.conf: 

root@aio140:~# cat /etc/redis/redis.conf | egrep -i "tcp-keepalive|timeout"
timeout 300
tcp-keepalive 60

root@aio140:~# vim /etc/redis/redis.conf
...

Specify the path for the Unix socket that will be used to listen for incoming connections. There is no default, so Redis will not listen on a Unix socket when not specified.

# unixsocket /var/run/redis/redis.sock
# unixsocketperm 755

Close the connection after a client is idle for N seconds (0 to disable) timeout 300 TCP keepalive. If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence of communication. This is useful for two reasons:
  1. Detect dead peers.
  2. Take the connection alive from the point of view of network equipment in the middle. On Linux, the specified value (in seconds) is the period used to send ACKs. Note that to close the connection, double the time is needed. On other kernels, the period depends on the kernel configuration. A reasonable value for this option is 60 seconds.
tcp-keepalive 60
...

Restart the redis-server to activate the tcp-keepalive and timeout setting: 

root@aio140:~# service redis-server restart
redis-server stop/waiting
redis-server start/running, process 4151


As a comparison, see the difference shown below for the TCP connection:

Before the modification: 
root@aio140:~# netstat -naplo | egrep -i 6379 | egrep -i \.158\:
tcp        0      0 10.173.150.140:6379     10.173.150.158:45530    ESTABLISHED 18568/redis-server  off (0.00/0/0)

After the modification and redis-server restart: 
root@aio140:~# netstat -naplo | egrep -i 6379 | egrep -i \.158\:
tcp        0      0 10.173.150.140:6379     10.173.150.158:45530    FIN_WAIT2   -                timewait (25.31/0/0)
tcp        0      0 10.173.150.140:6379     10.173.150.158:45534    ESTABLISHED 21856/redis-server  keepalive (38.79/0/0)


More tests shown below to check the TCP KA and idle timeout behaviors:

  1. The TCP KA is every 1 min. per setting and redis-server will tear down a TCP connection immediately after missed 1 KA there: 

    root@aio140:~# date;netstat -nap | egrep -i 6379 | egrep -i \.158\:
Wed Feb 27 13:37:18 PST 2019

root@aio140:~# date;netstat -nap | egrep -i 6379 | egrep -i \.158\:
Wed Feb 27 13:38:48 PST 2019
tcp        0      0 10.173.150.140:6379     10.173.150.158:60592    ESTABLISHED 4151/redis-server 1

root@aio140:~# tcpdump -nei vhost0 "tcp port 6379"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vhost0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:38:09.046312 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 74: 10.173.150.158.60592 > 10.173.150.140.6379: Flags [S], seq 928348488, win 29200, options [mss 1460,sackOK,TS val 261299050 ecr 0,nop,wscale 7], length 0
13:38:09.046380 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 74: 10.173.150.140.6379 > 10.173.150.158.60592: Flags [S.], seq 244637735, ack 928348489, win 28960, options [mss 1460,sackOK,TS val 259599176 ecr 261299050,nop,wscale 7], length 0
13:38:09.046496 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 66: 10.173.150.158.60592 > 10.173.150.140.6379: Flags [.], ack 1, win 229, options [nop,nop,TS val 261299050 ecr 259599176], length 0
13:39:09.173765 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 66: 10.173.150.140.6379 > 10.173.150.158.60592: Flags [.], ack 1, win 227, options [nop,nop,TS val 259614208 ecr 261299050], length 0
...

root@aio140:~# date;netstat -nap | egrep -i 6379 | egrep -i \.158\:
Wed Feb 27 14:38:23 PST 2019

  2. The idle timeout is set for every 5 minutes, and redis-server will tear down a connection after it is idle for 5 minutes: 

    root@aio140:~# date;netstat -nap | egrep -i 6379 | egrep -i \.158\:
Wed Feb 27 14:45:36 PST 2019
tcp        0      0 10.173.150.140:6379     10.173.150.158:60650    ESTABLISHED 4151/redis-server 1

root@aio140:~# tcpdump -nei vhost0 "tcp port 6379"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vhost0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:45:26.300414 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 74: 10.173.150.158.60650 > 10.173.150.140.6379: Flags [S], seq 846053790, win 29200, options [mss 1460,sackOK,TS val 262308363 ecr 0,nop,wscale 7], length 0
14:45:26.300525 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 74: 10.173.150.140.6379 > 10.173.150.158.60650: Flags [S.], seq 2513521079, ack 846053791, win 28960, options [mss 1460,sackOK,TS val 260608489 ecr 262308363,nop,wscale 7], length 0
14:45:26.300679 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 66: 10.173.150.158.60650 > 10.173.150.140.6379: Flags [.], ack 1, win 229, options [nop,nop,TS val 262308363 ecr 260608489], length 0

14:46:26.421806 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 66: 10.173.150.140.6379 > 10.173.150.158.60650: Flags [.], ack 1, win 227, options [nop,nop,TS val 260623520 ecr 262308363], length 0
14:46:26.422133 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 66: 10.173.150.158.60650 > 10.173.150.140.6379: Flags [.], ack 1, win 229, options [nop,nop,TS val 262323393 ecr 260608489], length 0
14:47:26.453774 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 66: 10.173.150.140.6379 > 10.173.150.158.60650: Flags [.], ack 1, win 227, options [nop,nop,TS val 260638528 ecr 262323393], length 0
14:47:26.453972 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 66: 10.173.150.158.60650 > 10.173.150.140.6379: Flags [.], ack 1, win 229, options [nop,nop,TS val 262338401 ecr 260608489], length 0
14:48:26.489804 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 66: 10.173.150.140.6379 > 10.173.150.158.60650: Flags [.], ack 1, win 227, options [nop,nop,TS val 260653537 ecr 262338401], length 0
14:48:26.490057 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 66: 10.173.150.158.60650 > 10.173.150.140.6379: Flags [.], ack 1, win 229, options [nop,nop,TS val 262353410 ecr 260608489], length 0
14:49:26.517750 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 66: 10.173.150.140.6379 > 10.173.150.158.60650: Flags [.], ack 1, win 227, options [nop,nop,TS val 260668544 ecr 262353410], length 0
14:49:26.518014 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 66: 10.173.150.158.60650 > 10.173.150.140.6379: Flags [.], ack 1, win 229, options [nop,nop,TS val 262368417 ecr 260608489], length 0
14:50:26.549837 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 66: 10.173.150.140.6379 > 10.173.150.158.60650: Flags [.], ack 1, win 227, options [nop,nop,TS val 260683552 ecr 262368417], length 0
14:50:26.550201 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 66: 10.173.150.158.60650 > 10.173.150.140.6379: Flags [.], ack 1, win 229, options [nop,nop,TS val 262383425 ecr 260608489], length 0
14:50:27.090202 52:54:00:4d:92:bf > 52:54:00:cf:be:e8, ethertype IPv4 (0x0800), length 66: 10.173.150.140.6379 > 10.173.150.158.60650: Flags [F.], seq 1, ack 1, win 227, options [nop,nop,TS val 260683687 ecr 262383425], length 0
14:50:27.094282 52:54:00:cf:be:e8 > 52:54:00:4d:92:bf, ethertype IPv4 (0x0800), length 66: 10.173.150.158.60650 > 10.173.150.140.6379: Flags [.], ack 2, win 229, options [nop,nop,TS val 262383562 ecr 260683687], length 0
^C
15 packets captured
15 packets received by filter
0 packets dropped by kernel

root@aio140:~# date;netstat -nap | egrep -i 6379 | egrep -i \.158\:
Wed Feb 27 15:05:30 PST 2019

In summary, the redis-server now could tear down a KA missed connection or an idle connection after modifying the tcp-keepalive and timeout setting in the file /etc/redis/redis.conf and then restarting the redis-server service. This will benefit the redis-server service from the TCP connection exhaustion issue.