Description

When multiple contrail components need to establish connection with the zookeeper server, they will create sessions with zookeeper. To check client session information, cons command can be used.

Solution

The following command can be used to list the active zookeeper connections:

echo cons | nc localhost 2181

A sample output is shown below. The cons command provides information about client IP address, port number, session ID, and other packet statistics.  

# echo cons | nc localhost 2181

 /172.18.101.101:34539[1](queued=0,recved=189638,sent=189638,sid=0x16df603bab90009,lop=PING,est=1571790861463,to=10000,lcxid=0x29,lzxid=0xffffffffffffffff,lresp=1572385305088,llat=1,minlat=0,avglat=0,maxlat=6)
 /172.18.101.101:54438[1](queued=0,recved=297183,sent=297183,sid=0x26d9d24aec60004,lop=PING,est=1571790504492,to=6000,lcxid=0x34e,lzxid=0xffffffffffffffff,lresp=1572385307563,llat=0,minlat=0,avglat=0,maxlat=8)
 /172.18.101.102:47848[1](queued=0,recved=451,sent=451,sid=0x16df603bab90004,lop=PING,est=1571790503650,to=4000000,lcxid=0x4,lzxid=0xffffffffffffffff,lresp=1572385081512,llat=0,minlat=0,avglat=0,maxlat=2)
 /172.18.101.102:47830[1](queued=0,recved=4474,sent=4475,sid=0x16df603bab90002,lop=PING,est=1571790502921,to=400000,lcxid=0x6,lzxid=0xffffffffffffffff,lresp=1572385231932,llat=0,minlat=0,avglat=0,maxlat=3)
 /172.18.101.103:41946[1](queued=0,recved=4466,sent=4466,sid=0x26d9d24aec60008,lop=PING,est=1571790751550,to=400000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=1572385202475,llat=0,minlat=0,avglat=0,maxlat=1)
 /172.18.101.101:54406[1](queued=0,recved=4472,sent=4472,sid=0x16df603bab90005,lop=PING,est=1571790503787,to=400000,lcxid=0x4,lzxid=0xffffffffffffffff,lresp=1572385211672,llat=0,minlat=0,avglat=0,maxlat=2)
 /0:0:0:0:0:0:0:1:37451[0](queued=0,recved=1,sent=0)
 /172.18.101.102:47835[1](queued=0,recved=297168,sent=297168,sid=0x26d9d24aec60005,lop=PING,est=1571790503424,to=6000,lcxid=0xc8,lzxid=0xffffffffffffffff,lresp=1572385307415,llat=0,minlat=0,avglat=0,maxlat=15)
 /172.18.101.101:34518[1](queued=0,recved=189689,sent=189689,sid=0x16df603bab90008,lop=PING,est=1571790861246,to=10000,lcxid=0x44,lzxid=0xffffffffffffffff,lresp=1572385305294,llat=0,minlat=0,avglat=0,maxlat=9)
 /172.18.101.103:41923[1](queued=0,recved=446,sent=446,sid=0x16dcc595f400005,lop=PING,est=1571790749655,to=4000000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=1572383995324,llat=0,minlat=0,avglat=0,maxlat=5)
 /172.18.101.102:57197[1](queued=0,recved=189608,sent=189609,sid=0x16df603bab90007,lop=PING,est=1571790861205,to=10000,lcxid=0x39,lzxid=0xffffffffffffffff,lresp=1572385306411,llat=0,minlat=0,avglat=0,maxlat=8)
 /172.18.101.101:54377[1](queued=0,recved=4472,sent=4472,sid=0x16df603bab90003,lop=PING,est=1571790503603,to=400000,lcxid=0x4,lzxid=0xffffffffffffffff,lresp=1572385203263,llat=0,minlat=0,avglat=0,maxlat=2)
 /172.18.101.101:34699[1](queued=0,recved=189501,sent=189501,sid=0x16df603bab9000a,lop=PING,est=1571790868215,to=10000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=1572385306929,llat=0,minlat=0,avglat=0,maxlat=7)
 /172.18.101.103:38405[1](queued=0,recved=297191,sent=297191,sid=0x36d9d24aea50003,lop=PING,est=1571790503757,to=6000,lcxid=0x9d,lzxid=0xffffffffffffffff,lresp=1572385306805,llat=0,minlat=0,avglat=0,maxlat=8)
 /172.18.101.101:54349[1](queued=0,recved=453,sent=454,sid=0x16df603bab90000,lop=PING,est=1571790502812,to=4000000,lcxid=0x6,lzxid=0xffffffffffffffff,lresp=1572385081403,llat=0,minlat=0,avglat=0,maxlat=4)
 /172.18.101.102:57193[1](queued=0,recved=189694,sent=189695,sid=0x16df603bab90006,lop=PING,est=1571790861196,to=10000,lcxid=0x4b,lzxid=0xffffffffffffffff,lresp=1572385307661,llat=1,minlat=0,avglat=0,maxlat=5)
 /172.18.101.101:59966[1](queued=0,recved=4466,sent=4466,sid=0x16d9d24b538000c,lop=PING,est=1571790751228,to=400000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=1572385207714,llat=0,minlat=0,avglat=0,maxlat=2)
 /172.18.101.103:38384[1](queued=0,recved=4474,sent=4475,sid=0x16df603bab90001,lop=PING,est=1571790502913,to=400000,lcxid=0x6,lzxid=0xffffffffffffffff,lresp=1572385225335,llat=0,minlat=0,avglat=0,maxlat=3)
 /172.18.101.103:41944[1](queued=0,recved=4466,sent=4466,sid=0x16d9d24b538000d,lop=PING,est=1571790751534,to=400000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=1572385198840,llat=0,minlat=0,avglat=0,maxlat=4)

With this output, check which component created one specific session. For example, the first session is from  172.18.101.101:34539 , we can find this client is a contrail-snmp-collector instance with the commands below.

Run the following command from the system that owns the IP found in the connection output above

# netstat -antp | grep 34539
tcp    0   0 172.18.101.101:34539  172.18.101.101:2181   ESTABLISHED 6637/python     
tcp6   0   0 172.18.101.101:2181   172.18.101.101:34539  ESTABLISHED 26913/java      

Then, using the pid found from netstat, locate the command from contrail-status output to determine the connected contrail process name

# contrail-status -d | grep 6637
contrail-snmp-collector    active   pid 6637, uptime 6 days, 21:12:38