Description

When searching the zookeeper log file, "end of stream exception" may be observed. This exception is an indicator of the end of the data stream of a session. It usually occurs when closing a connection to zookeeper. This article provides an example on how it is triggered.  

Symptoms

During zookeeper troubleshooting, the following exception may be seen in zookeeper.log. This exception does not signal a defect on the zookeeper side. Instead, it shows a connection to client is reset or closed.  

EndOfStreamException: Unable to read additional data from client sessionid 0x16e18e9152f002f, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)

Solution

Lets use svc-monitor as an example. For a svc-monitor, a session is established with zookeeper, and a znode will be created for svc-monitor inside the zookeeper. The data part of this znode will be pid of svc-monitor.

Checking pid and status on contrail node.

# contrail-status -d | grep svc
contrail-svc-monitor       active       pid 8785 , uptime 0:01:36

Use the command, ' /usr/share/zookeeper/bin/zkCli.sh ' to enter zookeeper CLI. Then look for children znode created under /svc-monitor , one of which should have data matching pid of this svc-monitor. The ephemeralOwner of this znode will be session id of this svc-monitor. 

# /usr/share/zookeeper/bin/zkCli.sh
Connecting to localhost:2181
<snipped>
[zk: localhost:2181(CONNECTED) 1] get /svc-monitor/02efe5df71eb4613a3591c86a0c684d6__lock__0000000007
8785
cZxid = 0x595
ctime = Tue Oct 29 13:44:00 PDT 2019
mZxid = 0x595
mtime = Tue Oct 29 13:44:00 PDT 2019
pZxid = 0x595
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16e18e9152f002f
dataLength = 4
numChildren = 0

If svc-monitor is restarted, we can see corresponding log event showing in zookeeper log.  

# service contrail-svc-monitor restart 
contrail-svc-monitor: stopped
contrail-svc-monitor: started

Notice this log event also provides session ID information, so we can confirm this exception is caused by svc-monitor restarting and disconnecting operation.  

2019-10-29 13:45:58,543 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x16e18e9152f002f, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2019-10-29 13:45:58,544 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /10.176.0.250:48980 which had sessionid 0x16e18e9152f002f

For more on troubleshooting Zookeeper, refer to KB31144 - [juniper.net] Contrail Getting Started - Administration, Configuration & Troubleshooting (JumpStation)