This article demonstrates how session ID increases after Zookeeper client reconnects. 

Observe how a znode changes after Zookeper node communication fails.

CAUTION: Do NOT use this process on a production system.

Initially, svc-monitor is running with pid 5060:

# contrail-status -d | grep svc
contrail-svc-monitor        active       pid 5060, uptime 0:05:20  

Also, the related znode in zookeeper has session ID 0x16e18e9152f000c :

# /usr/share/zookeeper/bin/zkCli.sh
<snipped>
[zk: localhost:2181(CONNECTED) 1] ls /svc-monitor
[71805c1e12da4aa29c15558b4ba332ba__lock__0000000003]

[zk: localhost:2181(CONNECTED) 11] get /svc-monitor/71805c1e12da4aa29c15558b4ba332ba__lock__0000000003
5060
cZxid = 0x401
ctime = Tue Oct 29 12:13:53 PDT 2019
mZxid = 0x401
mtime = Tue Oct 29 12:13:53 PDT 2019
pZxid = 0x401
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16e18e9152f000c
dataLength = 4
numChildren = 0

Using local firewall, blocking port 2181 to BREAK the connection on Contrail node running svc-monitor until the connection is down.

# iptables -A OUTPUT -p tcp --dport 2181 -j DROP
# netstat -alnp | grep 5060 | grep 2181
tcp        0     12 10.176.0.250:41068      10.176.0.250:2181       ESTABLISHED 5060/python
# netstat -alnp | grep 5060 | grep 2181

Now, unblock port 2181 again:

# iptables -D OUTPUT -p tcp --dport 2181 -j DROP

When checking znode again with command '/usr/share/zookeeper/bin/zkCli.sh' , the old znode is gone. Instead, a new znode with sequence number and session ID increased. The sequence number increased from  0000000003 to  0000000004 , and session ID increased from  0x16e18e9152f000c to  0x16e18e9152f0028 .

[zk: localhost:2181(CONNECTED) 13] get /svc-monitor/71805c1e12da4aa29c15558b4ba332ba__lock__0000000003
Node does not exist: /svc-monitor/71805c1e12da4aa29c15558b4ba332ba__lock__0000000003

[zk: localhost:2181(CONNECTED) 14] ls /svc-monitor
[2471d34cf45d4061ab05c2860ff3138b__lock__0000000004]

[zk: localhost:2181(CONNECTED) 17] get /svc-monitor/2471d34cf45d4061ab05c2860ff3138b__lock__0000000004
18621
cZxid = 0x524
ctime = Tue Oct 29 12:27:34 PDT 2019
mZxid = 0x524
mtime = Tue Oct 29 12:27:34 PDT 2019
pZxid = 0x524
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16e18e9152f0028
dataLength = 5
numChildren = 0

For more on troubleshooting Zookeeper, refer to KB31144 - Contrail Getting Started - Administration, Configuration & Troubleshooting (JumpStation) [juniper.net]