For all svc-monitors running in the same cluster, only one is active while the others are backups. When the active one is down, any of the the backups can be selected as the new active one. Zookeeper is used to decide which svc-monitor will be the active one.
This article explains how to check the active svc-monitor in Zookeeper.
When accessing Zookepper CLI, we see a znode is created for svc-monitor lock under /svc-monitor . It has several children. Each of them represents a svc-monitor instance running on Contrail node.
# /usr/share/zookeeper/bin/zkCli.sh Connecting to localhost:2181 <snipped> [zk: localhost:2181(CONNECTED) 0] [zk: localhost:2181(CONNECTED) 0] ls / [api-server, consumers, svc-monitor, contrail_cs, device-manager, controller_epoch, lockpath, id, isr_change_notification, fq-name-to-uuid, admin, zookeeper, api-server-election, config, schema-transformer, controller, brokers] [zk: localhost:2181(CONNECTED) 1] ls /svc-monitor [f222871e20d047cfade1003f9f485740__lock__0000000573, 140cb1cea3d34227a34194eb4102e5b9__lock__0000000572, ead8d60a310c448ab530809bc15bc354__lock__0000000559]
# /usr/share/zookeeper/bin/zkCli.sh Connecting to localhost:2181 <snipped>
[zk: localhost:2181(CONNECTED) 0] [zk: localhost:2181(CONNECTED) 0] ls / [api-server, consumers, svc-monitor, contrail_cs, device-manager, controller_epoch, lockpath, id, isr_change_notification, fq-name-to-uuid, admin, zookeeper, api-server-election, config, schema-transformer, controller, brokers] [zk: localhost:2181(CONNECTED) 1] ls /svc-monitor [f222871e20d047cfade1003f9f485740__lock__0000000573, 140cb1cea3d34227a34194eb4102e5b9__lock__0000000572, ead8d60a310c448ab530809bc15bc354__lock__0000000559]
Here, we have 3 children for svc-monitor znode. They are sequential ephemeral nodes, and named using a format of random uuid generated by kazoo lib, plus '__lock__' and a sequence number. For example, znode /svc-monitor/ead8d60a310c448ab530809bc15bc354__lock__0000000559 has random uuid 'ead8d60a310c448ab530809bc15bc354' and sequence number '0000000559' . The svc-monitor instance with the lowest sequence number is the first instance that connects to zookeeper, creates the znode and acquires the lock. Thus, it should be the active one while the others are backups. We can see znode /svc-monitor/ead8d60a310c448ab530809bc15bc354__lock__0000000559 has the lock, and the svc-monitor creating this zonde should be in "active" status.
get
[zk: localhost:2181(CONNECTED) 2] get /svc-monitor/f222871e20d047cfade1003f9f485740__lock__0000000573 10632 cZxid = 0x700000000c ctime = Tue Oct 22 17:28:23 PDT 2019 mZxid = 0x700000000c mtime = Tue Oct 22 17:28:23 PDT 2019 pZxid = 0x700000000c cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x16df603bab90005 dataLength = 5 numChildren = 0 [zk: localhost:2181(CONNECTED) 3] get /svc-monitor/140cb1cea3d34227a34194eb4102e5b9__lock__0000000572 29731 cZxid = 0x7000000006 ctime = Tue Oct 22 17:28:22 PDT 2019 mZxid = 0x7000000006 mtime = Tue Oct 22 17:28:22 PDT 2019 pZxid = 0x7000000006 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x16df603bab90002 dataLength = 5 numChildren = 0 [zk: localhost:2181(CONNECTED) 4] get /svc-monitor/ead8d60a310c448ab530809bc15bc354__lock__0000000559 22166 cZxid = 0x6c000001c9 ctime = Sat Oct 05 11:27:33 PDT 2019 mZxid = 0x6c000001c9 mtime = Sat Oct 05 11:27:33 PDT 2019 pZxid = 0x6c000001c9 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x26d9d24aec60008 dataLength = 5 numChildren = 0 [zk: localhost:2181(CONNECTED) 5]
While getting data from each znode, we can see the data of the znode is actually the pid of svc-monitor on contrail nodes.
# contrail-status -d | grep contrail-svc-monitor contrail-svc-monitor backup pid 10632, uptime 6 days, 17:57:02 # contrail-status -d | grep contrail-svc-monitor contrail-svc-monitor backup pid 29731, uptime 6 days, 17:57:11 # contrail-status -d | grep contrail-svc-monitor contrail-svc-monitor active pid 22166, uptime 23 days, 23:50:33