Description

This article describes a workaround to address the issue of Zookeeper failing to come up and logging the "Unable to load database on disk" error.

Symptoms

Users may observe that Zookeeper fails to come up and logs the "Unable to load database on disk" error. In zookeeper.log , an error sequence similar to the following may be seen: 

2019-09-10 02:00:47,452 - ERROR [main:FileTxnSnapLog@210] - Parent /isr_change_notification missing for /isr_change_notification/isr_change_0000002662
2019-09-10 02:00:47,453 - ERROR [main:QuorumPeer@453]- Unable to load database on disk
java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /isr_change_notification
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
        at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /isr_change_notification
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
        ... 6 more
2019-09-10 02:00:47,455 - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:454)
       at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /isr_change_notification
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
        ... 4 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /isr_change_notification
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
        ... 6 more

 

Solution

This error is due to Zookeeper's internal snapshot being wiped out or snapshot corruption.  

 

On the Contrail node where Zookeeper fails to come up or keeps bouncing and the aforementioned error is observed in zookeeper.log , we can take the following actions to bring up the Zookeeper service:

  1. Make sure that the Zookeeper service is indeed shut down by service zookeeper stop .

  2.  Take a backup of the contents in /var/lib/zookeeper/version-2 to a temp location, and then remove the contents in /var/lib/zookeeper/version-2 by using rm -rf /var/lib/zookeeper/version-2/* .

  3. Restart the Zookeeper service by using service zookeeper start .

  4. Run service zookeeper status and monitor zookeeper.log to ensure that the service starts up correctly.