Description

 

This article covers a specific bug in Contrail release 1909.30 due to which users are unable to take backup and an error "​ <class 'kazoo.exceptions.NoNodeError'> " is shown. It also provides details of how to fix the issue.

 

Symptoms

 

When users execute the following script to take a backup:

 
docker exec -it config_api_1 bash
cd /usr/lib/python2.7/site-packages/cfgm_common/
python db_json_exim.py --export-to /root/db-dump.json --api-conf /etc/contrail/contrail-api.conf
 

The below error is seen:

 
python db_json_exim.py --export-to /root/db-dump.json --api-conf /etc/contrail/contrail-api.conf
<class 'kazoo.exceptions.NoNodeError'>
Python 2.7.5: /usr/bin/python
Tue Oct 20 02:16:24 2020
 

A problem has occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order in which they occurred:

 
/usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py in <module> ()
258 db_exim.db_export()
259 # end main
260
261 if __name__ == '__main__':
262 main(' '.join(sys.argv[1:]))
main = <function main>
' '.join undefined
sys = <module 'sys' (built-in)>
sys.argv = ['db_json_exim.py', '--export-to', '/root/db-dump.json', '--api-conf', '/etc/contrail/contrail-api.conf']

/usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py in main(args_str='--export-to /root/db-dump.json --api-conf /etc/contrail/contrail-api.conf')
256 db_exim.db_import()
257 if 'export-to' in args_str:
258 db_exim.db_export()
259 # end main
260
db_exim = <__main__.DatabaseExim object>
db_exim.db_export = <bound method DatabaseExim.db_export of<__main__.DatabaseExim object> >

/usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py in db_export(self= <__main__.DatabaseExim object> )
234 zk = kazoo.client.KazooClient(self._api_args.zk_server_ip)
235 zk.start()
236 nodes = get_nodes(self._api_args.cluster_id+'/')
237 zk.stop()
238 db_contents['zookeeper'] = json.dumps(nodes)
nodes undefined
get_nodes = <function get_nodes>
self = <__main__.DatabaseExim object>
self._api_args = Namespace(aaa_mode='no-auth', admin_password='',....)
self._api_args.cluster_id = ''

/usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py in get_nodes(path='/')
228 nodes = []
229 for child in zk.get_children(path):
230 nodes.extend(get_nodes('%s%s/' %(path, child)))
231
232 return nodes

nodes.extend = <built-in method extend of list object>
get_nodes = <function get_nodes>
path = '/'
child = u'api-server'

/usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py in get_nodes(path=u'/api-server/')
228 nodes = []
229 for child in zk.get_children(path):
230 nodes.extend(get_nodes('%s%s/' %(path, child)))
231
232 return nodes
nodes = []
nodes.extend = <built-in method extend of list object>
get_nodes = <function get_nodes>
path = u'/api-server/'
child = u'subnets'

/usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py in get_nodes(path=u'/api-server/subnets/')
228 nodes = []
229 for child in zk.get_children(path):
230 nodes.extend(get_nodes('%s%s/' %(path, child)))
231
232 return nodes

nodes.extend = <built-in method extend of list object>
get_nodes = <function get_nodes>
path = u'/api-server/subnets/'
child = u'default-domain:default-project:addr-alloc

/usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py in get_nodes(path=u'/api-server/subnets/default-domain:default-project:addr-alloc)
228 nodes = []
229 for child in zk.get_children(path):
230 nodes.extend(get_nodes('%s%s/' %(path, child)))
231
232 return nodes
nodes = []
nodes.extend = <built-in method extend of list object>
get_nodes = <function get_nodes>
path = u'/api-server/subnets/default-domain:default-project:addr-alloc
child = u'17'

 

Solution

 

See https://review.opencontrail.org/c/Juniper/contrail-controller/+/60866 for details about the fix, wherein the logic for appropriate handling of the NoNodeError exception is added during db_json_exim execution.

If you are using Contrail release 1909.30 and facing this issue, you can do the following to take a backup:

  1. Add the changes in  db_json_exim as per  https://review.opencontrail.org/c/Juniper/contrail-controller/+/60866/1/src/config/common/cfgm_common/db_json_exim.py ​.

  2. Ensure that you take a backup of db_json_exim.py , which is located at /usr/lib/python2.7/site-packages/cfgm_common/ inside container config_api_1 . ( Note: Container name may vary as per deployment type.)

Steps: 

docker cp config_api_1:/usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py .
ls -al | grep db_json_exim.py >> to ensure that the file is copied to present working directory

To fix this issue: 

  1. docker exec -it config_api_1 bash
  2. cd /usr/lib/python2.7/site-packages/cfgm_common/
  3. vi db_json_exim.py

Make appropriate changes as per https://review.opencontrail.org/c/Juniper/contrail-controller/+/60866/1/src/config/common/cfgm_common/db_json_exim.py#b224) .

Before Modification

def get_nodes(path):
    if not zk.get_children(path):
        return [(path, zk.get(path))]

After Modification

def get_nodes(path):
    try:
       if not zk.get_children(path):
          return [(path, zk.get(path))]
    except kazoo.exceptions.NoNodeError:
            return []

  1. Save the file and execute the backup.