This article covers a specific problem in Contrail release 3.2.18 due to which users are unable to take backup of configuration database without any prompt for long time.

• When users executed below script, it will not return any prompt for long time

python /usr/lib/python2.7/dist-packages/cfgm_common/db_json_exim.py --export-to /var/tmp/`hostname`_`date +"%Y%m%d-%H%M"`_db-dump.json

• Modified the script to increase the cassandra timeout value from 5 to 10. Even then Backupt script not working and hence no backup file.

pool = pycassa.ConnectionPool( full_ks_name, self._api_args.cassandra_server_list, pool_timeout=120, max_retries=0, timeout=10)

• Backup script works only if config_db_uuid keyspace is omitted

python /usr/lib/python2.7/dist-packages/cfgm_common/db_json_exim.py --export-to /var/tmp/`hostname`_`date +"%Y%m%d-%H%M"`_db-dump.json --omit-keyspace config_db_uuid

• Cassandra system logs has more tombstone cells related messages

system.log:WARN [SharedPool-Worker-1] 2026-03-30 17:45:44,664 SliceQueryFilter.java:307 - Read 0 live and 4886 tombstone cells in config_db_uuid.obj_fq_name_table for key: 726f7574655f746172676574 (see tombstone_warn_threshold). 10000000 columns were requested, slices=[7461726765743a36353531323a383030303032303a-7461726765743a36353531323a383030303032303b]

system.log:WARN [SharedPool-Worker-1] 2026-03-30 17:45:44,727 SliceQueryFilter.java:307 - Read 0 live and 4886 tombstone cells in config_db_uuid.obj_fq_name_table for key: 726f7574655f746172676574 (see tombstone_warn_threshold). 10000000 columns were requested, slices=[7461726765743a36353531323a383030303032303a-7461726765743a36353531323a383030303032303b]

system.log:WARN [SharedPool-Worker-1] 2026-03-30 17:45:47,865 SliceQueryFilter.java:307 - Read 36 live and 9790 tombstone cells in config_db_uuid.obj_uuid_table for key: 36613366653963342d633632382d343231392d613261652d633630313665343632306132 (see tombstone_warn_threshold). 10000000 columns were requested, slices=[-]

system.log:WARN [SharedPool-Worker-1] 2026-03-30 17:45:51,674 SliceQueryFilter.java:307 - Read 1 live and 4895 tombstone cells in config_db_uuid.obj_fq_name_table for key: 7669727475616c5f6e6574776f726b (see tombstone_warn_threshold). 10000000 columns were requested, slices=[64656661756c742d646f6d61696e3a5056545f32333537355f52444d36425f5243395f30313a7376632d766e2d6c6566743a-64656661756c742d646f6d61696e3a5056545f32333537355f52444d36425f5243395f30313a7376632d766e2d6c6566743b]

The DB backup is failing due to excessive tombstone accumulation in the config_db_uuid keyspace.

Cassandra must scan thousands of tombstones per read (up to ~9,800 per key), causing extreme slowness and timeouts. Increasing the timeout value alone will not resolve this. User has to execute below steps

to solve the problem.

1. Run nodetool repair on config_db_uuid first

— this ensures all nodes are consistent before any tombstone cleanup

nodetool repair -pr config_db_uuid

2. Reduce gc_grace_seconds from 10 days to 3 days

Check the gc_grace_seconds

cassandra@cqlsh> SELECT columnfamily_name, gc_grace_seconds FROM system.schema_columnfamilies WHERE keyspace_name = 'config_db_uuid';

columnfamily_name | gc_grace_seconds

-------------------+------------------

obj_fq_name_table | 864000

obj_shared_table | 864000

obj_uuid_table | 864000

Reduce gc_grace_seconds using these Contrail 3.x compatible commands:

ALTER TABLE config_db_uuid.obj_uuid_table WITH gc_grace_seconds = 259200;

ALTER TABLE config_db_uuid.obj_fq_name_table WITH gc_grace_seconds = 259200;

ALTER TABLE config_db_uuid.obj_shared_table WITH gc_grace_seconds = 259200;

3. Run compaction during a maintenance window (this will be I/O intensive on a large keyspace)

nodetool compact config_db_uuid

4. Retry the full DB backup including config_db_uuid