This article details a specific scenario of Cassandra database repair, its impact, a workaround, and the fixed version information.

The schema container is crashing in all three Contrail nodes:

[contrail-schema] [ERROR]: Cassandra connection down. Exception in<unbound method ColumnFamily.multiget>: Retried 16 times. Last failure was timeout: timed out

contrail-config-database-nodemgr.log indicates failed to get database usage: Timed out and failed to get nodetool compactionstats

The issue is fixed in Contrail R1911 and later where the contrail-cassandra-repair.py script is invoked to perform periodic nodetool repair -pr from nodemgr every 24 hours by default.

Meanwhile, as a workaround, you can run the following query on all Contrail nodes:

docker exec -it config_database_cassandra_1 nodetool -p 7201 repair -pr config_db_uuid

Note: The container name may change based on deployment type.

Depending on the database size, sometimes you may be required to run nodetool with the --full argument: nodetool repair -p 7201 -pr --full .

Reference: https://review.opencontrail.org/c/Juniper/contrail-controller/+/54625