Analytics cassandra podman container has been stuck in a restart loop with the following error.
ERROR [main] 2025-10-07 06:21:46,828 JVMStabilityInspector.java:196 - Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Mutation checksum failure at 10338262 in Next section at 10337648 in CommitLog-7-1756308722663.log
at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387)
at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200)
at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223)
at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
Analytics Cassandra container stuck in restart loop.
An error similar to the following indicates that a Cassandra commit log has become corrupted:
CommitLogReadException: Mutation checksum failure at 10338262 in Next section at 10337648 in CommitLog-7-1756308722663.log
Cassandra writes data to commit logs before flushing it to disk. A corruption in these files often points to a I/O failure. In most cases, performing a commit log cleanup is sufficient to restore normal operation.
Procedure: Commit Log Cleanup
This procedure clears the corrupted commit logs and allows the analytics database to heal itself.
Step 1: Identify the Analytics Database Container
sudo podman ps | grep analytics_database
Step 2: Locate the Commit Logs Directory
Inspect the container to find the _data path:
sudo podman inspect <analytics_database_container_name> | grep "_data"
The output will look similar to:
"Source": "/var/lib/containers/storage/volumes/<volume_id>/_data"
Step 3: Verify the Commit Logs Path
sudo ls -lrth /var/lib/containers/storage/volumes/<volume_id>/_data
Example output:
total 0
drwxr-xr-x. 2 root root 6 Aug 26 02:09 data
drwxr-xr-x. 2 root root 6 Aug 26 02:09 commitlog
drwxr-xr-x. 2 cassandra cassandra 49 Oct 3 17:21 saved_caches
drwxr-xr-x. 2 cassandra cassandra 138 Oct 14 15:36 hints
Step 4: Move the Commit Logs to a Backup Directory
This effectively clears the logs while keeping a backup for reference.
sudo mkdir -p /tmp/backups/
sudo mv /var/lib/containers/storage/volumes/<volume_id>/_data/commitlog /tmp/backups/
Step 5: Verify Cassandra Stability
After clearing the commit logs, Cassandra should become stable. Allow 30–60 minutes for the analytics database to automatically heal.
If Cassandra does not recover after commit log cleanup, follow these additional steps.
Step 1: Stop the Cassandra and Collector Containers
sudo podman stop <cassandra_container_name> <collector_container_name>
Step 2: Backup and Clear the Data Directory
sudo mv /var/lib/docker/volumes/analytics_database_analytics_cassandra/_data /tmp/backups/
Step 3: Restart the Services
sudo podman start <collector_container_name> <cassandra_container_name>
Once the containers are stable the database will sync and it should take about 30-60 seconds for data sync to complete, so please monitor for any errors.