Description

Leveled Compaction Strategy (LCS) causes load increase, especially when running on Hard Disk Drive (HDD). This is the default compacting strategy on builds older than 2016-12-08.

Symptoms

With the increase of usage, Cassandra is no longer able to keep up with the compaction. As a result, you may be slowly running out of disk space for analytics.

Solution

Try the method described in Contrail Analytics DB data purge . 

If that does not help, and you are running a build older than 2016-12-08, change the compacting strategy manually, as you are affected by the performance limitation described in LP1648252 .

Use the following procedure to change the compaction strategy for the analytics tables in Cassandra. The procedure uses 2 files:
  • cql-truncate-table-analytics
  • cql-alter-compaction-strategy-analytics

  1. Run service supervisor-analytics stop on all nodes

  2. Truncate analytics table and alter compaction strategy

        a. cqlsh <1st-database-node-ip> -f cql-truncate-table-analytics
        b. nodetool status , should show truncate command has cleared the data - data in MB rather than GB
        c. cqlsh <1st-database-node-ip> -f cql-alter-compaction-strategy-analytics
        d. cqlsh <1st-database-node-ip> to verify that the compaction strategy is changed
           >>> describe keyspace "ContrailAnalyticsCql"
           should show new compaction strategy
Note: Steps 3 and 4 are required only if you are changing concurrent_compactors

  1. Change concurrent_compactors to 4 in /etc/cassandra/cassandra.yaml on all nodes

  2. Rolling restart of cassandra

    On each node serially:
        a. nodetool drain
        b. service contrail-database restart
        c. nodetool status to verify that it has joined the cluster

  3. service supervisor-analytics start on all nodes

Files used in fix

Create the following files to be used in the fix provided above.

cql-truncate-table-analytics:

truncate table "ContrailAnalyticsCql".flowrecordtable;
truncate table "ContrailAnalyticsCql".objectvaluetable;
truncate table "ContrailAnalyticsCql".messagetabletimestamp;
truncate table "ContrailAnalyticsCql".statstablebyu64tagv3;
truncate table "ContrailAnalyticsCql".systemobjecttable;
truncate table "ContrailAnalyticsCql".objecttable;
truncate table "ContrailAnalyticsCql".messagetable;
truncate table "ContrailAnalyticsCql".flowtablesvnsipver2;
truncate table "ContrailAnalyticsCql".flowtablevrouterver2;
truncate table "ContrailAnalyticsCql".messagetablemessagetype;
truncate table "ContrailAnalyticsCql".messagetablecategory;
truncate table "ContrailAnalyticsCql".messagetablesource;
truncate table "ContrailAnalyticsCql".flowtableprotdpver2;
truncate table "ContrailAnalyticsCql".statstablebyu64u64tagv3;
truncate table "ContrailAnalyticsCql".statstablebystrtagv3;
truncate table "ContrailAnalyticsCql".messagetablemoduleid;
truncate table "ContrailAnalyticsCql".statstablebydbltagv3;
truncate table "ContrailAnalyticsCql".messagetablekeyword;
truncate table "ContrailAnalyticsCql".statstablebyu64strtagv3;
truncate table "ContrailAnalyticsCql".statstablebystrstrtagv3;
truncate table "ContrailAnalyticsCql".statstablebystru64tagv3;
truncate table "ContrailAnalyticsCql".flowtabledvndipver2;
truncate table "ContrailAnalyticsCql".flowtableprotspver2;

cql-alter-compaction-strategy-analytics:

alter table "ContrailAnalyticsCql".flowrecordtable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".objectvaluetable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetabletimestamp WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebyu64tagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".systemobjecttable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".objecttable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtablesvnsipver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtablevrouterver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablemessagetype WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablecategory WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablesource WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtableprotdpver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebyu64u64tagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebystrtagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablemoduleid WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebydbltagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablekeyword WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebyu64strtagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebystrstrtagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebystru64tagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtabledvndipver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtableprotspver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};

Finishing step & cleanup

According to the Cassandra documentation, you should set read repair to 0 when using DateTieredCompactionStrategy :

"The compaction strategy DateTieredCompactionStrategy precludes using read repair, because of the way timestamps are checked for DTCS compaction.
In this case, you must set read_repair_chance to zero. For other compaction strategies, read repair should be enabled with a read_repair_chance value of 0.2 being typical." So be sure to add the following:

alter table "ContrailAnalyticsCql".flowrecordtable WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtablesvnsipver2 WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtablevrouterver2 WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtableprotdpver2 WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtabledvndipver2 WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtableprotspver2 WITH read_repair_chance = 0.0;

Truncate command in the procedure will create snapshot directories. To clean that data, use " nodetool clearsnapshot"  on each node.