Cassandra state detected down - ganeshahv/Contrail_SRE GitHub Wiki

1]. Issue with missing tables

Error

ERROR [main] 2020-07-21 07:46:12,972 SchemaKeyspace.java:1019 - No partition columns found for table ContrailAnalyticsCql.statstablev4 in system_schema.columns.  This may be due to corruption or concurrent dropping and altering of a table. If this table is supposed to be dropped, restart cassandra with -Dcassandra.ignore_corrupted_schema_tables=true and run the following query to cleanup: "DELETE FROM system_schema.tables WHERE keyspace_name = 'ContrailAnalyticsCql' AND table_name = 'statstablev4'; DELETE FROM system_schema.columns WHERE keyspace_name = 'ContrailAnalyticsCql' AND table_name = 'statstablev4';" If the table is not supposed to be dropped, restore system_schema.columns sstables from backups.
Exception (org.apache.cassandra.schema.SchemaKeyspace$MissingColumns) encountered during startup: Columns not found in schema table for ContrailAnalyticsCql.statstablev4
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for ContrailAnalyticsCql.statstablev4
	at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1098)
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:1044)
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:998)
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:957)
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:934)
	at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:922)
	at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:92)
	at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:82)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:262)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)

Cause

Schema corruption

Solution/Workaround

i). Get the mount point of the data

root@juju-d9dadc-1-kvm-1:/home/ubuntu# docker inspect analyticsdatabase_cassandra_1 | grep Source

ii). docker-compose down of analytics

iii). remove all the files under the path

/var/lib/docker/volumes/analyticsdatabase_analytics_cassandra/_data

iv). docker-compose up -d

2]. Issue with Heap Size can be resolved by following these steps:

1. Login to docker container using command : docker exec -it analytics_database_cassandra bash
2. Navigate to /etc/Cassandra using command : cd /etc/cassandra
3. Edit Cassandra-env.sh and search for values of MAX_HEAP_SIZE and HEAP_NEWSIZE and set it as 16G and 2G 
  respectively.

Before changes:
#MAX_HEAP_SIZE="4G"
#HEAP_NEWSIZE="800M"
 
After changes:
MAX_HEAP_SIZE="16G"
HEAP_NEWSIZE="2G"