Contrail ConfigDB DR(Backup Restore) Procedure - ganeshahv/Contrail_SRE GitHub Wiki

NOTE: The restore procedure is meant for debugging purposes only and not configDB restore in a production environment.

Backup procedure

1]. On the first controller, create a backup directory.

mkdir /tmp/db-dump

2]. Get inside the config_api docker.

docker exec -it config_api_1 bash

3]. Use the exim utility to export the db-dump.

python /usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py --export-to db-dump.json

4]. Exit the docker and copy the contrail_api conf file and the db-dump to the backup directory on the node.

docker cp config_api_1:/etc/contrail/contrail-api.conf /tmp/db-dump/
docker cp config_api_1:/db-dump.json /tmp/db-dump/

Recovery procedure

1]. On the controller node, stop all the contrail services which register with Zookeeper.

for i in `` `docker ps -a | grep config | awk '{print $1}'` ``; do docker stop "$i"; done
for i in `` `docker ps -a | grep analytics | awk '{print }'` ``; do docker stop "$i"; done
for i in `` `docker ps -a | grep database | awk '{print }'` ``; do docker stop "$i"; done
for i in `` `docker ps -a | grep vcente | awk '{print }'` ``; do docker stop "$i"; done
for i in `` `docker ps -a | grep control | awk '{print }'` ``; do docker stop "$i"; done
for i in `` `docker ps -a | grep webui | awk '{print }'` ``; do docker stop "$i"; done
for i in `` `docker ps -a | grep redis | awk '{print }'` ``; do docker stop "$i"; done

2]. Backup the Zookeeper and Cassandra datastore and manually delete the data files.

cp -R /var/lib/docker/volumes/config_database_config_zookeeper/_data/version-2 /var/lib/docker/volumes/config_database_config_zookeeper/_data/version-2-slave
rm -f /var/lib/docker/volumes/config_database_config_zookeeper/_data/version-2/*
cp -R /var/lib/docker/volumes/config_database_config_cassandra/_data /var/lib/docker/volumes/config_database_config_cassandra/_data-slave
rm -rf /var/lib/docker/volumes/config_database_config_cassandra/_data/*

3]. Start only the config cassandra and zookeeper services.

for i in `` `docker ps -a | grep config | grep cassa  | awk '{print $1}'` ``; do docker start "$i"; done
for i in `` `docker ps -a | grep config | grep zoo  | awk '{print $1}'` ``; do docker start "$i"; done

4]. Start a new config_api docker and copy the saved config and db-dump into it.

docker image ls | grep config-api
docker run --rm -it -v /tmp/db-dump/:/tmp/ --network host --entrypoint=/bin/bash REPOSITORY:TAG

5]. Restore the config and db-dump.

cd /usr/lib/python2.7/site-packages/cfgm_common
python db_json_exim.py --import-from /tmp/db-dump.json --api-conf /tmp/contrail-api.conf

6]. Exit the docker and on the controller node, start all the contrail services which were stopped earlier.

for i in `` `docker ps -a | grep config | awk '{print $1}'` ``; do docker start "$i"; done
for i in `` `docker ps -a | grep analytics | awk '{print $1}'` ``; do docker start "$i"; done
for i in `` `docker ps -a | grep database | awk '{print $1}'` ``; do docker start "$i"; done
for i in `` `docker ps -a | grep vcente | awk '{print $1}'` ``; do docker start "$i"; done
for i in `` `docker ps -a | grep control | awk '{print $1}'` ``; do docker start "$i"; done
for i in `` `docker ps -a | grep webui | awk '{print $1}'` ``; do docker start "$i"; done
for i in `` `docker ps -a | grep redis | awk '{print $1}'` ``; do docker start "$i"; done