Backups Restoring - Orodan/Hilary GitHub Wiki
Mostly automatic through puppet and the duplicity module
The following outlines how production database backups can be restored on the staging nodes.
What you will need:
- The encryption key that was used to encrypt the backups. All key IDs in this doc are examples.
- An AWS access/secret key that has READ/LIST privilege on the Amazon S3 bucket where the backups are stored.
On each Cassandra node;
- Install required restore tools, which are not included by default in staging.
# apt-get install python-pip duplicity
# pip install boto
- Import the private key.
# gpg --import key-for-backups.pvt
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: key 281CF39B: public key "Backup Key <[email protected]>" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
- Verify if you have all the correct credentials.
# gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub 2048R/281CF39B 2013-06-22
uid Backup Key <[email protected]>
sub 2048R/D2E4C1E2 2013-06-22
# export AWS_ACCESS_KEY_ID=<aws key id>
# export AWS_SECRET_ACCESS_KEY=<aws secret access key>
# export encryptkey=<backup private key>
# duplicity --s3-use-new-style --encrypt-key=${encryptkey} list-current-files s3+http://oae-cassandra-backup/db0/cassandra
Import of duplicity.backends.sshbackend Failed: No module named paramiko
Import of duplicity.backends.giobackend Failed: No module named gio
Synchronizing remote metadata to local cache...
GnuPG passphrase:
Copying duplicity-full-signatures.20140328T040001Z.sigtar.gpg to local cache.
Copying duplicity-full-signatures.20140428T040002Z.sigtar.gpg to local cache.
Copying duplicity-full.20140125T040002Z.manifest.gpg to local cache.
Copying duplicity-full.20140225T040002Z.manifest.gpg to local cache.
Copying duplicity-full.20140328T040001Z.manifest.gpg to local cache.
Copying duplicity-full.20140428T040002Z.manifest.gpg to local cache.
Copying duplicity-inc.20140125T040002Z.to.20140126T040001Z.manifest.gpg to local cache.
Copying duplicity-inc.20140126T040001Z.to.20140127T040002Z.manifest.gpg to local cache.
Copying duplicity-inc.20140127T040002Z.to.20140128T040002Z.manifest.gpg to local cache.
...
Last full backup date: Mon Apr 28 04:00:02 2014
Sat Nov 23 12:09:25 2013 .
Thu Jul 11 16:52:31 2013 OpsCenter
Thu May 8 19:46:04 2014 OpsCenter/events
Sat Mar 15 05:58:00 2014 OpsCenter/events/OpsCenter-events-ic-4654-CompressionInfo.db
Sat Mar 15 05:58:00 2014 OpsCenter/events/OpsCenter-events-ic-4654-Data.db
Sat Mar 15 05:58:00 2014 OpsCenter/events/OpsCenter-events-ic-4654-Filter.db
Sat Mar 15 05:58:00 2014 OpsCenter/events/OpsCenter-events-ic-4654-Index.db
Sat Mar 15 05:58:00 2014 OpsCenter/events/OpsCenter-events-ic-4654-Statistics.db
Sat Mar 15 05:58:00 2014 OpsCenter/events/OpsCenter-events-ic-4654-Summary.db
Sat Mar 15 05:58:00 2014 OpsCenter/events/OpsCenter-events-ic-4654-TOC.txt
Thu Mar 27 21:22:26 2014 OpsCenter/events/OpsCenter-events-ic-5415-CompressionInfo.db
Thu Mar 27 21:22:26 2014 OpsCenter/events/OpsCenter-events-ic-5415-Data.db
Thu Mar 27 21:22:26 2014 OpsCenter/events/OpsCenter-events-ic-5415-Filter.db
Thu Mar 27 21:22:26 2014 OpsCenter/events/OpsCenter-events-ic-5415-Index.db
Thu Mar 27 21:22:26 2014 OpsCenter/events/OpsCenter-events-ic-5415-Statistics.db
Thu Mar 27 21:22:26 2014 OpsCenter/events/OpsCenter-events-ic-5415-Summary.db
..
If you get to to this point it means we're fully setup to pull down the data from S3.
- Stop Cassandra AND puppet agent on all the db nodes, otherwise the puppet agent will restart Cassandra.
# service puppet stop
# service dse stop
- Blow away all the data, commitlogs and saved caches:
# rm -rf /data/cassandra/data/* /var/lib/cassandra/*
- Restore the files e.g.: on db0
# export AWS_ACCESS_KEY_ID=<aws key id>
# export AWS_SECRET_ACCESS_KEY=<aws secret access key>
# export encryptkey=<backup private key>
# duplicity --s3-use-new-style --encrypt-key=${encryptkey} restore s3+http://oae-cassandra-backup/db0/cassandra /data/cassandra/data
Note: To interact with the restored staging with a browser you'll likely need to adjust nginx's config.