Technical details of backup to RDS ephemeral

Mounting the destination drive

The aichemyrds RDS store is mounted at /mnt/aichemyrds/ using a role account that has access to the aichemyrds RDS area.

In /etc/fstab there is a line:

# Mount aichemy RDS
//rds.imperial.ac.uk/rds/project/aichemyrds /mnt/aichemyrds cifs credentials=/root/smb.cred,vers=3.0,iocharset=utf8,_netdev,nofail,x-systemd.automount,uid=0,gid=6608,file_mode=0664,dir_mode=0775 0 0

The credentials of the role account are stored in /root/smb.cred.

Scheduling the backup

A script /nomad/backup_nomad.sh is run by a cronjob at 2am every day.

To add a line to the crontab, run sudo crontab -e and add a line like the below:

0 2 * * * /nomad/backup_nomad.sh >> /var/log/nomad_backup.log 2>&1

The backup_nomad.sh script itself handles both backing up the mongodb and the datastore of raw data files:

#!/bin/bash

# trigger automount
stat /mnt/aichemyrds/ephemeral >/dev/null 2>&1 || true
# Check for mounted RDS
if ! findmnt -rno TARGET /mnt/aichemyrds > /dev/null; then
    echo "MongoDB backup aborted: ephemeral volume not mounted"
    exit 1
fi

# Set the date format for filename
DATE=$(date +"%Y-%m-%d")
echo "[$(date)] Starting MongoDB backup"

# Set the dump output path
OUTPUT_DIR="/mnt/aichemyrds/ephemeral"
MONGO_FILENAME="nomad-mongodb-$DATE.dump"

# Run mongodump in docker and save to timestamped file
sudo docker exec -i nomad-mongodb-1 sh -c 'mongodump --archive' > "$OUTPUT_DIR/$MONGO_FILENAME"
if [ $? -eq 0 ]; then
    echo "[$(date)] MongoDB backup completed: $MONGO_FILENAME"
else
    echo "[$(date)] MongoDB backup FAILED"
    exit 1
fi

# === Backup /nomad/datastore ===
echo "[$(date)] Starting datastore backup"

DATASTORE_SOURCE="/nomad/datastore"
DATASTORE_FILENAME="mongo-datastore-backup-$DATE.tar.gz"
DATASTORE_PATH="$OUTPUT_DIR/$DATASTORE_FILENAME"

if [ -d "$DATASTORE_SOURCE" ]; then
    tar -czf "$DATASTORE_PATH" -C /nomad datastore
    if [ $? -eq 0 ]; then
        echo "[$(date)] Datastore backup completed: $DATASTORE_FILENAME"
    else
        echo "[$(date)] Datastore backup FAILED"
        exit 1
    fi
else
    echo "[$(date)] Datastore backup FAILED: directory not found"
    exit 1
fi