Backup server config - KeegMitch/Operations-Engineering-group-c GitHub Wiki

Basic puppet backup bash script (This one is used on mgmt server)

#!/bin/bash

# Define source directories
puppet_dir="/etc/puppetlabs/"
# modules_dir="/etc/puppetlabs/code/modules"
# site_pp="/etc/puppetlabs/code/environments/production/manifests/site.pp"
user="group-c"
backup_server="backup-c"
storage_server="20.211.153.89"


# Get the current date and time
current_date=$(date +"%B_%d_%Y_%H%M%S")

# create backup
log_dir="$HOME/logs"
log_file="$log_dir/rsync_puppetconf.log"

if [ ! -d "$log_dir" ]; then
    mkdir -p "$log_dir"
fi

# Define archive filename
archive_name="RSYNCpuppetbackup_$current_date.tar.gz"

# Create the mgmt_backups directory if it doesn't exist
backup_dir="$HOME/rsync_backup"

if [ ! -d "$backup_dir" ]; then
    # Create the directory
    sudo mkdir -p "$backup_dir"
        echo "Check manual_puppetconf.log for output"
    echo "Directory $backup_dir created." >> $log_file
else
        echo "Check manual_puppetconf.log for output"
    echo "Directory $backup_dir already exists. Backing up puppet config ..." >> $log_file
fi

# Create a compressed tar archive directly from source files
# sudo tar -czvf "$HOME/mgmt_backups/$archive_name" "$modules_dir" "$site_pp"
sudo tar -czvf "$backup_dir/$archive_name" "$puppet_dir" >> $log_file


# rsync into backup server

rsync -av -e "ssh -i /home/group-c/.ssh/id_rsa" "$backup_dir" $user@$backup_server:~/mgmt_backups/

# rsync into storage server

rsync -av -e "ssh -i /home/group-c/.ssh/id_rsa_offsite" "$backup_dir" $user@$storage_server:~/mgmt-c/

exit_status=$?

if [ $? -eq 0 ](/KeegMitch/Operations-Engineering-group-c/wiki/-$?--eq-0-); then
echo "Backup created: Check rsync_puppetconf.log for output"
echo "Backup created: $backup_dir/$archive_name" >> $log_file
else
  echo "Backup failed with exit code: $exit_status , Check rsync_puppetconf.log for output"
  echo "Backup failed with exit code: $exit_status" >> $log_file
fi

# automated by the sudo crontab as well that runs every day at 7pm NZT / 7am UTC (cron changed to NZT)

# following commands in sudo crontab (runs 4 times a day or every 6 hours):
# 0 0,6,12,18 * * * home/group-c/rsync_puppetlabs.sh > /logs/cron.log
# or this syntax
# 0 */6 * * * home/group-c/rsync_puppetlabs.sh > /logs/cron.log

To run the script: sudo chmod +x rsync_puppetlabs.sh ./rsync_puppetlabs.sh

Script will send a copy of the tar file to the backup server using SCP

This is the code that sends the copy, however this may mess with the cron as to transfer the file it requires the backup-c password.

For this to work the directory needs to be owned by the user so that the copy can be written to on the backup-c

sudo chown -R group-c:group-c test_backups

There also needs to be a ssh key generated.

ssh-keygen -t rsa -b 4096 -f /home/group-c/testingKey/id_rsa

exit_status=$?

if [ $? -eq 0 ](/KeegMitch/Operations-Engineering-group-c/wiki/-$?--eq-0-); then
echo "Backup created: Check manual_puppetconf.log for output"
echo "Backup created: $backup_dir/$archive_name" >> $log_file
 scp -i /home/group-c/testingKey/id_rsa "$backup_dir/$archive_name" group-c@backup-c:~/test_backups

    if [ $? -eq 0 ](/KeegMitch/Operations-Engineering-group-c/wiki/-$?--eq-0-); then
        echo "Backup archive transferred successfully to destination server."
    else
        echo "Failed to transfer backup archive to destination server."
    fi
else
  echo "Backup failed with exit code: $exit_status , Check manual_puppetconf.log for output"
  echo "Backup failed with exit code: $exit_status" >> $log_file
fi

Another way to backup files is with rsync

sudo rsync -av -e "ssh -i /home/group-c/.ssh/id_rsa" /etc/puppetlabs group-c@backup-c:~/rsync_backup

You do need to change the owner of the file your backing up to

sudo chown group-c:group-c rsync_backup/

# Define source directories
puppet_dir="/etc/puppetlabs/"

current_date=$(date +"%B_%d_%Y_%H%M%S")

# Define archive filename
archive_name="RsyncPuppetbackup_$current_date.tar.gz"

sudo rsync -av --link-dest=/home/group-c/rsync_backup -e "ssh -i /home/group-c/.ssh/id_rsa" /etc/puppetlabs group-c@backup-c:~/rsync_backup

if [ $? -eq 0 ](/KeegMitch/Operations-Engineering-group-c/wiki/-$?--eq-0-); then
echo "Backup created: $archive_name"

fi

Automate the script using the crontab task scheduler

Crontab guru is a good resource

Note: crontab works using the UTC timezone by default so any backups at 4pm NZT will be 4am UTC time, also you have to use sudo crontab

So if you want to have things backed up double check the time syntax, you can of course change the timezone as we ended up doing here

• Open the crontab editor using sudo crontab -e, and select the vim.basic option
• Add the HOME environment variable from the script: HOME=/home/group-c
• Add your backup script inside the file, e.g. the puppet backup script above: 0 4 * * * /home/group-c/rsync_puppetlabs.sh
• check that it's listed in the list of scheduled tasks: sudo crontab -l

You can redirect to a log as well, by default it uses an email thing, 0 4 * * * /home/group-c/rsync_puppetlabs.sh > logfile.log

Backup db-c server

Use a .env file

• sudo vim .env

• Inside the .env,add the following: MYSQL_PASSWORD=yourpassword

• Inside your db script, add these lines:

source "$HOME/.env"
mysql_password="$MYSQL_PASSWORD"

For the mysqldump command itself use the mysql_password variable (and yes you do need to have NO space between the -p), sudo mysqldump "$db" -p"$mysql_password" > "$db.sql"

• Manually run the script: ./db_backups.sh

• Set up a cron job for this script

• Add this to sudoers file to bypass sudo prompt:

group-c ALL=(ALL) NOPASSWD: /home/group-c/db_backups.sh

Full script (also inside our actual repo):

#!/bin/bash

# Prompt for MySQL password, which we need to automate
# read -s -p "Enter MySQL password: " mysql_password
# echo

source "$HOME/.env"

# Retrieve MySQL password from environment variable
mysql_password="$MYSQL_PASSWORD"

# Check if MySQL password environment variable is set
if [ -z "$mysql_password" ]; then
    echo "Error: MySQL password environment variable not set. Please set the MYSQL_PASSWORD environment variable before running the script." >&2
    exit 1
fi

log_file="dbbackup_log.log"


# Server credentials
remote_username="group-c"
backup_server="10.2.0.5" # private ip of backup-c
backup_host="backup-c"

storage_server="20.211.153.89"
storage_host="offsite"

remote_directory="/home/$remote_username/database_backup"
storage_directory="/home/$remote_username/db-c"


# Get list of databases
databases=$(sudo mysql -e "SHOW DATABASES;" | grep -Ev "(Database|information_schema|performance_schema|mysql)")

# Function to handle errors
handle_error() {
    echo "Error: $1"
	echo "$(date +"%Y-%m-%d %H:%M:%S") - Error: $1" >> "$log_file"
    exit 1
}

# Another function to handle log messages for said errors
log_message() {
    echo "$(date +"%Y-%m-%d %H:%M:%S") - $1" >> "$log_file"
}

# Initialize log file
echo "Backup Log $(date +"%Y-%m-%d %H:%M:%S")" > "$log_file"

# Iterate over each database and create a backup file
for db in $databases; do

 # Create backup file
    sudo mysqldump "$db" -p"$mysql_password" > "$HOME/$db.sql" || handle_error "Failed to create MySQL dump for database $db"
    log_message "Created MySQL dump for database $db"

    # Rsync to BACKUP server
    if sudo rsync -av -e "ssh -i /home/group-c/.ssh/id_rsa_db_1" "$HOME/$db.sql" "$remote_username@$backup_host:$remote_directory/"; then
        log_message "Rsynced backup to $backup_host"
    else
        handle_error "Failed to rsync backup to $backup_host"
    fi

       # Rsync to STORAGE server (commented out until we get no password prompt to the storage server)
#    if ! sudo rsync -av -e "ssh -i /home/group-c/.ssh/id_rsa_db_storage" "$HOME/$db.sql" "$user@$storage_server:$storage_directory/"; then
#        handle_error "Failed to rsync backup to $storage_host"
#    fi
#    log_message "Rsynced backup to $storage_host"



done

echo "Backup process completed successfully"
log_message "Backup process completed successfully"

# Manual testing commands
# sudo rsync -av -e "ssh -i /home/group-c/.ssh/id_rsa_db_1" "$HOME/owncloud.sql" group-c@backup-c:~
# sudo rsync -av -e "ssh -i /home/group-c/.ssh/id_rsa_db_storage" "$HOME/owncloud.sql" [email protected]:~/db-c

# sudo mysqldump -e -p owncloud > owncloud_backup.sql
# sudo mysqldump -e -p --all-databases > db_backupv2.sql

Puppet

On the mgmt-c server in the /etc/puppetlabs/code/modules/db_backup/manifests/init.pp

class db_backup {

  # Ensure necessary packages are installed
  package { 'mysql-client':
    ensure => installed,
  }

  # Deploy the .env file
  file { '/home/group-c/.env':
    ensure  => file,
    source  => 'puppet:///modules/db_backup/.env',
    owner   => 'group-c',
    group   => 'group-c',
    mode    => '0600',
  }

  # Deploy the backup script
  file { '/home/group-c/puppet_db_backups.sh':
    ensure  => file,
    source  => 'puppet:///modules/db_backup/puppet_db_backup.sh',
    owner   => 'group-c',
    group   => 'group-c',
    mode    => '0700',
  }

}

In the files of that module is the puppet_db_backup.sh and the .env file

In the app_backup module manifest init.pp

class app_backup {

    package { 'mysql-client':
      ensure => installed,
    }


  # Deploy the backup script
  file { '/home/group-c/puppet_app_backups.sh':
    ensure  => file,
    source  => 'puppet:///modules/app_backup/puppet_app_backup.sh',
    owner   => 'group-c',
    group   => 'group-c',
    mode    => '0700',
  }

}

In the files of that module is the puppet_app_backup.sh

In the backup_backup module manifest init.pp

class backup_backup {

    package { 'mysql-client':
      ensure => installed,
    }


  # Deploy the backup script
  file { '/home/group-c/puppet_backup2storage.sh':
    ensure  => file,
    source  => 'puppet:///modules/backup_backup/puppet_backup2storage.sh',
    owner   => 'group-c',
    group   => 'group-c',
    mode    => '0700',
  }

}