rclone setup - jigneshpshah/greycube_helpmanual GitHub Wiki

#Setup rclone on client server

  • Login as frappe user

  • sudo apt-get install zip

  • curl https://rclone.org/install.sh | sudo bash

  • rclone config

    • keep backup server username/pwd ready
    • when in doubt use default options.
    • advance option is not required.

Options value

name>greycube_backup_server
Storage>28 (sftp)
host>127.0.0.1
-It should be server IP address
user>greycube_backup_user
port> 
    password > {keyinhere}
key_use_agent> false
use_insecure_cipher> true
disable_hashcheck> false

will look like this

[greycube_backup_server]

type = sftp

host = XX.123.123.223

user = greycube_backup_user

pass = *** ENCRYPTED ***

key_file_pass = *** ENCRYPTED ***

key_use_agent = false

use_insecure_cipher = true

disable_hashcheck = false

Quit and from terminal

Check if rclone config smoothly by listing directory of remote server

  • rclone ls greycube_backup_server:

#Create backup_script.sh in frappe-bench folder

  • nano backup_script.sh
#!/bin/bash 

#server - bench level variables
rclone_remote_name="greycube_backup_server"
frappe_bench_path="/home/frappe/frappe-bench"
app_version_file='/app_version.json'

cd $frappe_bench_path

#for each site copy below snippet

# individual site variables
local_site_name=site1.local
local_site_backup_path="/home/frappe/frappe-bench/sites/site1.local/private/backups"
# convention backup/domain or server or client name / site name without special charachters , use _
remote_backup_path="backup/artfactory/site1_local"

echo "Backup Starts - "$local_site_name
/usr/local/bin/bench --site $local_site_name execute frappe.utils.change_log.get_versions > $local_site_backup_path$app_version_file
/usr/local/bin/bench --site $local_site_name backup --with-files
# delete all but 10 newest files
cd $local_site_backup_path
ls -1tr | head -n -10 | xargs -d '\n' rm -f --
cd $frappe_bench_path
rclone sync $local_site_backup_path $rclone_remote_name:$remote_backup_path
echo "Backup Ends- "$local_site_name


Check manually is it working

  • Go to bench folder
  • sh backup_script.sh

Setup cronjob

  • Login as frappe user
  • Check server time
    • date "+%H:%M:%S %d/%m/%y"
    • say it is 04:31:24 31/05/19
  • Check client time using google
    • say it is 6:31
    • i.e. server is 2 hours behind. Hence cronjob should run at 10 PM that would be midnight in client country
  • use https://crontab.guru/ to generate cron time entry
    • 0 22 * * *
  • Go to server using frappe user and set cronjob
    • crontab -e
    0 22 * * * /bin/sh /home/frappe/frappe-bench/backup_script.sh >> /home/frappe/frappe-bench/logs/autobackup_log.log 2>&1
    
    • sudo cron service restart

Setup rclone for s3 Object on client server

RClone for Contabo S3 Object setup


#!/bin/bash 
# https://satcit.me/index.php/2022/03/15/contabo-object-storage-rclone
#server - bench level variables
rclone_remote_name="greycube_backup_server_contabo2_s3"

frappe_bench_path="/home/frappe/frappe-bench"
frappe_sites_path="/home/frappe/frappe-bench/sites"
app_version_file='/app_version.json'

ip4=$(/sbin/ip -o -4 addr list eth0 | awk '{print $4}' | cut -d/ -f1)

#for each site 
cd $frappe_sites_path
for f in */ ;
do
        local_site_name=${f%/}
        if [ "${local_site_name}" != "assets" ]; then
		echo "$local_site_name"
			cd $frappe_bench_path
			local_site_backup_path="/home/frappe/frappe-bench/sites/${local_site_name}/private/backups"
			remote_backup_path="${ip4}-${local_site_name}"

			echo "Backup Starts - "$local_site_name
			/usr/local/bin/bench --site $local_site_name execute frappe.utils.change_log.get_versions > $local_site_backup_path$app_version_file
			/usr/local/bin/bench --site $local_site_name backup --with-files
			# delete all but 7 newest files [1 backup has 4 files i.e. private/public/site_config/DB and app_version]
			cd $local_site_backup_path
			ls -1tr | head -n -7 | xargs -d '\n' rm -f --
			cd $frappe_bench_path
			# create bucket
			rclone mkdir $rclone_remote_name:$remote_backup_path
			rclone sync $local_site_backup_path $rclone_remote_name:$remote_backup_path
			echo "Backup Ends- "$local_site_name		
	fi
done