Backup of project folders - GlascherLab/LabWiki GitHub Wiki

If you mirror your project to GIN, then this serves also as a online backup of your project.

The most convenient way to backup a project folder is done with the Linux tool rsync. The generic command to create and update a backup on an external hard disk is:

rsync -aA FROM TO

where FROM specifies the projects folder (e.g. glaescher@dendrite:/projects/crunchie/glaescher) and TO specifies the mount point of the backup disk (e.g. glaescher@currin:/media/usbX), where currin is the designated backup workstation located in the printer room in the east wing and /media/usbX is the mount point (e.g. /media/usb0)

1. Take you backup disk to the east printer room and plug it into the USB hub connected to the currin workstation
2. Log into the currin workstation using your Linux/Kerberos password and check the mount point of your backup disk. Note: plugging in the hard disk triggers a hotplug event that results in a notification on the screen. This might already list the mount point, so you may not have to log in at all
3. Return to your desk and start/resume a VNC session and ssh to the backup workstation or start a screen session (see Note below) on the backup workstation. Both will insure that you can start your backup, close the window and the backup will run in the background
4. Start your backup with the following command rsync -aA user@host:/projects/crunchie/<user>/folder /media/usbX. Example: Backup the entire folder /projects/crunchie/glaescher to an external hard disk:

glaescher@dendrite: ssh currin
glaescher@currin: rsync -aA glaescher@dendrite:/projects/crunchie/glaescher /media/usb0/

5. Once the command is done, you can exit your ssh session and walk over to the east printer room. Log into the currin workstation, open an terminal and type sync to insure that all write operation on any hard disks are completed. Unplug your hard disk and log out again.

Restoring a backup involves the reverse procedure as creating it.

1. Connect backup disk to backup workstation and identify mount point
2. ssh to the server where the backup should be restored (e.g. ssh glaescher@dendrite)
3. glaescher@dendrite: rsync glaescher@currin:/media/usb0/glaescher /projects/crunchie
4. After completion, unplug your backup disk form the backup workstation in the east printer room

• For some reason (only known to Peter Kammer, so ask him if you want to know the details), it is advisable to ssh to the workstation, which has the backup disk connected first, and then when using rsync pull the data from the FROM server toward you (on the TO workstation).
• Choice of the filesystem on the backup disk is important. Most of them come with an NTFS filesystem preconfigured. Linux can read and write onto that filesystem, but MacOS can only read NTFS (but not write). If you need to write to the backup disk from MacOS you can format the backup disk as extFS (note: this is NOT ext3/ext4 the native files system under Linux), but this will not preserve the ownership and file permissions from Linux (all files will be root:root and rwx which can cause some trouble when restoring the backup. Formatting the disk as APFS (Apple File System) is NOT recommended, because you would have to connect your disk to your Mac and run rsync on your Mac "over the ISN network". This has caused "broken pipe" error and terminated the rsync run.
• IMPORTANT: Make sure that the source folder is NOT terminated by a / when running rsync, e.g. rsync -aA /projects/crunchie/glaescher /media/usb0/. Only then you will get a glaescher folder on the backup disk with all subfolder in it. If you add a file /(which is done automatically by TAB expansion under Linux), you will copy the content of /projects/crunchie/glaescher (i.e. all subfolders) to the root folder of the backup disk.
• On MacOS there is no 'A' option (access control list). Then you can just use the -a option (archive).
• It is possible to use the same backup prodecure without a VNC client (for those Shell/Terminal lovers out there). The screen command under Linux will create a user session that continue to run in the background, even if you sign off / terminate your Linux shell. Here is how you would do it

### creating/updating a backup on a harddisk connect to the currin server
ssh currin                    # connect to the currin backup server
screen -S glaescher-backup    # create a screen session with the name 'glaescher-backup'
rsync -aA glaescher@dendrite:/projects/crunchie/glaescher /media/usb0/ # rsync command as above

### now you can quit the terminal and the session will continue to run in the background ...

### after a while to go back to the user session on currin to check the backup
ssh currin
screen -x glaescher-backup    # reconnect to user session 'glaescher-backup'

### retrieving a backup from harddisk connected to the currin server
ssh dendrite
screen -S glaescher-backup    # open a new user session on dendrite server under the name 'glaescher-backup' 
rsync -aA glaescher@currin:/media/usb0/ /projects/crunchie