HPSS (Archive) Storage on SciNet - CoBrALab/documentation GitHub Wiki

Cobralab has a large amount of "long term" storage on scinet on the HPSS or Archive system.

This system is not recommended for everyday use however it is excellent for backing up old CIVET or MAGeTbrain runs without deleting everything.

Full documentation is available here https://wiki.scinet.utoronto.ca/wiki/index.php/HPSS

This example script will backup one directory to HPSS, make sure to edit your /path/to/folder/to/backup and FILENAMETOSAVE

#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N tar_create_compressed_tarball_in_hpss_with_hsi_by_piping
#PBS -j oe
#PBS -m e

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

# When using a pipeline like this
set -o pipefail

# to put
tar -c /path/to/folder/to/backup | pigz | hsi put - : $ARCHIVE/FILENAMETOSAVE.tar.gz
status=$?
if [ ! $status == 0 ]; then
   echo 'TAR+PIGZ+HSI+piping returned non-zero code.'
   /scinet/gpc/bin/exit2msg $status
   exit $status
else
   echo 'TRANSFER SUCCESSFUL'
fi