Running Subjects on the Digital Research Alliance of Canada's infrastructure - McIntosh-Lab/tvb-ukbb GitHub Wiki

This page contains instructions for running subjects through the TVB-UKBB pipeline on the Advanced Research Compute (ARC) infrastructure from the Digital Research Alliance of Canada.

module load StdEnv/2020 gcc/9.3.0 afni/20.3.05 freesurfer/7.1.0 
module load apptainer

If you are only using the Cam-CAN_dev_CPU or ADNI3_dev_CPU branches, run the following instead:

module load StdEnv/2020 gcc/9.3.0
module load apptainer

Once the Singularity Image has been loaded, there are three options for submitting a single subject:

1. Running submit_subject.sh
2. Running singularity run
3. Running bb_pipeline.py from inside the Singularity Image

cd to the directory containing your subjects before running your subject with the sbatch command. <subject> should be the name of the subdirectory for your subject (e.g. sub-0001)

Please note that if you'd like to re-process an previously processed subject (including failed processing), you will need to clear your subject's folder so that it only contains the rawdata subdirectory. The exception for this requirement is when reparcellating, in which case the subject folder can be left intact.

cd <path/to/dir/containing/subjects>
sbatch -J <subject> <path/to/submit_subject.sh> <subjDir>

Here is an example submit_subject.sh script (modify file paths accordingly):

#!/bin/bash
#SBATCH --account=<account_name>
#SBATCH --mail-user=<email_address>
#SBATCH --mail-type=FAIL
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=6
#SBATCH --mem=32G
#SBATCH --time=0-6:00
#SBATCH --output=log_%x_%j.o
#SBATCH --error=log_%x_%j.e
	
singularity run --nv -B /scratch -B /cvmfs -B /project <path/to/singularity/image/file> ${1} <path/to/tvb-ukbb_pipeline>

Note: the TVB-UKBB pipeline requires a P100-type GPU to properly run the CUDA-enabled versions of BEDPOSTX, PROBTRACKX, and EDDY. These are upstream requirements due to the way these FSL libraries are implemented. Please check your Compute Canada node's available systems and their specifications to see if these are available. If they aren't, consider using the batched CPU version instead or switching to a node that fits these requirements.

This method allows you to directly execute the container on a subject without using SLURM. This will run on the machine you're logged into so be careful if you're just on a login node. Make sure to allocate an interactive session on Compute Canada before running this. Ensure you are in the directory containing all subject folders. <subjDir> should be the name of the subdirectory for your subject (e.g. sub-0001)

singularity run --nv -B /scratch -B /cvmfs -B /project <path/to/singularity/image/file> <subjDir> <path/to/tvb-ukbb/>

This method creates an interactive session where you execute the container as a terminal. From inside the container you can run subjects, test out changes, etc. Interactive mode will likely not complete a full subject if you're running it in a timed live allocation, so keep this in mind. Again, make sure to allocate an interactive session on Compute Canada before running this.

Enter the Singularity Shell:

singularity shell --nv -B /scratch -B /cvmfs -B /project -B ~/.Xauthority <path/to/singularity/image/file>

Initialize environment variables with:

. <path/to/tvb-ukbb/init_vars>

cd to the directory containing your subject directory (e.g. subject directory named <subjDir>. <subjDir> must be the name of the subject's directory only - not the full file path to that directory).

Run the subject with

python <path/to/tvb-ukbb>/bb_pipeline_tools/bb_pipeline.py <subjDir>

Your best bet for submitting multiple subjects is to use a combination of some sort of for-loop and approach 1), the submit_subject.sh script. A script like this:

#!/bin/bash
#
#  This script submits all subject_list.txt subs in the current working directory to be run.
#
#  Usage:   ./batch_subjs.sh  <path>/<to>/subject_list.txt
#  
#

while IFS= read -r subjname; do
	sbatch -J $subjname /<path>/<to>/submission_scripts/submit_subject.sh $subjname		#TO BE MODIFIED BY USER
done < "$1"

would allow you to iterate over a .txt file containing one subject name per line and submit each subject. Be sure that the Singularity container and pipeline command parameters are set correctly.