Installing on a General HPC System - McIntosh-Lab/tvb-ukbb GitHub Wiki

General and HPC installation is now relatively straightforward thanks to the encapsulation of the core software dependencies in a Singularity container. This streamlines the process for submitting subjects as jobs to a HPC cluster using a job scheduler, e.g. SLURM. If you do not already have Singularity installed on your target machine, you can check out their installation instructions here.

Installation Steps

Download the TVB-UKBB Singularity Container

There are currently two versions of the container. If you are using the Cam-CAN_dev_CPU or ADNI3_dev_CPU branches, you can download the Singularity container from here to your desired location.

wget "https://public.boxcloud.com/d/1/b1!K3QLzPHRpJdCqRrpMfUFzjEfIZKlgkzrfEansIMWCESHYZM5XNgAIr-VhAsXKIXHYPC5qFvhOlW34Dw9s-uvD8qTR_06oZMpKN9zbCXz1nRK0nKT5eMLiMgt3h_YAh-8gIXgjFDGwzDm3a0AZkUqJd-V0uAuoRU6kgp-LoUsG-WT1z11fMehUOnvDd_Tqr8lLKkUtEACCxuPBzkq21rm2lWwdcurpptpjgD1aggJC2zC4j1ljNylF9VbY-AagS4ws-_69rQJfmWrfUflqNRReiVLf9djgH4k8H-e1Zwazn5T7YiqIhLYPEJquC28Xnc9D3lr6Tg5YzQGdlK60OGI69B84DjJp6tNk4tNl9o7YPZJALdvjI0aLusZEBk2O6X6mVCyKaXaNU99v7VJyPj--X9aASbJPRrBUV2vwhElF1mVzZAL3IX_7HWt9jHvopYgx87LjnX8r00IeZxhnnQVgYMOasiKQp3JyD4IqRJtrd3LwaPqNfPJqSZX6sl0MNsul8rWcwkyhW3llIVqPGvcZY7xgv7q1u194xqifPOa-ueJAUXOxFfXLYqB8rQX6rgcG57TxUom7pRV0vI6hjCuNHEW0FHSUiB1P-NKnD1QkzB73Vl0Lbc33oRN1149stEoFzHw2eOG3gLPnJIYLyvjTcJHBQjzfgxheEp4g-Tde37Dn9fLTALHOFzhyBxPTDNHt-igAZrwI2FJeGFtU1_3QOtIn66q_l8R81bcfwqD_QHsxG1CLumTENjnZUUMYJA3U-DVNTZ5OTk_QwC2WZWglibGXppBwrC1toSwf0TtL5BTzEsB2zk_Tlct6pJXjFhek1_4pEQQHjZt4Xb-OPEWu57IMOIZs19cc_d9wsXOo2-tX6m-ypB6gYHNNnWF8dk-LwWWRc2MprPvgIc5xrnFcJyTnprJ8XAiBg7F7rgOCvNhUxmzpmzV1awAK09J8w9kTLS2UnnnMr-PuNUePbSt3upWJs0u3AMkkqLhR_-0FUxKPoQC6IghKpdL2NJ9EnMyzgC09uHTW9_OkplCbDr0zYCKiVZkscgoqDh_rra6co0A51L6LbAU8e2iyBk3oIVqE4E2SezqBanyWqS1_GzCDO9N95W_1gmQgZtwmppWk08TFL8E9_-DCejIInWAOxO9MqSXQDxWSkYJmb1ZBxthnF3txn035psQLqaURYVEOsXFzkT8YBIXkKZXADdKYnnAQ_r39ST-EV5s0dKNsPNTM3U3rpg-XeQKLWVbNfNmYf3TvZCawnXj017lb9DwjbjoY2GR4DV5zvXkiTNzRl6DlDA2uZd4DUzEacfpBE9eQA../download"

Otherwise, pull the Singularity container from the Sylabs Cloud library to wherever convenient. You can check out the webpage for it here.

singularity pull library://noahfl/tvb/tvb-ukbb:latest

Install git-lfs

Install git-lfs so large files from the repository can be downloaded properly. This only needs to be run once for your user account:

git lfs install

Download the TVB-UKBB pipeline

Note: the container doesn't have the repository installed into it since the container is read-only, so you have to have your own copy of the pipeline installed. Clone the git repository to wherever convenient:

git clone https://github.com/McIntosh-Lab/tvb-ukbb.git

We release bugfixes and introduce new features on a rolling basis. If you have been working on a local version or a fork of the pipeline, please be sure to pull the most recent version of the pipeline before you run your data.

Modify init_vars

Modify init_vars in the repository's root directory so that the appropriate environment variables point to the correct locations of your remaining installed dependencies. Here are the variables you'll most likely need to change:

AFNIDIR

FREESURFER_HOME (might be commented out; if so please uncomment)

ANTSPATH

If your machine does not have a GPU, change the following to false:

PROBTRACKX_GPU

Other variables you might want change include:

SUBJECT_AGE_LIST

AGE_SPECIFIC_TEMPLATE_LIST

PARC_IMG

PARC_LUT

SUSCEPT_ROIS

SUSCEPT_PARC_IMG

TRAINING_FILE (may need to create training file yourself)

Usage

Running a subject

To try out the container on a subject, (modify and) try the following:

cd /path/to/subs

singularity run --nv -B /home /path/to/tvb-ukbb_latest.sif subject_dir /path/to/tvb-ukbb

Using an interactive session

To run an interactive session using the container, (modify and) try the following:

singularity shell --nv -B /home /path/to/tvb-ukbb_latest.sif

Once in the shell, source your init_vars file:

. /path/to/tvb-ukbb/init_vars

Now you can run parts of the pipeline individually or use the environment interactively.

Next Steps

Visit this page for instructions for running a subject on Compute Canada.

HPC Notes

If you'd like to see how our configuration looks for our HPC setup on Compute Canada, check out the Compute Canada installation and usage page.