Contact the HPC team to get an account on the HPC cluster. They will give you the instructions to connect to the cluster via ssh.

Once you have an account on the HPC cluster, you can access the cluster via ssh. You can use the following command to access the cluster:

ssh -l <username> login<node>-hpc.rockefeller.edu

Once you are logged in, you can setup Spack by adding the following lines to your .bashrc file:

# Spack
SPACK_RELEASE=spack_2020b
source /ru-auth/local/home/ruitsoft/soft/spackrc/spackrc.sh

Latest instructions for setting up Spack can be found here

Restart your terminal and run the following command to initialize Spack:

lmodinit

Load the modules needed for building ANTs:

module load gcc/10.2.0-h4gobj
module load cmake/3.17.3-3rjy3k

Note: You can cross-reference the modules available on the HPC cluster using module avail gcc and module avail cmake with the suggested versions for compiling ANTs as mentioned here, and choose the most appropriate version.

Install ANTs using the instructions here. When configuring using CMake, make sure to have BUILD_ALL_ANTS_APPS as ON.

Then follow the instructions here to setup Conda. Once you have setup Conda, you can create a new environment for running the ANTs software by running the following command:

conda create -n ants python
conda activate ants

Then you can install poetry by running the following command:

curl -sSL https://install.python-poetry.org | python3 -

Newer instructions for installing poetry can be found here.

Follow the instructions from step 2 in the previous section to complete the setup. It is preferred to setup the repository in the scratch folder on the HPC cluster for the lab. For the Kronauer lab, the scratch folder is located at /rugpfs/fs0/kron_lab/scratch/<username>.

An easy way to deal with scratch is to use a symlink to the scratch folder from your home directory. To do this, run the following command:

ln -s /rugpfs/fs0/kron_lab/scratch/<username> ~/scratch

Then you can navigate to the scratch folder by using the following command:

cd ~/scratch

Then you can clone the repository by running the following command:

git clone https://github.com/neurorishika/ant_template_builder.git 

Go inside the ant_template_builder folder and run the following command to build the environment:

poetry install

At this point, you should have the ANTs software installed and the environment setup. You can now run the template generation pipeline as described in the previous section. The only difference is that you will have to requisition a node on the HPC cluster to run the registration.

RECOMMENDED PARTITIONS ON RU HPC CLUSTER(AS OF DEC 2023):

1. bigmem (for running the template generation pipeline with a massive dataset with maximum parallelization)
2. hpc_a10 (for running the template generation pipeline with a small dataset with maximum parallelization or a medium dataset with moderate parallelization (recommended for most cases))

Copy data from your local computer to the HPC cluster using scp or rsync. For example, to copy the data from your local computer to the HPC cluster, run the following command:

rsync -avz <local_folder> <username>@login04-hpc.rockefeller.edu:/rugpfs/fs0/kron_lab/scratch/<username>/<remote_folder>

Copy data from the HPC cluster to your local computer using scp or rsync. For example, to copy the data from the HPC cluster to your local computer, run the following command:

rsync -avz <username>@login04-hpc.rockefeller.edu:/rugpfs/fs0/kron_lab/scratch/<username>/<remote_folder> <local_folder>

In most cases you will either need to copy the cleaned data to the HPC cluster or copy the final template from the HPC cluster to your local computer.

Use screen to run the python scripts on the HPC cluster using interactive mode. This will allow you to run the scripts in the background and disconnect from the HPC cluster without interrupting the scripts. First connect to the HPC cluster using ssh and then run the following command:

screen -S <screen_name>

Check which partitions have nodes that are idle by running the following command:

sinfo | grep idle

If you find a hpc_a10 or bigmem partition that has idle nodes, start a new interactive session on the HPC cluster on an exclusive node by running the following command:

srun -p <hpc_a10 or bigmem> --time=<however long you need in DD-HH:MM:SS format> --exclusive --pty -i /bin/bash

Once you are on the node, navigate to the scratch folder and start the python scripts of your choice as described in the previous section. Once the scripts are running, you can disconnect from the HPC cluster by pressing Ctrl + A + D. To reconnect to the screen, run the following command:

screen -r <PID>.<screen_name>

Please check the status of the scripts by reconnecting to the screen and checking the status of the scripts. Once the scripts are complete, you can exit the screen by pressing Ctrl + D.