How to run MAGeT for mouse brains on the CIC cluster or Niagara - CoBrALab/documentation GitHub Wiki

Purpose of this page

This page gives instructions to run MAGeT on mouse data on Niagara. Other MAGeT-related pages on the wiki are mostly of interest if you are looking at human data.

https://github.com/CoBrALab/documentation/wiki/MAGeTBrain & https://github.com/CoBrALab/documentation/wiki/MAGeTbrain-Morphology. Those give a quick description of MAGeT and instructions that are specific to human data.
https://github.com/CoBrALab/documentation/wiki/Best-Templates-for-MAGeT & https://github.com/CoBrALab/documentation/wiki/Demographically-Stratified-Templates-for-MAGeTbrain. The instructions for template selection in MAGeT. In the case of mouse data, this step is not necessary and is simpler. For template selection, you can select 25 scans you think are representative of your dataset (in terms of age, sex, genotype) and scored high in terms of quality control (no movement, no artefacts), and change their names manually so they would appear first in the list of scans to be submitted for segmentation (eg: add a "0" at the beginning).
https://github.com/CoBrALab/documentation/wiki/Generating-MAGeTbrain-Quality-Control-Images and https://github.com/CoBrALab/documentation/wiki/MAGeT-Brain-Quality-Control-(QC)-Guide. This page explains how to check the MAGeT output for human data. You should definitely check visually the output of your MAGeT run for mouse data too, but you will have to define your own standards.
https://github.com/CoBrALab/documentation/wiki/pydpiper:-Running-MAGeT.py-on-longitudinal-data. This page describes how to run MAGeT locally on mouse data. It also describes how to analyze the MAGeT outputs after the run, so you should definitely read it. Please note that for longitudinal data, we prefer to run MAGeT separately from Pydpiper.

Running MAGeT on the CIC cluster

First, log into the CIC cluster (eg: machine #25), and head to whichever folder you want to run MAGeT in:

ssh anachl@cicws25
ssh -Y anachl@cicws25
cd /data/chamal/projects/chloe/2017_lifestyle/Chloe/3_derivatives/scan/MAGeT_DORR/

If you plan to run MAGeT with the Dorr atlas as input, create a shell script named maget-run.sh for example with the following (updated November 2019):

module load minc-toolkit/1.9.17 anaconda/5.1.0-python3 minc-stuffs/0.1.24^minc-toolkit-1.9.17 qbatch pydpiper/2.0.13
export PYDPIPER_CONFIG_FILE=/opt/quarantine/pydpiper/2.0.13/CIC-maget.cfg

MAGeT.py --verbose --pipeline-name=whatever-name-you-want \
--num-executors 150 \
--subject-matter mousebrain \
--files /data/chamal/projects/chloe/2017_lifestyle/Chloe/2_preprocessed/scan/LSQ6/QC1-pass/*lsq6.mnc

The last line is for the input files, which were preprocessed, QCed, and from which 25 templates were chosen.

If instead, you would rather use Stephanie Tullo's tweaked version of the Allen Brain Institute Atlas, the shell script should be:

module load minc-toolkit/1.9.17 anaconda/5.1.0-python3 minc-stuffs/0.1.24^minc-toolkit-1.9.17 qbatch pydpiper/2.0.13
export PYDPIPER_CONFIG_FILE=/opt/quarantine/pydpiper/2.0.13/CIC-maget.cfg

MAGeT.py --verbose --pipeline-name=whatever-name-you-want \
--num-executors 150 \
--subject-matter mousebrain \
--files /data/chamal/projects/chloe/2017_lifestyle/Chloe/2_preprocessed/scan/LSQ6/QC1-pass/*lsq6.mnc \
--atlas-library=/data/chamal/projects/stephanie/Tau35-collab-Vernon/derivatives/maget_ABA_merged_20190910/extracted_atlas

Either way, your next step will be:

module load qbatch/2.1.3
bash maget-run.sh

Running MAGeT on Niagara

If you have more than a couple hundred scans, then it is very likely that you should instead run MAGeT on Niagara. First, and if you dont have one already, create a Compute Canada on https://ccdb.computecanada.ca/security/login. From that account you can apply to obtain a Niagara account. Then, follow the instructions here: https://github.com/CoBrALab/documentation/wiki/Getting-Started-on-Niagara to have all modules enabled on your account (only needs to be done once).

Now you can set up the data into your Niagara account, for example by copying your input files from a CIC workstation into your Niagara home directory. From the CIC folder containing the files:

rsync -avz ./*lsq6.mnc [email protected]:/home/m/mchakrav/username/maget-input

Once you have those files in Niagara, you can log into your Niagara account:

ssh [email protected]

Then you just need to create a shell script with the following (updated November 2019):

module load cobralab

export PYDPIPER_CONFIG_FILE=/gpfs/fs0/project/m/mchakrav/gdevenyi/quarantine/2019b/pydpiper/2.0.13/niagara-maget.cfg

MAGeT.py --verbose --pipeline-name=whatever-name-you-want \
--subject-matter mousebrain \
--files /home/m/mchakrav/username/maget-input/*lsq6.mnc

This will use the Dorr atlas as input. Use bash to run the script:

bash maget-run.sh

If you would like to use a different atlas, copy the config file (/gpfs/fs0/project/m/mchakrav/gdevenyi/quarantine/2019b/pydpiper/2.0.13/niagara-maget.cfg) locally and make edits to the -atlas-library with the directory where the new maget atlas is located; don't forget to change the export PYDPIPER_CONFIG_FILE command line in your run-maget.sh script