Goal

On the Trillium cluster there is a maximum of 500 submitted jobs per user. In order to best use the cluster, we want to submit as many small jobs as possible while remaining under the limit. At the same time we need to specify enough time for the commands to finish. This document teaches you how to determine the optimal job packing and runtime limit to maximize cluster usage.

Purpose

This page is a general guide on how to test, time, and schedule DBM jobs. The gist is, by breaking your jobs into smaller “chunks”, you can often take advantage of backfill scheduling to reduce queue wait times. This applies all the time (and is especially useful while Niagara is running on reduced capacity).

1. Start an Interactive Debug Session

$ debugjob

This will allocate you one machine for one hour in interactive mode. You can experiment with time and memory usage without waiting in the normal job queue. Use this to run quick tests to see how long a single registration step might take. It eliminates guesswork when setting walltime or memory limits for your real jobs.

2. Set Up Your Environment

$ module load cobralab

3. Manually Run One Registration

First, identify your:

• Fixed image (fixed_image.nii.gz or your template)
• Moving image (subject_image.nii.gz)
• Masks (e.g., starting_target_mask.nii.gz, subject_mask.nii.gz), if you use masks to refine registration.

Then, run (example command; adjust to suit your DBM pipeline.):

> /usr/bin/time -v \
 antsRegistration_affine_SyN.sh \
 --fixed-mask starting_target_mask.nii.gz \
 --moving-mask subject_mask.nii.gz \
 subject.nii.gz dorr.nii.gz \
 /tmp/prefix

After the run, the output of time -v line Elapsed (wall clock) time will provide elapsed (wall-clock) time.

4. Check Elapsed Time & Memory

• Review the time -v output:
• Elapsed (wall clock) time: Approx. how long one registration takes.

5. Adjust qbatch Environment Variables & Chunksize

modelbuild.sh and twolevel_modelbuild.sh can submit multiple registrations in batches. Environment variables (QBATCH_CHUNKSIZE) control how many subjects per chunk. Smaller chunks → shorter individual jobs → better chance of fitting into backfill. If you put QBATCH_CHUNKSIZE=1, it means that qbatch will split your job to the smallest indivisible jobs, which are each registration/DBM run. The maximum number of jobs to be submitted on Trillum is 500, so you have to make sure that your total number of jobs is less than 500.

500 jobs maximum export QBATCH_CHUNKSIZE=<PICK NUMBER 1-40> changes how much work is "packed" Make jobs as small as possible (stay under 500) and scale walltime appropriately modelbuild.sh has ~ subjects*5+4 jobs twolevel_modelbuild.sh has ~ scans*5+4 + subjects*5+4 jobs Walltime = estimated time above * QBATCH_CHUNKSIZE