On Trillium, each user can have at most 500 jobs submitted (pending + running) and 150 jobs running at once. Each task of an array job counts as one job. To get model builds and DBM through the queue quickly, we want jobs that are:

• small enough to fit into backfill gaps (short, accurate walltimes), and
• few enough to stay under the 500-job limit.

This page shows you how to time your registrations, then choose a chunk size and walltimes that balance these two goals.

Trillium gives every job a full node (192 cores) and charges our allocation for the whole node for as long as the job runs. Jobs also wait in the queue based on our group's fairshare. When our usage is high, a job can wait 12–30 hours for priority.

Backfill lets short jobs skip that wait: the scheduler slots them into gaps before larger jobs start. This only works if your walltime is short and accurate. A stage that runs for 2 minutes but asks for 4.5 hours will never fit a gap.

Backfill also tests only 3 jobs per user in each cycle, and a cycle runs about once a minute. Thus backfill tests about 180 of your jobs per hour. A stage with 235 jobs needs about 80 minutes before backfill tests each job once. If you pack the stage into 34 jobs, this takes about 11 minutes (see "How to pack jobs"). Jobs that wait for a dependency are not tested until the dependency is complete.

The pipeline's default walltimes are much longer than most of our jobs need, so always set them yourself (step 5).

debugjob

This gives you one node for one hour, without waiting in the normal queue. Use it for quick timing tests only. Real work should be submitted as batch jobs, which can run up to 24 h.

module load cobralab

Pick one representative subject and identify:

• the fixed image (your starting target or template)
• the moving image (the subject scan)
• masks, if you use them

Then run one registration under /usr/bin/time -v (adjust the command to match your pipeline):

/usr/bin/time -v \
  antsRegistration_affine_SyN.sh \
  --fixed-mask starting_target_mask.nii.gz \
  --moving-mask subject_mask.nii.gz \
  subject.nii.gz template.nii.gz \
  /tmp/prefix

The default nlin stage of modelbuild.sh runs a full linear + non-linear registration, so time it without --skip-linear. Add --skip-linear only if you use the nlin-only or volgenmodel-nlin stages.

From the time -v output, note these lines:

You don't need to time the short stages (mask average, mask resample, shape update). They take seconds.

Two qbatch variables control how commands go into jobs:

export QBATCH_CHUNKSIZE=<c>   # commands per job
export QBATCH_CORES=1         # commands that run at the same time in a job (1 = one after another)

Always set both. module load cobralab sets QBATCH_PPJ=192, and both variables default to that value. If you set neither, a job gets up to 192 registrations, which all run at the same time.

• Smaller chunks -> shorter jobs -> better backfill, but more jobs.
• Larger chunks -> fewer jobs, but longer walltimes.

The pipeline submits all stages and iterations at once, so the full count is in the queue together. With n inputs, chunk size c and k = ⌈n / c⌉, each iteration of each stage submits:

Multiply by the number of iterations (default 4) and stages (default nlin only). Example: 30 subjects, masks, defaults, c=1 → 4 × (2 × 30 + 2) = 248 jobs. With c=8 → 4 × (2 × 4 + 2) = 40 jobs.

Extra one-off jobs: --starting-target dumb +1, --starting-target com +(2k + 2), --final-target +1.

twolevel_modelbuild.sh runs one modelbuild.sh for each subject with 2 or more scans (n = that subject's scans), plus one second-level modelbuild.sh (n = number of subjects). Subjects with one scan add no first-level jobs. Add the counts.

Pick the smallest chunk size that keeps your total under 500, counting any jobs you already have in the queue.

modelbuild.sh --dry-run prints the commands without submitting anything, but it does not show the job count. Use the table above.

Warning: do not use --dry-run with twolevel_modelbuild.sh. It is ignored, and all jobs are submitted.

modelbuild.sh also has per-stage options (--chunksize-short, --chunksize-linear, --chunksize-nonlinear). --chunksize-short chunks only the mask resample jobs. Mask average and shape update are always 1 job per iteration.

Trillium's minimum walltime is 15 minutes.

The "× chunk size" rule applies only when QBATCH_CORES=1, so that the commands in a job run one after another. If you run several at the same time, see "How to pack jobs".

Example: one registration takes 4 minutes and you use QBATCH_CHUNKSIZE=3 QBATCH_CORES=1: 1.5 × 4 min × 3 = 18 min → 00:20:00.

twolevel_modelbuild.sh passes these flags through to modelbuild.sh. Put them after the input CSV. If you put them before it, the script fails:

export QBATCH_CHUNKSIZE=3
export QBATCH_CORES=1
twolevel_modelbuild.sh ... inputs.csv \
  --walltime-short 00:15:00 \
  --walltime-linear 00:20:00 \
  --walltime-nonlinear 00:20:00

Once the first round of stages has run, compare the real run times with your limits:

sacct -u $USER -S today -X --format=jobname%60,elapsed,timelimit,state

If your jobs finish far below their limits, lower the walltimes for your next run.

• Timed one registration with /usr/bin/time -v
• Set both QBATCH_CHUNKSIZE and QBATCH_CORES
• Chose a chunk size that keeps total jobs under 500 (counted with the table in step 4)
• Set --walltime-short 00:15:00
• Set --walltime-linear and --walltime-nonlinear from your timing (min 15 min)
• Checked real run times after the first iteration and adjusted

If you pack registrations (see "How to pack jobs"):

• Noted W (User time + System time) and M from the timing test
• Made a first guess for N and checked that N × M < ~650 GB
• Ran the two-point test in a debug session and computed the largest N for a 10-minute job
• Set QBATCH_CHUNKSIZE=N and QBATCH_CORES=N
• For twolevel_modelbuild.sh, put the walltime flags after the input CSV
• After the first iteration, checked with seff <jobid> that packed jobs finish well under the walltime

A Trillium node has 192 cores, and a single registration rarely uses them all. Instead of one registration per job, run several at once in each job. Each job still finishes quickly enough for backfill, but you submit far fewer jobs and get much more done per node.

From the /usr/bin/time -v output of one registration (see "Time one registration" above):

Aim for a job time of about 10 minutes. That gives a 1.5× margin under a 15-minute walltime.

• First guess: N ≈ 600 × 192 ÷ W, rounded down
• Memory check: N × M must stay under ~650 GB

This guess assumes that the node is 100 % busy. Real jobs are a little slower and have a fixed start-up cost, so measure in step 4 before you decide.

export QBATCH_CHUNKSIZE=<N>   # registrations per job
export QBATCH_CORES=<N>       # run them all at once

Each registration can use any cores that are free. When some registrations finish early, the others use the free cores. You do not need to set threads.

Then pass the walltime to the pipeline, e.g. --walltime-nonlinear 00:15:00 (after the input CSV for twolevel_modelbuild.sh).

Job time grows in a straight line with N: job time = a × N + b. Two test runs give you a and b. Use your own data: the answer depends on your images and settings.

debugjob
module load cobralab
# subjects.txt: one image per line, at least 2N different subjects
for N in <N> <2N>; do
  export THREADS_PER_COMMAND=$((192 / N))   # what qbatch sets in a real job
  head -n $N subjects.txt | while read f; do
    echo "antsRegistration_affine_SyN.sh --clobber $f template.nii.gz out_N${N}_$(basename $f .nii.gz)"
  done > commands_N$N.txt
  /usr/bin/time -v parallel -j $N < commands_N$N.txt 2> time_N$N.log
  grep -E 'Elapsed|Percent of CPU' time_N$N.log
done

From the two Elapsed times t₁ (at N₁) and t₂ (at N₂):

• a = (t₂ − t₁) ÷ (N₂ − N₁)
• b = t₁ − a × N₁
• Largest N for a 10-minute job = (600 − b) ÷ a, rounded down

Pick a value of N that fits a debug hour. Both runs together should take less than ~50 minutes. After the first real iteration, check a few jobs with seff <jobid>.

This example shows the method. Your numbers will be different.

One nonlinear registration (antsRegistration_affine_SyN.sh --skip-linear). These numbers are for the non-linear part only, as for an nlin-only stage. For the default nlin stage, time the full registration as in step 3. It takes longer (about 20 % in our tests), so your W will be larger.

On its own, this registration uses about 59 of 192 cores, so roughly 70% of the node sits idle.

First guess: N ≈ 600 × 192 ÷ 12,500 ≈ 9 → try N = 4 and N = 8 in step 4. Memory: 8 × 6.3 GB = 51 GB, well under the limit.

Suppose the two test runs give t₁ = 340 s at N = 4 and t₂ = 620 s at N = 8:

• a = (620 − 340) ÷ 4 = 70 s per registration
• b = 340 − 70 × 4 = 60 s
• Largest N = (600 − 60) ÷ 70 ≈ 7.7 → N = 7

Settings:

export QBATCH_CHUNKSIZE=7
export QBATCH_CORES=7
twolevel_modelbuild.sh ... inputs.csv \
  --walltime-short 00:15:00 \
  --walltime-linear 00:15:00 \
  --walltime-nonlinear 00:15:00

For one registration stage of 235 subjects, that is ⌈235 ÷ 7⌉ = 34 jobs instead of 235 in each iteration.