[TOC]

Between the moment you type qunex hcp_pre_freesurfer … and the moment that command's Python function is called, QuNex has to answer three questions: which command is being run, which sessions it is being run over, and what value every one of its parameters has. This paper is about how those three answers are found.

It is written for developers who are about to write, extend or debug a QuNex command, and it assumes no previous acquaintance with the dispatch code. It starts at the shell prompt, follows one invocation through to the function call, and ends with what a command author has to do to take part.

Two shorter documents sit alongside it. Command registry and command types describes how a command is registered and what a command type means. Developer utilities and classes lists the classes themselves. The companion paper The QuNex logging whitepaper picks the story up where this one ends: this paper takes you to the point where a command is called, and that one describes what it writes while it runs.

Everything described here is the Python implementation. The shell front end and the container launcher have parts to play and are described where they do, but every decision about sessions and parameters is made in Python, in one place.

A QuNex invocation carries three kinds of information, mixed together on one command line.

Which command. The first word. The registry turns it into a qx_command entry, which knows the command's language, its type, the arguments in its signature and the options its documentation declares.

Which sessions. Three parameters answer this, and they divide the job cleanly between them:

What every parameter is set to. A value can be stated in five different places, and they form a fixed order of precedence. Section 5 is about that order.

The rest of this paper follows those three answers being found, in the order the dispatcher finds them.

bin/qunex.sh (reachable as qunex, qx and bin/qunex) is what a user normally types. It sets the environment up, answers --version, --help and --envsetup itself, remaps a deprecated command name, and then hands over.

The hand-off is a list. gmri -available prints the names of the commands the dispatcher runs, and the front end forwards anything on that list:

# ask the dispatcher which commands it runs
gmri_commands=`gmri -available`
# ...and if the requested command is one of them, forward the whole call
if [[ $is_gmri_command == 1 ]]; then
    qxutil_command_to_run="$1"
    bash_call_execute          # composes and runs `gmri <command> <arguments>`
    exit $?
fi

gmri_commands() in python/qx_registry.py is that list, and it is short to read:

def gmri_commands(self) -> List[str]:
    # every registered command except run_turnkey, which keeps its own path
    return [c.name for c in self.iter() if c.name not in ("run_turnkey",)]

So the front end forwards every command of every language — Python, MATLAB and Bash alike — with run_turnkey as the single exception it handles itself.

python/qx_utilities/gmri is where the work happens. It is an ordinary Python program with no .py extension, and it can be called directly: gmri hcp_pre_freesurfer --batchfile=… does exactly what qunex hcp_pre_freesurfer --batchfile=… does. Its main() parses the command line into a dictionary, and runCommand(command, args) does everything else.

The parse is deliberately plain — --key=value becomes args["key"] = "value", and a bare --flag becomes args["flag"] = True:

for n in range(1, len(args)):
    if "=" in args[n]:                     # --key=value
        k, v = args[n].split("=", 1)
        opts[k.strip("-")] = v
    else:                                  # --flag
        opts[args[n].strip("-")] = True

Nothing is interpreted here. Which of those names is a real parameter, what its default is and what type it should be are all settled further down.

bin/qunex_container is a different kind of entry point: it runs outside the container, on a login node, in whatever Python the host provides. Its job is to work out how big the run is, write one or more job scripts, and submit them. Each job script invokes qunex inside the container, so every path in this paper is travelled again there. Section 11 covers what the launcher decides for itself.

Before anything is read, the three steering parameters are put into their canonical form. This happens in general/commands_support.py, in normalize_session_parameters, called from check_deprecated_parameters — one place, for every entry point and every command.

Two legacy spellings are recognised and remapped:

• --sessionids is an alias of --sessions. It is remapped with a warning. Giving both, with different values, is an error rather than a silent choice.
• --sessions=<path to a batch file> is remapped to --batchfile=<path>, with a warning.

Whether a value is a path is decided from its shape:

def is_batchfile_path(sessions):
    # a session specification is a list; a path is a single item
    if len(sessions.split()) > 1 or "," in sessions or "|" in sessions:
        return False

    # a *.list file is a session specification of its own
    extension = os.path.splitext(sessions)[1].lower()
    if extension == ".list":
        return False

    # what is left is a path if it has a folder component or a file extension
    return os.sep in sessions or extension != ""

So a batch file that is not called *.txt is recognised, and a session id that happens to contain .txt is not mistaken for a file.

The remap of --sessions=<batch file> is a warning rather than an error because run_turnkey.sh hard-codes that spelling in its internal calls. One constant governs it:

# set to True to make the legacy spelling an error; that is the whole change
SESSIONS_AS_BATCHFILE_IS_ERROR = False

python/qx_utilities/general/batch_io.py reads batch files and list files and selects sessions within them. It is the only implementation of that job in the suite.

batch_io imports fnmatch, glob, os, re, collections.UserList and copy.deepcopy — the standard library and nothing else. It raises its own BatchError rather than using general.exceptions.

The reason is bin/qunex_container, which cannot import QuNex at all (it runs on the login node, outside the container) and therefore carries a spliced copy of this file. python/tests/test_batch_io.py holds the module to the standard library, so the rule cannot quietly lapse.

resolve is the whole of the "which sessions" logic in one readable function:

# a batch file is read as a *.list file if it has that extension, as a batch
# file otherwise — and only a batch file has a header
if batchfile and batchfile.strip():
    if re.match(r".*\.list$", batchfile):
        records = read_list(batchfile)
    else:
        records, header = read_batch(batchfile)

# a *.list file may also arrive through --sessions: it is the source of the
# sessions when there is no batch file, and selects within one when there is
if sessions and re.match(r"^\s*\S+\.list\s*$", sessions):
    ...

# finally, narrow by the session specification and the filter
slist = select_sessions(records, sessions=sessions, filter=filter,
                        sessionsfolder=sessionsfolder)

Session ids and filter values are matched as globs — *, ?, [abc] — through fnmatch.fnmatchcase, everywhere and for every entry point. --sessions=S1* selects S10 and S11; --sessions=S1 selects S1 and nothing else.

A session specification is split on commas, spaces or pipes, so --sessions='S01,S02', --sessions='S01 S02' and --sessions='S01|S02' are the same request.

--filter takes <key>:<value> pairs against the arbitrary key-value information a batch file records for each session:

--filter="group:control"                 # one criterion
--filter="group:control|group:patient"   # OR — either one
--filter="group:patient&task:rest"       # AND — both

Only one operator may appear in one filter; mixing | and & raises a BatchError. An empty filter selects everything.

general/core.resolve_sessions() is what the rest of QuNex calls. It is a thin wrapper that adds the two things batch_io cannot have:

try:
    slist, header = bio.resolve(batchfile=batchfile, sessions=sessions,
                                filter=filter, sessionsfolder=sessionsfolder)
except bio.BatchError as e:
    # the QuNex error type, tagged with the command that was being run
    raise ge.CommandFailed(command, "Could not compile the list of sessions "
                           "to process", str(e), "Please check your parameters!")

# and, when running as a SLURM job array, this task's share of the sessions
if "SLURM_ARRAY_TASK_ID" in os.environ:
    slurm_array_ix = int(os.environ["SLURM_ARRAY_TASK_ID"])
    slurm_array_size = int(os.environ["SLURM_ARRAY_TASK_MAX"]) + 1
    slist = slist[slurm_array_ix::slurm_array_size]

return slist, header

It returns a pair: the sessions, and the parameters the batch file's header states. That second half is what the next section is about.

A batch file that is absent or unreadable is always an error.

A parameter's value can be stated in five places. They form a fixed order, each tier overriding the ones before it:

Tiers 1 to 4 are merged once per invocation. Tier 5 is applied per session, as each session is reached, because it differs between them.

general/process.merge_options(command, args, header) performs the merge and returns (options, sources, stated).

options = {"command_ran": command}
sources = {"command_ran": "default"}

def take(source, items):
    # each tier writes over the one before it, and records that it did
    for key, value in items:
        options[key] = value
        sources[key] = source

take("default", [(line[0], line[1]) for line in arglist if len(line) == 3])

if header:
    # the header goes through the same deprecation remapping as a command line
    take("batch file", gcs.check_deprecated_parameters(header, command).items())

# the command line — or the recipe, for a step a recipe started
from_recipe = set(filter(None,
                  os.environ.get(gcs.RECIPE_PARAMETERS, "").split(",")))

for key, value in args.items():
    source = "recipe" if key in from_recipe else "command line"
    ...

options is the merged dictionary, with every value put through the arglist converters. This is what a processing command is handed.

sources is a parallel dictionary of the same keys, each naming the tier its value came from. Keeping it beside the values rather than replacing each value with a (value, source) pair means that every existing reader of options is unaffected — options["overwrite"] is still a bool, not a tuple.

stated is what the tiers actually wrote, captured after the deprecation remap and the variable expansion but before the converters run:

# what the tiers said, before the recode below has its way with it
stated = {key: options[key] for key, source in sources.items()
          if source != "default"}

This third dictionary exists because the two audiences want the same value spelled differently. A processing command is handed the whole options dictionary and documents overwrite as a bool. A utility command is called with parameters and its signature says overwrite="no", its body doing overwrite.lower() == "yes". Filling the converted True into that signature raises an AttributeError. So a command called with parameters is filled from stated, and a command handed the dictionary gets options.

A recipe runs each of its steps as a separate process, and a command line carries values but never says where they came from. So run_recipe names them in the step's environment:

# every parameter the recipe put on this step's command line came from the
# recipe, whatever wrote it — the recipe file, its global parameters, or the
# call that started the recipe
with subprocess.Popen(
        command,
        env={**os.environ, gcs.RECIPE_PARAMETERS: ",".join(command_parameters)},
) as process:
    ...

merge_options reads that variable and tags those names recipe instead of command line. The batch file a step is given it reads for itself, so that tier still names itself.

The sequence matters, and one part of it is not obvious: the batch file can name a different study, and the study is what the logging settings and the log folder are read from. So the sessions and the merge have to be settled before anything is logged.

gmri.runCommand runs in this order:

# 1. a first guess at the study, sessions and log folders, from the command line
folders = gc.deduce_folders(args, command, timestamp)

# 2. deprecated command names and deprecated parameters, including the
#    canonical form of --batchfile / --sessions / --filter (section 3)
command = gcs.check_deprecated_commands(command)
args = gcs.check_deprecated_parameters(args, command)

qx_command = qx_commands.get(command.strip())

# 3. which sessions, and what the batch file's header states — for every
#    command class, because every command class can use the header
sessions, batch_header = gc.resolve_sessions(
    batchfile=args.get("batchfile"), sessions=args.get("sessions"),
    filter=args.get("filter"), sessionsfolder=folders["sessionsfolder"],
    command=command)

# 4. the tiers merged into one dictionary (section 5)
options, sources, stated = gp.merge_options(command, args, batch_header)

# 5. the parameters a command declares that it was not given (section 7)
filled, _ = gcs.select_parameters(stated, sources, qx_command)
filled = {key: value for key, value in filled.items() if key not in args}

# 6. the threads the command may use
gcs.set_omp_threads(options)

# 7. the folders again — the header may have moved the study
if batch_header:
    folders = gc.deduce_folders({**batch_header, **args}, command, timestamp)

# 8. how, and whether, this run is logged — now on the settled study
log_settings = gl.resolve_logging(command, args, qx_command=qx_command,
                                  studyfolder=folders["basefolder"], ...)
gl.set_active(log_settings)

# 9. one runlog for the whole invocation, shared by every call it makes
run = gl.RunContext(command, args, log_settings, folders, timestamp=timestamp)

# 10. the runlog header, and the parameter report (section 8)
...

# 11. dispatch, by command class (section 9)

Step 7 reads {**batch_header, **args} rather than the merged options for the same reason stated exists: the merged dictionary answers for folders nobody named, because arglist gives them defaults.

set_omp_threads at step 6 sets OMP_NUM_THREADS to --omp_threads if the run states one, and otherwise to the cores this process may run on divided between the parallel jobs it was asked for, clamped between one and eight. An environment that already states the variable is left alone.

There are two ways a command receives its parameters, and which one applies depends on the command class.

A processing command is handed the merged dictionary. gp.run passes options through, and the command reads options["hcp_brainsize"] and the rest of what it needs. Nothing has to be selected: the command takes the whole thing.

Every other class is called with parameters. A Python utility command is called as function(sourcefolder=…, overwrite=…); a MATLAB or Bash command is called with --parameter='value' on a command line. So the merged dictionary has to be narrowed to what that command can actually accept.

def select_parameters(options, sources, qx_command):
    declared = declared_parameters(qx_command)

    accepted, dropped = {}, []
    for key, value in options.items():
        source = sources.get(key, "default")

        if source == "default":
            continue                    # nobody stated it; the command's own
                                        # default is the better answer
        elif key in declared:
            accepted[key] = value       # the command takes this one
        elif key not in extra_parameters and source not in RUN_WIDE_SOURCES:
            dropped.append(key)         # stated for this command, and unusable

    return accepted, dropped

Two rules are worth spelling out.

Only stated values are returned. A parameter nobody named is left out, so the command keeps its own default for it. This is what makes the whole scheme safe to apply to every command: filling in a value the run never mentioned would override a command's own considered default with arglist's generic one.

The command line is layered back on top. In gmri, immediately after the call:

filled, _ = gcs.select_parameters(stated, sources, qx_command)
# what the user typed reaches the command exactly as the user typed it
filled = {key: value for key, value in filled.items() if key not in args}
bargs = {**bargs, **filled}

That second line is what "never override" means in practice. A batch file header fills a parameter in; a command line always wins.

declared_parameters asks the registry for both halves of a command's interface:

def declared_parameters(qx_command):
    # the signature arguments, and the options the documentation declares
    return {arg.name for arg in qx_command.args} | \
           {option.name for option in qx_command.options}

This distinction matters. For a Python command the signature is the whole story. For a MATLAB or Bash command there is no Python signature at all — its parameters are documented ones and live in qx_command.options. Asking only about the signature would answer "declares nothing" for every Bash command in the suite.

A parameter that a command cannot take is either a mistake or perfectly normal, and which one it is depends on the tier that stated it:

# the tiers that state parameters for a run rather than for one command
RUN_WIDE_SOURCES = ("batch file", "batch file (session)", "recipe run")

A batch file's header states parameters for every command that will ever be run against that study, and a recipe's global parameters state them for every step. A value of theirs that this command cannot take was meant for another command, and is dropped without a word.

A parameter written against the command itself — on its command line, or against that step in a recipe — that the command cannot take is a mistake, and is named. run_recipe prints it:

for param in dropped:
    warning = ("\nWARNING: %s is not a parameter of %s and was not passed to "
               "it. Please check the recipe!\n" % (param, command_name))

extra_parameters, the third category, is the list of run-level parameters that steer how QuNex runs a command rather than what the command does — --batchfile, --sessions, --filter, --scheduler, --parsessions, --logfolder and their kin. They are never a command's business unless the command declares one of them itself.

Every run prints a table of the parameters the command is about to be run with, and where each value came from, before the command runs. The same text goes into the runlog and into the command's comlog.

---> Parameters for hcp_pre_freesurfer

     parameter        source         value
     ---------------------------------------------------------------
     batchfile        command line   /study/processing/batch.txt
     hcp_brainsize    batch file     170
     hcp_t2           batch file     NONE
     overwrite        default        False

and, when a session's own entry overrides one of them, a second block inside that session:

---> Parameters for hcp_pre_freesurfer on session S02

     parameter       source                 value
     --------------------------------------------------
     hcp_brainsize   batch file (session)   150

gcs.report_parameters(qx_command, options, sources, session=None) renders it. The scoping rule is:

• normally, the parameters the command declares — its signature arguments and its documented options;
• for a command that declares none this run has a value for, what was specified, since a table of several hundred defaults says nothing about a command that does not take them;
• inside a session, only the keys the per-session tier states, the rest having been reported once for the run.

The banner reports the value the command receives, which is why it is given {**options, **stated} for every class except processing. A report that says overwrite True above a call echo that says overwrite="yes" is worse than no report.

The registry records a language and a type for each command, and the dispatch branches on the pair.

gp.run(qx_command, args, sessions, options, sources, run_context) receives everything already settled and is left with the per-session tier and the execution:

def session_options(session):
    """The per session tier, applied and -- when there is one -- reported."""
    soptions, ssources = gcs.update_options(session, options, sources)
    if ssources != sources:
        banner = gcs.report_parameters(qx_command, soptions, ssources,
                                       session=session)
        print(banner)
        run_context.write(banner)
    return soptions

update_options is where the _key: and --key: lines in a session's own batch file block are applied, and it is the only implementation of that mapping.

A utility command is called with keyword parameters, filled as section 7 describes. Whether it is run once or once per session is decided by whether the dispatcher has a way to hand it a single session:

# a command runs over the sessions only when the dispatcher can hand it one:
# it either takes a session's folder, or it forwards the run's parameters on
# (`eargs`) and fans out itself
if not any(qx_command.has_arg(e) for e in ["sourcefolder", "folder", "eargs"]):
    sessions = None

Note what this does not ask. A --batchfile is not a request to run over its sessions — it is also where the header comes from, and every command can use the header. So qunex import_dicom --batchfile=b.txt picks up --unzip and --gzip from the header and runs once.

Both are called by composing a command line. general/run_bash.py shows the shape:

declared = gcs.declared_parameters(qx_command)

for key, value in args.items():
    # the run-level parameters steer how qunex runs the command, not what the
    # script does — unless the script declares one of them itself
    if key in gcs.extra_parameters and key not in declared:
        continue
    if value is True:
        arglist.append("--%s" % (key))          # a valueless flag
    else:
        arglist.append("--%s='%s'" % (key, value))

com = " ".join(["bash", script] + arglist)

A Bash script is written for one session, so gmri calls it once per session, passing the id under the name the script actually reads:

def _bash_sessions(qx_command, sessions, args):
    # a run that works *across* sessions gets the whole list in one call
    if not sessions or args.get("runtype") in ["group", "list"]:
        return [None]

    # a script that names no session is called once
    if not (gcs.declared_parameters(qx_command) & {"session", "sessions"}):
        return [None]

    return [session["id"] for session in sessions]

def session_parameter(qx_command):
    """
    The name under which a bash script is handed a single session: `session`
    when it declares that, `sessions` otherwise.
    """
    return "session" if "session" in gcs.declared_parameters(qx_command) \
                     else "sessions"

Each call gets its own comlog, named <command>[_<calculation>|_<modality>]_<session>_<timestamp>, and the exit code decides the outcome: a script that exits non-zero fails the run.

The same command can be reached six ways, and all of them converge on gmri.runCommand.

run_through_scheduler chunks the sessions by --parsessions and writes one job per chunk, re-emitting the canonical parameters:

# set the sessions this job runs; the later value wins over the earlier one
c_str = c_base + ' --sessions="%s"' % sessionids_array[i]

Under a SLURM job array the split happens the other way round — every job is given the whole list and resolve_sessions takes this task's share of it, using SLURM_ARRAY_TASK_ID (section 4).

A recipe step re-enters at gmri rather than at qunex: for a command the dispatcher runs, the shell front end would only ask the dispatcher for its command list (one Python start of its own), search the MATLAB tree, create a log folder the dispatcher then does not use, re-quote the arguments and eval the same call. The user-facing spelling is still what the recipe reports, because qunex <command> is what a reader of the log would type.

bin/qunex_container runs on the login node, outside the container, in the host Python. It reads --batchfile, --sessions and --filter to work out how many jobs to submit and which sessions go in each, and then passes all three on to the command unchanged:

# the launcher decides the job sizing from these...
slist, _ = resolve(batchfile=batchfile, sessions=sessions, filter=batchfilter,
                   sessionsfolder=sessionsfolder)

# ...and the command inside the container gets them verbatim
if batchfile is not None:
    qx_command += f'--batchfile="{batchfile}" '
if batchfilter is not None:
    qx_command += f'--filter="{batchfilter}" '

Because it cannot import QuNex, it carries a copy of batch_io.py spliced into a generated region of the file:

# --- BEGIN GENERATED from general/batch_io.py — do not edit; run `qunex build_qx_container` ---
...
# --- END GENERATED ---

qunex build_qx_container (python/qx_container_build.py) performs the splice, and the result is committed, exactly as qx_commands.yaml is. python/tests/test_container_drift.py fails if the copy drifts from the source or is edited by hand.

If you change batch_io.py, run qunex build_qx_container and commit the result.

The launcher also provides list_sessions, which prints the comma-separated list of sessions a command would run over, given --batchfile, --sessions, --filter and --sessionsfolder. It is useful for checking a selection before committing a job to it, and for driving job arrays from a shell script.

Most of this machinery needs nothing from you. A registered command receives its parameters, its sessions and its banner without asking. Four things are worth knowing.

Declare batchfile if the command should see the study's parameters. A command that takes --batchfile is filled from the batch file's header, and the header is where a study states things once for every command. Declaring it is either an argument in the signature or an option in the .. qx_command: block — declared_parameters reads both.

Declare what you accept, and no more. select_parameters narrows the tiers to declared_parameters. A parameter that is neither in the signature nor in the documentation cannot be filled in, and a documented parameter the command does not read is a promise it does not keep.

Say which sessions with the canonical three. --batchfile for the file, --sessions for the ids, --filter for the key-value criteria. When you need the resolved list inside Python, call gc.resolve_sessions(); do not read the file yourself.

A Bash command names its session parameter, and QuNex uses that name. Declare session if the script reads --session, sessions if it reads --sessions. run_bash.session_parameter looks the answer up in the registry.

If you are adding a new dispatch path, one rule governs the ordering: the batch file can name a different study, so the sessions and the merge come before resolve_logging, and the folders are deduced again after the header has been read.

• One parser. batch_io reads batch files; gc.resolve_sessions is how QuNex code calls it. Do not write a second reader, in any language.
• batch_io stays a leaf. Standard library only. A test enforces it, and the container depends on it.
• Three parameters, canonical meanings. --batchfile is a path, --sessions selects by id or glob, --filter selects by key and value.
• Normalise at the front door. A legacy spelling is remapped once, in normalize_session_parameters, not per command.
• Merge once, above the dispatch. Every command class starts from the same dictionary, and the merge happens before anything is logged.
• Fill, never override. A tier fills a parameter in; what is stated above it always wins. Only stated values are filled, so a command keeps its own defaults.
• options for a processing command, stated for everything else. The converters recode values for the dictionary audience; a signature wants what was written.
• Match with globs. *, ?, [abc] — everywhere, for every entry point.
• A run says where its values came from. report_parameters writes the table before the command runs, to the console and to the logs.
• Rebuild the container after touching batch_io.py. qunex build_qx_container, then commit.