[TOC]

Before a QuNex command runs, three things have to be settled: which sessions it acts on, what value each of its parameters has, and where each of those values came from. This page is the working guide to that machinery: the ideas you need, the calls you will use, the rules that hold across the code base, and the mistakes that are easy to make.

For the full account — every entry point, the dispatch order and why it is that order, and the container splice — see The QuNex invocation whitepaper. Command types and the registry are covered in Command registry and command types; the classes themselves in Developer utilities and classes.

Three parameters say which sessions. --batchfile is the path to the batch file, --sessions selects within it by id or glob pattern, and --filter selects within it by a <key>:<value> pair of the session's own entry. They mean the same thing at every entry point and for every command.

One parser reads them. general/batch_io.py is the only implementation of "read a batch file and pick sessions out of it" in the suite. gc.resolve_sessions() is how QuNex code calls it.

Parameters arrive in tiers. The defaults, then the batch file's header, then a recipe, then the command line, then what the batch file says for that individual session. Each tier fills a parameter in; it never overrides one stated above it.

The tiers are merged once, above the dispatch. gmri.runCommand merges them before it decides anything else, so every command class starts from the same dictionary — and so does the logging, because a batch file can name a different study.

Every value carries its origin. A parallel sources dictionary names the tier each value came from, and every run prints that as a table before the command runs.

A batch file is not a request to run over its sessions. It is also where the header comes from, and every command can use the header. qunex import_dicom --batchfile=b.txt picks --unzip and --gzip up from the header and runs once.

import qx_utilities.general.core as gc

# the single entry point for "which batch file, which sessions"
sessions, header = gc.resolve_sessions(
    batchfile=options["batchfile"],       # the path to the batch file
    sessions=options["sessions"],         # ids or globs, selecting within it
    filter=options.get("filter"),         # <key>:<value> criteria
    sessionsfolder=options["sessionsfolder"],
    command="my_command",                 # named in the error if this fails
)

It returns a pair: a SessionList of the selected sessions, and the parameters the batch file's header states. A batch file that is absent or unreadable is always an error, raised as ge.CommandFailed tagged with the command.

If there is no batch file, --sessions names the sessions themselves — matched against the folders in sessionsfolder when one is given, and taken as plain ids when it is not.

Inside gmri this call has already been made, and the result is handed to the dispatch. You need resolve_sessions when you are writing something that reads sessions on its own account.

Session ids and filter values are globs, matched with fnmatch.fnmatchcase: * for any characters, ? for one, [abc] for one of a set. --sessions=S1 selects S1 alone; --sessions='S1*' selects S1, S10 and S11.

A session specification splits on commas, spaces or pipes, so these are the same request:

--sessions='S01,S02'
--sessions='S01 S02'
--sessions='S01|S02'

A filter joins <key>:<value> pairs with | for OR or & for AND, one operator at a time:

--filter="group:control"                 # one criterion
--filter="group:control|group:patient"   # either
--filter="group:patient&task:rest"       # both

An empty --sessions or --filter selects everything.

Tiers 1 to 4 are merged once per invocation, in gmri.runCommand. Tier 5 is applied by gp.run as each session is reached, through gcs.update_options.

# the merge, once, above the dispatch
options, sources, stated = gp.merge_options(command, args, batch_header)

Three dictionaries come back, and knowing which one you want matters:

• options — the merged values, recoded by the arglist converters. This is what a processing command is handed.
• sources — the same keys, each naming the tier its value came from.
• stated — what the tiers actually wrote, before the converters ran. This is what a command called with parameters is filled from.

The distinction is not cosmetic. A header's overwrite: yes is the bool True in options and the string "yes" in stated. A processing command documents overwrite as a bool; a utility command's signature says overwrite="no" and its body does overwrite.lower() == "yes". Filling the bool into that signature is an AttributeError.

# what the tiers stated that this command declares, and what it does not
filled, dropped = gcs.select_parameters(stated, sources, qx_command)

Only stated values come back. A parameter nobody named is left out, so the command keeps its own default for it.

Layer the command line back on top. This is what "never override" means when you write it:

# what the user typed reaches the command exactly as the user typed it
filled = {key: value for key, value in filled.items() if key not in args}
bargs = {**bargs, **filled}

A command's interface is its signature and its documentation.

def declared_parameters(qx_command):
    return {arg.name for arg in qx_command.args} | \
           {option.name for option in qx_command.options}

For a Python command the signature is the whole story. For a MATLAB or Bash command there is no Python signature at all — its parameters are documented ones. Ask gcs.declared_parameters, not qx_command.has_arg, whenever the answer has to hold for every class.

Whether a dropped parameter is a mistake depends on the tier. A batch file's header and a recipe's global parameters state values for every command of a run, so a value of theirs this command cannot take was meant for another one and is dropped silently. A parameter written against the command itself is a mistake, and dropped names it so you can say so.

A run can refuse a batch tier outright. unset_batch_header_parameters and unset_batch_session_parameters are run level parameters — in gcs.extra_parameters, so they never reach a command — that name what must not be taken from the header and from a session's own entry. Each accepts a name, a comma separated list, an array, a glob (hcp_*), * or all; matching is gcs.is_unset, fnmatchcase throughout.

They exist because the header now reaches every command class, so a name two commands share is a value they share: targetfile is declared by four commands that write four different files.

Two implementation points worth knowing before you touch this:

• The header filter runs twice in gmri, around check_deprecated_parameters. The names a header states and the names it contributes are not the same set — _log: study contributes log and comlog_folders, the second under a name nobody wrote — so filtering on both sides is what makes either spelling work. The remap is idempotent, so merge_options remapping again is a no-op.
• The per-session filter lives in gcs.update_options, which gp.run alone calls, so only processing commands ever see that tier and only they are affected by unsetting it.

A recipe can say what a step is not to inherit, in either of two spellings — unset_parameters, taking a name, a comma separated list or an array, or the name written with a leading - and no value. Both are read and removed by recipe._unset_names before select_parameters runs, at step and at recipe level; in global_parameters they raise. Neither can be an absent value: '' is what img_suffix and its kind default to and None is a value export_hcp documents, so a recipe has to be able to state both.

And one run-wide value is withheld without being asked, because a recipe can create the file its own parameter names. recipe.PRODUCERS maps a command to the parameter naming what it writes and the run-wide parameter that names the same file — create_batch: (targetfile, batchfile), one row today. Before the first step runs, _withheld_by_production decides from two facts nothing downstream can know:

Deciding from the state at the start rather than per step is what makes a recipe do the same thing twice: a batch.txt left by an earlier run is otherwise indistinguishable from the one this run just wrote. A batchfile written against a step, or given on the command line, is passed through untouched and still raises when the file is missing.

Every run prints this before the command runs, and writes the same text to the runlog and the comlog:

---> Parameters for hcp_pre_freesurfer

     parameter        source         value
     ---------------------------------------------------------------
     batchfile        command line   /study/processing/batch.txt
     hcp_brainsize    batch file     170
     hcp_t2           batch file     NONE
     overwrite        default        False

banner = gcs.report_parameters(qx_command, options, sources)

It reports the parameters the command declares. A command that declares none this run has a value for reports what was specified instead. Pass session= to report the per-session tier, and only the keys that tier states are shown.

1. One parser, one entry point. Read sessions with gc.resolve_sessions(). Do not open a batch file yourself, in any language — that is how four parsers and two filter dialects accumulated.

2. batch_io imports nothing but the standard library. It is spliced into bin/qunex_container, which runs on a login node with no QuNex on the path. python/tests/test_batch_io.py enforces it.

3. Rebuild the container after touching batch_io.py.

qunex build_qx_container    # splices batch_io.py into bin/qunex_container

Then commit the result, exactly as with qx_commands.yaml. python/tests/test_container_drift.py fails if the copy drifts.

4. Normalise a legacy spelling once, at the front door. gcs.normalize_session_parameters is that door. Adding a per-command exemption puts you back where five conversions along one call chain did.

5. The merge comes before the logging. A batch file can name a different study, and resolve_logging reads that study's settings. If you add a dispatch path, keep resolve_sessions → merge_options → deduce_folders above resolve_logging.

6. Fill, never override. A tier fills a parameter in. What is stated above it always wins, and only stated values are filled at all.

7. Ask the registry what a command takes. gcs.declared_parameters for the interface, gcs.extra_parameters for the run-level parameters that steer QuNex rather than the command.

8. Declare batchfile if the command should see the study's parameters. Either as a signature argument or as an option in the .. qx_command: block.

• Filling options into a signature. The converters recoded it. Use stated for anything called with parameters, options for a processing command that is handed the dictionary.
• Asking qx_command.has_arg for a MATLAB or Bash command. It answers for the Python signature, which those commands do not have, so it is False for every parameter they take. This is how bash scripts came to be called with no arguments at all.
• Treating --batchfile as "run over these sessions". It is where the header comes from as well. Ask whether the dispatcher can hand the command one session — whether it takes a sourcefolder or folder — not whether a batch file was given.
• Deducing the study folder before the header is read. The header can move it, and then the runlog lands in the wrong study for every run that names one.
• A second reader of the batch file. Including in shell and in the container. The container gets a spliced copy of the one parser; nothing else needs one.
• Matching with a regular expression. Selection is glob matching everywhere. An unanchored regex made --sessions=S10 select S1, S100 and xS10y.
• Reporting a value the command does not receive. report_parameters is given {**options, **stated} for the classes called with parameters, so the banner and the call echo below it agree on how a value is spelled.

• gc.resolve_sessions() — sessions and header in one call, with the QuNex error types and the SLURM job-array split already handled.
• SessionList — filter_by_key, filter_by_string, group_by_key, have_key, dont_have_key, get_list_by_key. They return new SessionLists, so they chain. See Developer utilities and classes.
• gcs.select_parameters(stated, sources, qx_command) — narrowing plus the list of what was left out, so you can warn about it.
• gcs.report_parameters(...) — the table, as a string. Print it and write it to the run context; do not build a second one.
• gcs.update_options(session, options, sources) — the _key: / --key: per-session overrides, applied and recorded. The only implementation of that mapping.
• gcs.declared_parameters(qx_command) — signature plus documented options, for every language.
• qunex list_sessions --batchfile=… --filter=… — prints the sessions a run would act on. The quickest way to check a selection before committing a job to it.
• python/tests/test_batch_io.py — the parser, the selection and the filter, and the assertion that the module stays a standard-library leaf.
• python/tests/test_gmri_dispatch.py — what each command class receives, and where. The place to add a case when you change the dispatch.