Includes and Defaults - ptweezy/cronstable GitHub Wiki
This page documents how cronstable loads configuration: the -c argument as a
single file or a directory, the defaults section and its merge precedence, the
include directive, and the special list-merge rules that govern how job
settings inherit from defaults. cronstable/config.py implements all of this
behavior.
The -c/--config argument defaults to /etc/cronstable.d on POSIX. On
Windows it defaults to %ProgramData%\cronstable when that directory holds
configuration, otherwise %APPDATA%\cronstable, falling back to ~ if
APPDATA is unset. See the CLI reference and
running on Windows.
parse_config(config_arg) resolves the argument:
- If
config_argis a directory,_parse_config_dirloads it (directory mode, next section). - Otherwise
parse_config_fileloads it as a single file.parse_configwraps that call in atry/except OSError, so a missing or unreadable file (OSError) is re-raised as aConfigErrorand the CLI reports it cleanly rather than surfacing a bareOSError.
A single file is read as UTF-8, validated against CONFIG_SCHEMA with
strictyaml, and parsed by parse_config_string. The top level accepts an empty
document (EmptyDict()) or a mapping with the optional keys defaults, jobs,
dags, pools, include, web, mcp, cluster, state, logging,
notify, and push (all Opt(...), none is required). See the
configuration reference for the per-job and web
schemas, and logging configuration for the logging
schema.
When -c points at a directory, _parse_config_dir enumerates the directory's
entries with os.scandir and processes them in sorted filename order. Sorting
makes job ordering and "first config found" error messages deterministic rather
than dependent on filesystem order.
For each entry, _parse_config_dir splits the name into base and extension:
- The entry is skipped if the first character of its base name is
_or.. This lets you keep shared include fragments (such as_inc.yaml) and dotted files in the same directory without them being loaded as standalone configs. - The entry is skipped unless its extension is
.ymlor.yaml, or its name marks it as a classic crontab (*.crontab,*.cron, or a file namedcrontab; see classic crontabs).
parse_config_file parses each remaining file independently, then its results
are aggregated across the directory:
| Aggregate | Behavior across files |
|---|---|
jobs |
Concatenated in sorted-filename order (jobs.extend(...)). |
dags |
Concatenated in sorted-filename order. |
pools |
Merged across files. A pool name defined in two files raises ConfigError("duplicate pool '<name>'"). |
defaults |
Merged across files with mergedicts into a single directory-wide job_defaults (used only as the returned job_defaults; see the following caveat). |
web |
At most one. A second file containing a web block raises ConfigError("Multiple 'web' configurations found: first in <file>, now in <file>"). |
logging |
At most one. A second file containing a logging block raises ConfigError("Multiple 'logging' configurations found: ..."). |
cluster, state, mcp, notify, push
|
At most one each. A second file containing the same block raises ConfigError("Multiple '<kind>' configurations found: first in <file>, now in <file>"). |
Per-file parse errors are collected, keyed by path. If any occurred, they are
raised together as a single ConfigError whose message joins the individual
errors with \n---. An empty directory, or one where every entry is skipped,
yields an empty CronstableConfig (empty jobs, no web, empty
job_defaults, no logging) rather than an error.
This is the most important caveat of directory mode: the defaults section in
each file applies only to the jobs defined in that same file. Each file is
parsed by its own parse_config_string call, which merges that file's
defaults into its own jobs before the files are aggregated.
The directory-wide job_defaults returned by _parse_config_dir is the merge of
every file's defaults, but it is not applied retroactively to jobs that
were already constructed in other files. To share defaults across files in a
directory, put them in an include fragment and include it from each file (see
the include directive later).
A defaults mapping supplies inherited values for the jobs defined alongside
it. It accepts the same options as a job (_job_defaults_common) except for the
job-only required keys name, command, and schedule. Every option in
defaults is optional. Jobs may override any inherited value.
defaults:
shell: /bin/bash
utc: false
environment:
- key: PATH
value: /bin:/usr/bin
jobs:
- name: test-01
command: echo "foobar" # inherits /bin/bash, utc:false, PATH
schedule: "*/5 * * * *"
- name: test-02
command: echo "zbr"
shell: /bin/sh # overrides the inherited shell
schedule: "*/5 * * * *"The shell field works on every OS; the preceding /bin/... paths are POSIX
examples. On Windows, set shell to a shell such as powershell, leave it
empty to use cmd.exe (through %ComSpec%), or pass command as a list to
bypass the shell entirely (see running on Windows).
A DAG task is a job invocation, so its launch fields
inherit the defaults: block the same way a job's do. A global shell,
environment, env_file, workingDirectory, priority,
captureStdout/captureStderr, monitorResources, resourceLimits,
run-scoped secrets, or reporter (onFailure/onSuccess/…) block covers DAG
tasks too, with the task's own value winning on any key it sets. Approval gates
ignore an inherited resourceLimits, because they run no process.
The DAG-node fields that shape the graph (dependsOn, triggerRule, retries,
expand, onReject, the poke settings) are graph structure, not launch config,
and are never touched by defaults:. The DAG's synthetic schedule-trigger job
runs a placeholder true on every tick and deliberately stays on the built-in
defaults, so a global reporter fires per DAG run, not on every tick.
As with jobs, in directory mode and across
includes, a defaults: block reaches only the jobs
and DAGs defined in the same file.
Within a single parsed file, parse_config_string builds each job's effective
configuration by successively merging with mergedicts, in this order (later
wins):
-
DEFAULT_CONFIG: the built-in defaults, such asshell: /bin/shon POSIX,captureStderr: true,utc: true,killTimeout: 30. On Windows,shellis empty and a stringcommandroutes through%ComSpec%/cmd.exe; see running on Windows. The configuration reference has the full list. -
Included files' defaults: the
defaultsblocks of any files named by this file'sincludedirective, merged together in include order. - This file's
defaultsblock. - Per-job overrides: the keys set on the individual job.
Each job dict is mergedicts(defaults, config_job), where defaults is
mergedicts(mergedicts(DEFAULT_CONFIG, included_defaults), this_files_defaults).
JobConfig then validates the resulting per-job dict (numeric ranges,
timezone, user/group).
On Windows, user/group switching is unsupported: a job with user or group
set raises a configuration error (Job <name>: changing user/group is not supported on Windows); see running on Windows. The merged
defaults (steps 1–3) is also returned as the file's job_defaults.
mergedicts(dict1, dict2) is a recursive deep merge where, on key collision,
dict2 takes precedence. Its rules:
| Case | Behavior |
|---|---|
| Both values are mappings | Recurse (deep merge). |
dict1 value is a mapping, dict2 value is None
|
Keep dict1's mapping (treat None as "not overridden"). |
Both values are lists, key is environment
|
Merge by key (described later). |
Both values are lists, key is secrets
|
Merge by name (described later). |
Both values are lists, key is fingerprint
|
Replace with dict2's list (no concatenation). |
| Both values are lists, any other key |
Concatenate (v1 + v2). |
| Otherwise (scalars, type mismatch) | Take dict2's value. |
Key only in dict1
|
Keep dict1's value. |
Key only in dict2
|
Keep dict2's value. |
A job's resourceLimits map merges into the inherited one key by key, as the
first row says. A bare resourceLimits: is a schema error, so it cannot clear
an inherited map; to lift an inherited cap, set it to max (see
resource limits).
environment is a list of {key, value} mappings. When a job and a default
both define environment, the two lists merge into a dictionary keyed by key
(default entries first, then the job's), so a job's variable overrides the
default with the same name instead of producing two list entries with the same
key. See
commands and environment for environment and
env_file.
defaults:
environment:
- key: FOO
value: foo
- key: BAR
value: bar
jobs:
- name: test-01
command: env
schedule: "* * * * *"
environment:
- key: FOO # overrides the default FOO -> single FOO=xpto entry
value: xpto
- key: ZBR # added alongside the inherited BAR
value: blahThe preceding job runs with FOO=xpto, BAR=bar, ZBR=blah.
Run-scoped secrets are a list of {name, ...} mappings and merge the same way
environment does. Entries are keyed by name (default entries first, then
the job's), so a job's secret overrides a same-named default instead of
staging two secrets under one name. The job's entry wins wholesale, including
its value/fromFile/fromEnvVar source. See
durable state for what a secret block does
and how a job reads one.
The Sentry fingerprint (a list of strings, default
["cronstable", "{{ environment.HOSTNAME }}", "{{ name }}"]) is a
replace-not-append setting: a job or defaults block that supplies its own
fingerprint overrides the default list entirely. Plain list concatenation
would silently prepend the three default entries, making custom Sentry issue
grouping impossible. See reporting for the Sentry reporter.
environment, secrets, and fingerprint are the only exceptions; all
other list-valued options concatenate.
include is an optional list of file paths. Each path is resolved relative to
the directory of the including file
(os.path.join(os.path.dirname(path), include)).
An include path is an ordinary string value, so it may itself use a ${VAR}
reference, such as - ${ENVIRONMENT:-prod}.yaml. Each file is also expanded
against the environment as it is parsed, so an included file resolves its own
${VAR} references (see
environment-variable interpolation).
Classic crontab files are not interpolated: ${...} in a crontab reaches the
job verbatim.
An included file may be YAML or a classic crontab (same
recognition rules as -c). Crontab entries always carry the built-in
defaults, so, like any included file's jobs, they do not pick up the including
file's defaults.
Included files are parsed recursively with parse_config_file, and their
results are merged into the including file as follows:
-
Jobs from included files are appended to this file's job list. Crucially,
they arrive already fully constructed by the included file's own
parse_config_string, so they carry only their own file's defaults (andDEFAULT_CONFIG). Adefaultsblock in the including file does not retro-apply to jobs from an included file. -
Defaults from included files are merged together (in include order) into
inc_defaults_merged, which is folded into step 2 of the earlier merge precedence. That is, included defaults affect only this file's inline jobs, not the included files' own jobs. -
webfrom an included file is adopted if this file has none. If both defineweb, parsing raisesConfigError("multiple web configs"). -
loggingfrom an included file is adopted if this file has none. If both definelogging, parsing raisesConfigError("multiple logging configs"). -
cluster,state,mcp,notify, andpushare adopted the same way; a block defined in both files raisesConfigError("multiple <kind> configs"). - DAGs from included files are appended to this file's DAG list.
-
Pools from included files are merged with this file's pools. A pool name
defined in both raises
ConfigError("duplicate pool '<name>'").
A file that two files of one include tree both include is loaded under each
of them. Each includer takes the shared file's defaults. Everything else
arrives once for each includer, so only a file that holds defaults alone
can be shared. A shared file that defines a job fails the load with
duplicate job name(s): <name>; the scheduler tracks jobs by name, so all but the last definition of a name would silently never run. Rename the duplicates. A shared pools block fails with duplicate pool '<name>',
and a shared web, logging, cluster, state, mcp, notify, or
push section fails with multiple <kind> configs.
The intended use is to put common definitions (reporting defaults, shell, and
environment) in a fragment named so directory mode skips it, for example with
a leading underscore, and include it from each real config file. For example:
_inc.yaml (skipped by directory mode; provides shared defaults):
defaults:
shell: /bin/bash
concurrencyPolicy: Allow
environment:
- key: FOO
value: foo
- key: BAR
value: bartest.yaml (a real config file that includes the fragment):
include:
- _inc.yaml
jobs:
- name: test-01
command: echo "foobar" # inherits shell:/bin/bash and FOO/BAR from _inc.yaml
schedule: "@reboot"
captureStdout: trueBecause _inc.yaml contains only defaults (no jobs), its defaults flow into
test.yaml's inline jobs through the include-defaults merge. If _inc.yaml
also defined jobs, those jobs would carry _inc.yaml's own defaults and would
not pick up test.yaml's defaults.
parse_config_file receives in _seen the absolute paths of the files whose
parse is in progress, from the top-level file down to the file that includes
this one, and raises when its own path is in that set. A file that includes
itself, directly or through other files, therefore raises a clear
ConfigError("include cycle detected at <path>") instead of recursing until a
RecursionError. <path> is the include path as joined to the including
file's directory.
Each file passes its includes a copy of the set with its own path added, so a
file under one include is never in the set that another include receives. A
file that two files of one include tree both include is therefore not
flagged as a cycle. Loading a main.yaml that includes team_a.yaml and
team_b.yaml, each of which includes common.yaml, succeeds. See
the include directive for what a file that is
included twice contributes.
In directory mode, each file of the directory starts with an empty set.
- Configuration Reference: full option schema and built-in defaults.
-
Commands and Environment:
environmentandenv_file. -
Environment-Variable Interpolation:
${VAR}expansion in config string values, applied per file at parse time. -
Durable State: run-scoped
secrets. -
Reporting: Sentry
fingerprintand the report blocks. -
Logging Configuration: the
loggingsection. -
CLI Reference: the
-cargument and--validate-config.