Includes and Defaults - ptweezy/cronstable GitHub Wiki

Includes, defaults, and multi-file config

This page documents how cronstable loads configuration: the -c argument as a single file or a directory, the defaults section and its merge precedence, the include directive, and the special list-merge rules that govern how job settings inherit from defaults. cronstable/config.py implements all of this behavior.

Config loading entry points

The -c/--config argument defaults to /etc/cronstable.d on POSIX. On Windows it defaults to %ProgramData%\cronstable when that directory holds configuration, otherwise %APPDATA%\cronstable, falling back to ~ if APPDATA is unset. See the CLI reference and running on Windows.

parse_config(config_arg) resolves the argument:

  • If config_arg is a directory, _parse_config_dir loads it (directory mode, next section).
  • Otherwise parse_config_file loads it as a single file. parse_config wraps that call in a try/except OSError, so a missing or unreadable file (OSError) is re-raised as a ConfigError and the CLI reports it cleanly rather than surfacing a bare OSError.

A single file is read as UTF-8, validated against CONFIG_SCHEMA with strictyaml, and parsed by parse_config_string. The top level accepts an empty document (EmptyDict()) or a mapping with the optional keys defaults, jobs, dags, pools, include, web, mcp, cluster, state, logging, notify, and push (all Opt(...), none is required). See the configuration reference for the per-job and web schemas, and logging configuration for the logging schema.

Directory mode

When -c points at a directory, _parse_config_dir enumerates the directory's entries with os.scandir and processes them in sorted filename order. Sorting makes job ordering and "first config found" error messages deterministic rather than dependent on filesystem order.

For each entry, _parse_config_dir splits the name into base and extension:

  • The entry is skipped if the first character of its base name is _ or .. This lets you keep shared include fragments (such as _inc.yaml) and dotted files in the same directory without them being loaded as standalone configs.
  • The entry is skipped unless its extension is .yml or .yaml, or its name marks it as a classic crontab (*.crontab, *.cron, or a file named crontab; see classic crontabs).

parse_config_file parses each remaining file independently, then its results are aggregated across the directory:

Aggregate Behavior across files
jobs Concatenated in sorted-filename order (jobs.extend(...)).
dags Concatenated in sorted-filename order.
pools Merged across files. A pool name defined in two files raises ConfigError("duplicate pool '<name>'").
defaults Merged across files with mergedicts into a single directory-wide job_defaults (used only as the returned job_defaults; see the following caveat).
web At most one. A second file containing a web block raises ConfigError("Multiple 'web' configurations found: first in <file>, now in <file>").
logging At most one. A second file containing a logging block raises ConfigError("Multiple 'logging' configurations found: ...").
cluster, state, mcp, notify, push At most one each. A second file containing the same block raises ConfigError("Multiple '<kind>' configurations found: first in <file>, now in <file>").

Per-file parse errors are collected, keyed by path. If any occurred, they are raised together as a single ConfigError whose message joins the individual errors with \n---. An empty directory, or one where every entry is skipped, yields an empty CronstableConfig (empty jobs, no web, empty job_defaults, no logging) rather than an error.

Defaults are scoped per YAML file in directory mode

This is the most important caveat of directory mode: the defaults section in each file applies only to the jobs defined in that same file. Each file is parsed by its own parse_config_string call, which merges that file's defaults into its own jobs before the files are aggregated.

The directory-wide job_defaults returned by _parse_config_dir is the merge of every file's defaults, but it is not applied retroactively to jobs that were already constructed in other files. To share defaults across files in a directory, put them in an include fragment and include it from each file (see the include directive later).

The defaults section

A defaults mapping supplies inherited values for the jobs defined alongside it. It accepts the same options as a job (_job_defaults_common) except for the job-only required keys name, command, and schedule. Every option in defaults is optional. Jobs may override any inherited value.

defaults:
  shell: /bin/bash
  utc: false
  environment:
    - key: PATH
      value: /bin:/usr/bin
jobs:
  - name: test-01
    command: echo "foobar"   # inherits /bin/bash, utc:false, PATH
    schedule: "*/5 * * * *"
  - name: test-02
    command: echo "zbr"
    shell: /bin/sh           # overrides the inherited shell
    schedule: "*/5 * * * *"

The shell field works on every OS; the preceding /bin/... paths are POSIX examples. On Windows, set shell to a shell such as powershell, leave it empty to use cmd.exe (through %ComSpec%), or pass command as a list to bypass the shell entirely (see running on Windows).

Defaults also cover DAG tasks

A DAG task is a job invocation, so its launch fields inherit the defaults: block the same way a job's do. A global shell, environment, env_file, workingDirectory, priority, captureStdout/captureStderr, monitorResources, resourceLimits, run-scoped secrets, or reporter (onFailure/onSuccess/…) block covers DAG tasks too, with the task's own value winning on any key it sets. Approval gates ignore an inherited resourceLimits, because they run no process.

The DAG-node fields that shape the graph (dependsOn, triggerRule, retries, expand, onReject, the poke settings) are graph structure, not launch config, and are never touched by defaults:. The DAG's synthetic schedule-trigger job runs a placeholder true on every tick and deliberately stays on the built-in defaults, so a global reporter fires per DAG run, not on every tick.

As with jobs, in directory mode and across includes, a defaults: block reaches only the jobs and DAGs defined in the same file.

Merge precedence

Within a single parsed file, parse_config_string builds each job's effective configuration by successively merging with mergedicts, in this order (later wins):

  1. DEFAULT_CONFIG: the built-in defaults, such as shell: /bin/sh on POSIX, captureStderr: true, utc: true, killTimeout: 30. On Windows, shell is empty and a string command routes through %ComSpec%/cmd.exe; see running on Windows. The configuration reference has the full list.
  2. Included files' defaults: the defaults blocks of any files named by this file's include directive, merged together in include order.
  3. This file's defaults block.
  4. Per-job overrides: the keys set on the individual job.

Each job dict is mergedicts(defaults, config_job), where defaults is mergedicts(mergedicts(DEFAULT_CONFIG, included_defaults), this_files_defaults). JobConfig then validates the resulting per-job dict (numeric ranges, timezone, user/group).

On Windows, user/group switching is unsupported: a job with user or group set raises a configuration error (Job <name>: changing user/group is not supported on Windows); see running on Windows. The merged defaults (steps 1–3) is also returned as the file's job_defaults.

Merge semantics (mergedicts)

mergedicts(dict1, dict2) is a recursive deep merge where, on key collision, dict2 takes precedence. Its rules:

Case Behavior
Both values are mappings Recurse (deep merge).
dict1 value is a mapping, dict2 value is None Keep dict1's mapping (treat None as "not overridden").
Both values are lists, key is environment Merge by key (described later).
Both values are lists, key is secrets Merge by name (described later).
Both values are lists, key is fingerprint Replace with dict2's list (no concatenation).
Both values are lists, any other key Concatenate (v1 + v2).
Otherwise (scalars, type mismatch) Take dict2's value.
Key only in dict1 Keep dict1's value.
Key only in dict2 Keep dict2's value.

A job's resourceLimits map merges into the inherited one key by key, as the first row says. A bare resourceLimits: is a schema error, so it cannot clear an inherited map; to lift an inherited cap, set it to max (see resource limits).

environment merges by key

environment is a list of {key, value} mappings. When a job and a default both define environment, the two lists merge into a dictionary keyed by key (default entries first, then the job's), so a job's variable overrides the default with the same name instead of producing two list entries with the same key. See commands and environment for environment and env_file.

defaults:
  environment:
    - key: FOO
      value: foo
    - key: BAR
      value: bar
jobs:
  - name: test-01
    command: env
    schedule: "* * * * *"
    environment:
      - key: FOO        # overrides the default FOO -> single FOO=xpto entry
        value: xpto
      - key: ZBR        # added alongside the inherited BAR
        value: blah

The preceding job runs with FOO=xpto, BAR=bar, ZBR=blah.

secrets merges by name

Run-scoped secrets are a list of {name, ...} mappings and merge the same way environment does. Entries are keyed by name (default entries first, then the job's), so a job's secret overrides a same-named default instead of staging two secrets under one name. The job's entry wins wholesale, including its value/fromFile/fromEnvVar source. See durable state for what a secret block does and how a job reads one.

fingerprint replaces, does not append

The Sentry fingerprint (a list of strings, default ["cronstable", "{{ environment.HOSTNAME }}", "{{ name }}"]) is a replace-not-append setting: a job or defaults block that supplies its own fingerprint overrides the default list entirely. Plain list concatenation would silently prepend the three default entries, making custom Sentry issue grouping impossible. See reporting for the Sentry reporter.

environment, secrets, and fingerprint are the only exceptions; all other list-valued options concatenate.

The include directive

include is an optional list of file paths. Each path is resolved relative to the directory of the including file (os.path.join(os.path.dirname(path), include)).

An include path is an ordinary string value, so it may itself use a ${VAR} reference, such as - ${ENVIRONMENT:-prod}.yaml. Each file is also expanded against the environment as it is parsed, so an included file resolves its own ${VAR} references (see environment-variable interpolation). Classic crontab files are not interpolated: ${...} in a crontab reaches the job verbatim.

An included file may be YAML or a classic crontab (same recognition rules as -c). Crontab entries always carry the built-in defaults, so, like any included file's jobs, they do not pick up the including file's defaults.

Included files are parsed recursively with parse_config_file, and their results are merged into the including file as follows:

  • Jobs from included files are appended to this file's job list. Crucially, they arrive already fully constructed by the included file's own parse_config_string, so they carry only their own file's defaults (and DEFAULT_CONFIG). A defaults block in the including file does not retro-apply to jobs from an included file.
  • Defaults from included files are merged together (in include order) into inc_defaults_merged, which is folded into step 2 of the earlier merge precedence. That is, included defaults affect only this file's inline jobs, not the included files' own jobs.
  • web from an included file is adopted if this file has none. If both define web, parsing raises ConfigError("multiple web configs").
  • logging from an included file is adopted if this file has none. If both define logging, parsing raises ConfigError("multiple logging configs").
  • cluster, state, mcp, notify, and push are adopted the same way; a block defined in both files raises ConfigError("multiple <kind> configs").
  • DAGs from included files are appended to this file's DAG list.
  • Pools from included files are merged with this file's pools. A pool name defined in both raises ConfigError("duplicate pool '<name>'").

A file that two files of one include tree both include is loaded under each of them. Each includer takes the shared file's defaults. Everything else arrives once for each includer, so only a file that holds defaults alone can be shared. A shared file that defines a job fails the load with duplicate job name(s): <name>; the scheduler tracks jobs by name, so all but the last definition of a name would silently never run. Rename the duplicates. A shared pools block fails with duplicate pool '<name>', and a shared web, logging, cluster, state, mcp, notify, or push section fails with multiple <kind> configs.

The intended use is to put common definitions (reporting defaults, shell, and environment) in a fragment named so directory mode skips it, for example with a leading underscore, and include it from each real config file. For example:

_inc.yaml (skipped by directory mode; provides shared defaults):

defaults:
  shell: /bin/bash
  concurrencyPolicy: Allow
  environment:
    - key: FOO
      value: foo
    - key: BAR
      value: bar

test.yaml (a real config file that includes the fragment):

include:
  - _inc.yaml
jobs:
  - name: test-01
    command: echo "foobar"   # inherits shell:/bin/bash and FOO/BAR from _inc.yaml
    schedule: "@reboot"
    captureStdout: true

Because _inc.yaml contains only defaults (no jobs), its defaults flow into test.yaml's inline jobs through the include-defaults merge. If _inc.yaml also defined jobs, those jobs would carry _inc.yaml's own defaults and would not pick up test.yaml's defaults.

Include cycle detection

parse_config_file receives in _seen the absolute paths of the files whose parse is in progress, from the top-level file down to the file that includes this one, and raises when its own path is in that set. A file that includes itself, directly or through other files, therefore raises a clear ConfigError("include cycle detected at <path>") instead of recursing until a RecursionError. <path> is the include path as joined to the including file's directory.

Each file passes its includes a copy of the set with its own path added, so a file under one include is never in the set that another include receives. A file that two files of one include tree both include is therefore not flagged as a cycle. Loading a main.yaml that includes team_a.yaml and team_b.yaml, each of which includes common.yaml, succeeds. See the include directive for what a file that is included twice contributes.

In directory mode, each file of the directory starts with an empty set.

Related pages

⚠️ **GitHub.com Fallback** ⚠️