Environment Variable Interpolation - ptweezy/cronstable GitHub Wiki

Environment-variable interpolation

cronstable expands ${VAR} references in a configuration's string values against its own process environment when it loads the file. You can then deploy one config unchanged to several environments, with the per-environment pieces (a listen address, a state path, a time zone, a webhook URL) supplied as environment variables rather than by a wrapper script that templates them in. The expansion is implemented in cronstable/config.py and runs on every config load, including a hot reload.

On this page: Syntax · What gets expanded · Skipped fields · Unset variables · Multi-file config · Effect on the job-set ID · Worked example

Syntax

Form Meaning
${VAR} The value of environment variable VAR. A set but empty variable expands to the empty string. An unset VAR is a load-time error (see Unset variables).
${VAR:-default} The value of VAR when it is set and nonempty, otherwise the literal default. As in the shell's :-, an unset or empty VAR yields default. The default runs to the closing } and may contain any character except }.
$$ A literal $. Use it to write a value that should contain a real dollar sign, or to keep a literal ${...} ($${VAR} stays ${VAR}).

A variable name is a letter or underscore followed by letters, digits, or underscores. Only these braced forms are recognized. A lone $, a bare $VAR without braces, and a malformed ${...} (unclosed, or a name with unsupported characters) are all left exactly as written, so a value that never used the syntax is passed through verbatim.

web:
  listen:
    - "http://0.0.0.0:${PORT}"            # PORT from the environment
    - "http://${BIND:-127.0.0.1}:8080"    # a default when BIND is unset
state:
  path: ${STATE_DIR}/cronstable    # a whole path segment from the environment
jobs:
  - name: report-${REGION}         # names, timezones, prefixes, ... all expand
    command: run-report
    schedule:
      minute: "0"
    timezone: ${TZ:-UTC}
    streamPrefix: "cost was $$5"    # a literal dollar sign

What gets expanded

Expansion runs after the document is validated against the schema, over the parsed values. It therefore applies to every field the schema accepts as a string, in every section except logging (see the next section). A ${VAR} may sit anywhere inside such a string.

Because it runs post-validation, a numeric key cannot itself be a bare ${VAR}. smtpPort: ${PORT} fails schema validation before expansion is ever reached, because ${PORT} is not an integer. Put the variable inside a string instead. A listen address carries its port inside a string ("http://0.0.0.0:${PORT}"), which is why the port can come from the environment.

Path-typed fields are strings, so they expand like any other. The one to watch is a job's workingDirectory, because the command beside it behaves the opposite way. The directory is expanded at load against the daemon's environment, then ~-expanded and made absolute. The command is handed to the runtime shell verbatim. workingDirectory: ${JOB_ROOT}/importer is settled once, at load; command: cd ${JOB_ROOT} is never touched.

Skipped fields

Some subtrees are left untouched, because their ${...} is another layer's expansion syntax rather than cronstable's.

Commands and shells reach the process unchanged:

  • a job's command and shell,
  • a DAG task's command and shell,
  • a verify block's command,
  • the shell in a defaults: block,
  • a shell reporter's whole report.shell block (its command and shell).

These are handed to a shell at run time. That shell is meant to expand their ${VAR} against the job's environment (its environment / env_file and staged secrets), which is assembled per run and is not the daemon's environment. So command: echo ${HOME} prints the home directory of the user the job runs as, exactly as under /bin/sh, and is never touched at load. To use a load-time environment variable inside a command, set it as a job environment value (which does expand) and reference that from the command.

The whole logging section is skipped, because it is passed to Python's logging.config verbatim, and a $-style formatter (style: "$") legitimately writes ${asctime} / ${message} in its format string. Interpolating those placeholders would treat them as environment variables and stop an otherwise valid config from loading, so cronstable leaves the section for logging.config. For a log path that needs an environment variable, supply the path through the process environment that logging.config itself reads, or template the file outside cronstable.

Unset variables

A ${VAR} with no :-default whose variable is unset is a hard ConfigError that names the variable, the config value it appeared in, and the file:

prod.yaml: config value web.listen[0] references environment variable
${NEEDED_PORT}, which is not set; export it, or write ${NEEDED_PORT:-default}
to supply a fallback

The failure happens at load, so cronstable --validate-config catches a missing variable in continuous integration or a deploy check and exits nonzero before the scheduler ever starts. Supply a :-default for anything that has a sensible fallback, and leave it off for the variables a deployment must provide.

Multi-file config

Each file is expanded against the environment as it is parsed, so an included file resolves its own ${VAR} references. Because an include: entry is itself an ordinary string value, you can also build an include path from the environment:

include:
  - ${ENVIRONMENT:-prod}.yaml

Classic crontab files are not interpolated: ${...} in a crontab reaches the job verbatim.

The daemon reads its own process environment, which is fixed when it starts. Restart it to pick up a changed variable.

Effect on the job-set ID

The job-set ID is a fingerprint of the effective config, and expansion happens before that fingerprint is taken, so it hashes the expanded values. A config that interpolates environment variables into fingerprinted fields therefore produces a different id per environment. Two deployments that resolve ${REGION} to different values are, by design, running different job sets. If you need one id to compare across environments, keep environment variables out of fingerprinted job fields. Otherwise, use them freely and expect the id to track the environment.

Worked example

A single platform.yaml deployed to every region, with the per-region values injected as environment variables at the container or unit level:

state:
  path: ${STATE_DIR}
web:
  listen:
    - "http://0.0.0.0:${WEB_PORT:-8080}"
jobs:
  - name: nightly-rollup
    command: rollup --region "$REGION"   # $REGION expanded by the job's shell
    schedule:
      minute: "0"
      hour: "2"
    timezone: ${TZ:-UTC}
    onFailure:
      report:
        webhook:
          url:
            value: ${ALERT_WEBHOOK}

STATE_DIR and ALERT_WEBHOOK are required, and an unset one fails --validate-config. WEB_PORT and TZ fall back to their defaults. $REGION inside command is left for the job's shell to expand at run time.

See also