Commands and Environment - ptweezy/cronstable GitHub Wiki

Commands and environment

This page documents how cronstable runs a job's command (through a shell or by direct exec), how it builds the job's environment from environment, env_file, and inherited defaults, and how privilege switching with user/group works. For schedule syntax, see schedules and time zones. For how defaults merge into jobs, see includes, defaults, and multi-file config.

Options

These options are members of each job and may also be set in a defaults block. The types and defaults come from the strictyaml schema and DEFAULT_CONFIG.

Option Type Default Description
command Str or Seq(Str) required The program to run. A string runs through a shell. On Windows, the default empty shell routes it through the native command processor %ComSpec% / cmd.exe. A list runs directly, with no shell, on every platform.
shell Str /bin/sh (POSIX) / empty (Windows) Shell that runs a string command. The empty Windows default routes a string command through the native command processor %ComSpec% / cmd.exe. To use PowerShell or another interpreter, set shell: explicitly, or pass command as a list, which bypasses the shell on every platform. See running on Windows.
environment Seq(Map({key, value})) [] Environment variables added to the subprocess environment. Each item has a key and a value, both Str.
env_file Str None Path to a KEY=VALUE file whose variables are merged into environment.
workingDirectory Str or null None Directory the subprocess starts in. With it unset, the job inherits cronstable's own working directory. See workingDirectory later on this page.
priority Enum(idle, below-normal, normal, above-normal, high) normal Scheduling priority of the subprocess. Descendants inherit a lowered level. A raised level applies to the subprocess itself on Windows. The default level is never applied: the job keeps cronstable's own nice on POSIX, and cronstable's own class on Windows only when cronstable is at idle or below-normal, otherwise NORMAL. See priority later on this page.
resourceLimits Map unset Per-run CPU, memory, and process-count caps on the subprocess and everything it starts, enforced with cgroup v2 on Linux and Job Objects on Windows. See resource limits.
user Str or Int unset User (login name or numeric uid) to run the subprocess as. POSIX only. Setting it on Windows raises a configuration error (see running on Windows).
group Str or Int unset Group (group name or numeric gid) to run the subprocess as. POSIX only. Setting it on Windows raises a configuration error (see running on Windows).

command is required on every job. shell has a platform-specific schema and DEFAULT_CONFIG default: /bin/sh on POSIX and an empty string on Windows (DEFAULT_SHELL in cronstable/platform.py), which makes a string command run through cmd.exe. environment defaults to an empty list.

env_file, user, and group are optional (Opt(...) in the schema) and unset by default. environment and env_file are also inheritable through defaults, but user and group are job-only fields in the schema. They appear in _job_defaults_common, so defaults technically accepts them, but resolution and the root-required check happen per job.

command and shell

command may be either a string or a list of strings, and the form determines how cronstable launches the process (RunningJob.start in cronstable/job.py):

  • String: run through a shell.
    • If shell is set, cronstable launches asyncio.create_subprocess_exec with argv [shell, "-c", command] (PowerShell reads -c as an abbreviation of -Command). With the default shell on POSIX, that is ["/bin/sh", "-c", command].
    • Windows naming cmd/cmd.exe is the exception (shell_spawn in cronstable/job.py). cmd.exe takes /c rather than -c, and it parses its command line by its own rules instead of the CommandLineToArgvW rules that argv rendering is built to be undone by. A command with embedded double quotes would therefore reach it with the escaping still in place. The daemon uses asyncio.create_subprocess_shell for it instead, which hands the command string to %ComSpec% /c untouched. An absolute shell: path is passed as the subprocess executable so it still wins over %ComSpec%.
    • If shell is falsy, cronstable also uses asyncio.create_subprocess_shell with the bare command string. On POSIX, the default /bin/sh makes the exec-with--c path the one that runs. On Windows, the default shell is empty (DEFAULT_SHELL in cronstable/platform.py), so create_subprocess_shell is the default path: the command goes to cmd.exe through %ComSpec%, the same path an explicit shell: cmd takes. See running on Windows.
  • List: run directly with asyncio.create_subprocess_exec, with no shell involved. The argv is taken verbatim from the list, with no word splitting, globbing, quoting, or $VAR expansion.

In all cases the argv goes through platform.encode_argv before the subprocess is created. On POSIX, each element is encoded to UTF-8 bytes, so the child's argv is independent of the locale. On Windows, the strings pass through unchanged (CreateProcessW works from str and rejects bytes).

Whatever the command form, cronstable spawns the subprocess with platform.new_process_group_kwargs() applied. On POSIX, the job starts in a fresh session (start_new_session), placing it and every descendant in their own process group, so a cancellation (an executionTimeout expiry, concurrencyPolicy: Replace, or an API cancel) can take the whole tree down as a unit.

On Windows, the job is created with CREATE_NEW_PROCESS_GROUP, which shields it from the daemon console's Ctrl-C and makes it a trappable CTRL_BREAK_EVENT target ahead of the forced stop of the whole tree. Those same Windows creation flags carry the job's priority. See cancellation and killTimeout.

jobs:
  # string form: run via /bin/bash -c "..."
  - name: via-shell
    command: echo "$HOME" && date
    shell: /bin/bash
    schedule: "*/5 * * * *"

  # list form: executed directly, no shell, no variable expansion
  - name: direct-exec
    command:
      - echo
      - foobar
    schedule: "*/5 * * * *"

Launch failures

If the process cannot be launched at all (for example, a list-form command whose executable does not exist, raising FileNotFoundError, or a SubprocessError/UnicodeEncodeError), cronstable logs the launch error, sets start_failed, and treats the run as a normal job failure with exit code 127 rather than crashing the scheduler. See failure detection and retries.

environment

environment is a list of {key, value} maps. Both key and value are strings in the schema.

When environment is nonempty, when the daemon injects run variables, or when the PyInstaller fixup applies, cronstable builds the subprocess environment from the full current process environment (dict(os.environ)). It applies the PyInstaller fixup, sets or overwrites each configured variable by key, and applies the daemon-injected variables last:

env = dict(os.environ)
fixup_pyinstaller_env(env)
for envvar in config.environment:
    env[envvar["key"]] = envvar["value"]
env.update(self.extra_env)

With a state: section configured, the daemon injects CRONSTABLE_STATE_URL, CRONSTABLE_STATE_TOKEN, CRONSTABLE_RUN_ID, CRONSTABLE_JOB_NAME, CRONSTABLE_ATTEMPT, CRONSTABLE_SCHEDULED_AT, and CRONSTABLE_HOST into every run (state.jobApi.enabled defaults to true). A workflow task receives its CRONSTABLE_DAG_* variables the same way. Because they are applied last, these variables override a job's own values for the same names. See durable state.

If environment is empty (the default), there is no env_file, and none of the other cases applies, no env is passed to the subprocess, so it inherits cronstable's environment unchanged.

jobs:
  - name: with-env
    command: printenv PATH
    schedule: "*/5 * * * *"
    environment:
      - key: PATH
        value: /bin:/usr/bin

HOSTNAME injection

When the cronstable.job module is imported, if HOSTNAME is not already present in os.environ, it is set to socket.gethostname():

if "HOSTNAME" not in os.environ:
    os.environ["HOSTNAME"] = gethostname()

Because this mutates the process environment, HOSTNAME is present in the inherited environment of every job.

The reporting templates' environment variable is the constructed subprocess env dict (self.env), which is only populated when the job has a nonempty environment, an env_file, or daemon-injected run variables. For a job with none of these, environment is None in templates and {{ environment.HOSTNAME }} renders empty. See reporting (mail, Sentry, shell, webhook).

env_file

env_file names a file of KEY=VALUE lines. parse_environment_file in cronstable/config.py parses it:

  • The file is opened as UTF-8; a leading byte-order mark is stripped.
  • Each line is stripped of surrounding spaces and a trailing newline.
  • Lines beginning with # and blank lines are ignored.
  • A line without an = raises ConfigError ("Invalid line in env_file: ...").
  • Each remaining line is split on the first = into key and value, and both are then space-stripped. There is no quote handling, and no # inline-comment handling beyond whole-line comments.
  • parse_environment_file itself raises a bare OSError if the file cannot be opened. The caller _merge_env_file wraps that as ConfigError("Could not load env_file: ...").

environment overrides env_file

When env_file is set, cronstable merges it at config-parse time (_merge_env_file). It parses the file into a dict, then applies the job's own environment entries on top, so environment overrides env_file per key.

The merged result replaces self.environment as a list of {key, value} items, which is then applied to the subprocess as described in environment earlier. When env_file is set, an env is therefore always passed to the subprocess, even if environment was originally empty.

jobs:
  - name: with-env-file
    command: printenv
    schedule: "*/5 * * * *"
    env_file: /etc/cronstable/job.env
    environment:
      - key: LOG_LEVEL
        value: debug   # overrides LOG_LEVEL from job.env, if present

Example env_file contents:

# comment lines and blank lines are ignored

PATH=/bin:/usr/bin
LOG_LEVEL=info

defaults.environment merge semantics

environment set in a defaults block merges into each job by key, not by list concatenation. mergedicts (cronstable/config.py) special-cases the environment list: it folds the default's entries and the job's entries into a single key-to-value mapping, with the job's value winning on conflict, then re-expands that mapping to a {key, value} list. The result has no duplicate keys: a job variable overrides the same-named default rather than appearing twice.

defaults:
  environment:
    - key: PATH
      value: /bin:/usr/bin
    - key: LANG
      value: C
jobs:
  - name: job-a
    command: printenv
    schedule: "*/5 * * * *"
    environment:
      - key: PATH
        value: /usr/local/bin:/bin   # overrides the default PATH
        # LANG=C is inherited from defaults

The precedence chain for a variable is therefore: inherited process environment (including injected HOSTNAME) < env_file < merged environment (defaults then job, job winning) < daemon-injected CRONSTABLE_* run variables (with a state: section). See includes, defaults, and multi-file config for how defaults and includes are merged overall.

workingDirectory

workingDirectory is the directory the job's process starts in. It is the cwd of the spawn, and the equivalent of the Start in (optional) box on a Task Scheduler action or of systemd's WorkingDirectory=.

jobs:
  - name: nightly-report
    command: python report.py
    schedule: "0 2 * * *"
    workingDirectory: /srv/reports

With the key unset, the job inherits cronstable's own working directory. Like the other launch fields, it can be set in a defaults: block and on a DAG task. Under a defaults: block that sets it, a bare workingDirectory: on one job opts that job back out to inheriting.

At load, cronstable expands ~ and makes the result absolute:

  • ~/jobs means the home directory, not a directory literally named ~ under wherever cronstable was started. On a job that also sets user, the expansion resolves against cronstable's own user, because it happens once at load while the demotion happens per run.
  • A ${VAR} reference expands from cronstable's own environment like any other config scalar, unlike command and shell, which cronstable leaves verbatim for the runtime shell to expand. See environment variable interpolation.
  • A relative value resolves against cronstable's own working directory at load rather than at each fire, so the directory a job runs in is settled once and the logs show the resolved form. Nothing requires the value to be absolute: on Windows, what ntpath.isabs counts as an absolute path differs across the Python versions cronstable supports, and which directory a job runs in must not depend on the interpreter that scheduled it.

At load, cronstable does not check whether the directory exists. Config load runs on every hot reload, and under --validate-config on machines that are not the target host, so one job naming a share that is not mounted yet must not fail the whole load.

The OS checks at spawn instead, and a directory that is not there is an ordinary launch failure: the run records exit 127, and the logged spawn line carries the cwd that was attempted. On Windows, that log line is the only place the directory appears, because the error the OS returns there (WinError 267, "The directory name is invalid") carries no filename of its own.

workingDirectory is deliberately not part of the job-set ID, so replicas that run the same jobs from paths their own hosts spell differently still agree on what they are running.

Two behaviors to know before relying on it:

  • The child changes directory before preexec_fn runs, so on a job that also sets user/group the change uses cronstable's privileges, not the target user's. A demoted child can therefore end up in a directory it cannot itself read.
  • A list-form command that names its program by a relative path, one containing a separator such as ./import.sh, resolves that path against workingDirectory on POSIX and fails on Windows, where CreateProcessW searches the calling process's directory instead. A bare name with no separator is looked up on PATH on both platforms and never comes from the working directory, because CPython builds the PATH candidate list in the parent, before the child changes directory. Name the program by full path, or set shell: and let the shell resolve it. See running on Windows.

priority

priority is how the job is scheduled against the other processes on the host. On Linux, those are the processes in its CPU group (see priority and Linux CPU groups). It takes one of five levels, lowest first:

jobs:
  - name: reindex
    command: reindex.sh
    schedule: "0 3 * * *"
    priority: idle
Level Windows priority class POSIX nice
idle IDLE_PRIORITY_CLASS 19
below-normal BELOW_NORMAL_PRIORITY_CLASS 10
normal (default) no class set; see the notes after this table inherited, no renice
above-normal ABOVE_NORMAL_PRIORITY_CLASS -5
high HIGH_PRIORITY_CLASS -10

normal is the one level cronstable never applies. On POSIX, that means the job keeps cronstable's own nice, which on a daemon started at nice 10 is nice 10, not nice 0. On Windows, the job gets cronstable's own class when cronstable runs at idle or below normal, and NORMAL when cronstable runs at normal or above, because that is what CreateProcess gives a child with no class flag.

Either way, cronstable never promotes a job whose config does not request a higher level. Like the other launch fields, it can be set in a defaults: block and on a DAG task.

The other four levels are absolute, not offsets: idle means nice 19 whatever nice the daemon runs at.

The two platforms apply it at different moments:

  • Windows sets the priority class at CreateProcess time, the one race-free place to set it, on the same creation flags that give the job its own process group. normal emits no class flag rather than NORMAL_PRIORITY_CLASS, because a child only defaults to NORMAL when its creator is not itself idle or below-normal. Emitting the flag would silently promote the jobs of a daemon that was launched below normal, which is what Task Scheduler does by default.
  • POSIX has no such spawn-time knob, so cronstable renices the job's process group (setpriority with PRIO_PGRP) as the first thing after a successful spawn. It renices the group rather than the process, so a helper the shell forked in the microseconds before the call is reniced too. This does not happen in a preexec_fn: that hook runs between fork and exec, where only async-signal-safe calls are sound, and it would put a fork-time hook on every spawn, including the jobs that have no privilege to drop.

How far the level reaches also differs, and the difference comes from the same CreateProcess rule. Descendants inherit a lowered level (idle, below-normal) on both platforms. A raised level (above-normal, high) applies to the job's own process on Windows and not to the tree it spawns, because an unflagged child of an above-normal or high parent gets NORMAL.

That shows up in the commonest Windows job shape: a shell: cmd job or a .cmd file at priority: high runs cmd.exe at HIGH, and every program cmd.exe launches runs at NORMAL. POSIX has no such asymmetry, because setpriority(PRIO_PGRP) covers the whole group and anything forked after it inherits the nice value. high is still useful where the job's own process is the work, but on Windows it does not raise a tree of helpers.

Lowering a priority is always allowed. Raising one on POSIX (any level whose nice sits below cronstable's own; from the usual nice 0 that means above-normal and high) needs CAP_SYS_NICE or RLIMIT_NICE headroom, and cronstable cannot know at load whether the kernel will grant it. What it does instead:

  • at config load, if the level is a raise from cronstable's own nice and cronstable is not root, it logs one WARNING naming the job, so the warning lands on the deployment that introduces the raise;
  • at run time, a refusal from the kernel does not fail the run. The job's processes keep the nice value they inherited, and cronstable logs the refusal at DEBUG. A minutely job on an unprivileged host would otherwise emit some 1,440 warnings a day about a condition that does not change until the deployment does. While cronstable weighs CPU per run, the run's cgroup weight follows the raised level (see priority and Linux CPU groups).

Windows grants all five classes to an unprivileged account, so none of that applies there.

The priority enum deliberately omits realtime. That class outranks the threads that service disk, keyboard, and mouse, so one runaway job at REALTIME can put the host out of reach of the operator who has to stop it. priority: realtime is a load error listing the accepted levels, not a silent downgrade.

When a job sets it, the level (never the nice number or priority class it resolves to) is part of the job-set ID, so replicas that disagree about how a job is scheduled show as drift. A set level also appears on the GET /jobs payload. See running on Windows for how the levels compare with Task Scheduler's own -Priority numbers.

Priority and Linux CPU groups

On Linux, a nice value orders a process only against the processes in its own CPU group: a cgroup whose parent has the cpu controller on. Each group gets a share of contended CPU by its weight, whatever the nice values inside it. What priority reaches depends on the setup:

  • Under the packaged systemd unit, which delegates only the memory and pids controllers, cronstable and its jobs share a CPU group with the other system services, and priority orders a job against all of them. Another unit can make systemd turn the cpu controller on for every service, and then the next case applies.
  • A unit with Delegate=yes, which CPU caps need, makes cronstable a CPU group of its own among the system's services. priority then orders a job only against the daemon and cronstable's other jobs.
  • Where the cpu controller is delegated, as with Delegate=yes, cronstable turns it on while a job has a resourceLimits.cpu cap. Every run then gets a cgroup of its own whose CPU weight (cpu.weight.nice) follows its priority, with the daemon's own nice for normal. The kernel shares CPU per run, not per thread, and the nice value orders the threads inside each run. After a reload removes the last cpu cap, this ends once no run with a cpu cap is still going. The packaged unit does not delegate the cpu controller.

While cronstable weighs CPU per run, a run's cgroup weight follows its configured priority even when the kernel refuses the renice. That weight orders the run only against cronstable's other runs, within cronstable's own share of CPU.

See CPU caps and priority. Windows priority classes have no such groups: a Job Object's CPU rate cap groups no other processes, so a class orders a job against every process on the host.

user and group (privilege switching)

user and group request that the subprocess run under a different identity. Resolution happens in JobConfig._resolve_user_group (cronstable/config.py):

Windows: this whole feature is POSIX-only. Windows has no setuid/setgid model, so a job with user or group set raises the configuration error Job <name>: changing user/group is not supported on Windows (config.py _resolve_user_group). The Root requirement, Resolution rules, and Demotion ordering sections later on this page all apply to POSIX only. See running on Windows.

Resolution rules

  • user as a name (Str): looked up with getpwnam. Sets uid from pw_uid, gid from pw_gid (the user's primary group), and the resolved login name (pw_name). A missing user raises ConfigError("User not found: ...").
  • user as a number (Int): uid is set to the number directly. _resolve_user_group also looks the uid up with getpwuid to derive the user's primary gid and login name. If group was not given, the derived primary gid is used, so a numeric user without group does not silently keep cronstable's gid 0. If the uid is not in the passwd database, no login name or derived gid is available, and that is not an error here.
  • group as a name (Str): looked up with getgrnam; gid set from gr_gid. A missing group raises ConfigError("Group not found: ...").
  • group as a number (Int): gid is set to the number directly.
  • If only user is given, the group defaults to the main group of that user. An explicit group overrides any gid derived from user.

The resolved login name (username) matters for supplementary-group handling in _demote, described later. It is None when the user is unknown.

Root requirement

If, after resolution, either uid or gid is set and the cronstable process is not running as root (os.geteuid() != 0), config parsing fails with:

Job <name> wants to change user or group, but cronstable is not running as superuser

On POSIX, any use of user or group therefore requires cronstable to run as root. The daemon needs no special privileges otherwise, and user/group switching is the only feature that requires root. On Windows, user/group are rejected outright with a configuration error, so this root requirement is a POSIX-only statement. See the earlier Windows note.

jobs:
  - name: as-www-data
    command: id
    schedule:
      minute: "*"
    captureStderr: true
    user: www-data        # group defaults to www-data's primary group

Demotion ordering (_demote)

When uid or gid is set, start passes preexec_fn=self._demote, which runs in the child process after fork while still privileged. The order is deliberate and is required for safety:

  1. Supplementary groups first. If both a login name and a gid are known, os.initgroups(username, gid) gives the child exactly the target user's supplementary groups. Otherwise os.setgroups([]) drops all supplementary groups. A failure raises RuntimeError("setgroups/initgroups: ...").
  2. Primary gid next. If gid is set, os.setgid(gid). A failure raises RuntimeError("setgid: ...").
  3. uid last. If uid is set, os.setuid(uid). A failure raises RuntimeError("setuid: ...").

Supplementary groups and the gid must be changed before setuid, because after the process drops root with setuid it can no longer call setgroups/setgid. Performing them in the other order would leave the child holding root's supplementary group memberships (the classic "forgot setgroups() before setuid()" privilege-escalation bug).

PyInstaller environment fixup

On a frozen build (or when _PYI_* variables are inherited) every job gets a scrubbed copy of the environment: POSIX builds restore LD_LIBRARY_PATH and LIBPATH from their _ORIG values, and all _PYI_* bootloader variables are removed (pyinstaller_env_leaks and fixup_pyinstaller_env in cronstable/job.py):

if getattr(sys, "frozen", False) and not platform.IS_WINDOWS:
    for env_var in "LD_LIBRARY_PATH", "LIBPATH":
        env[env_var] = env.get(f"{env_var}_ORIG", "")
for key in [k for k in env if k.startswith("_PYI_")]:
    del env[key]

PyInstaller's POSIX bootloader overwrites LD_LIBRARY_PATH and LIBPATH so the bundled binary can find its own libraries, saving the caller's original values in LD_LIBRARY_PATH_ORIG/LIBPATH_ORIG. Restoring them (or emptying the variable if there was no _ORIG) keeps a child process off the frozen interpreter's library paths. The _PYI_* variables link a bootloader to its direct child; left in place, they make any frozen binary the job runs, the cronstable CLI included, refuse to start. The scrub runs before the job's environment is applied, so an explicit environment entry wins.

See production and container deployment for the frozen-binary build.

⚠️ **GitHub.com Fallback** ⚠️