Configuration Reference - ptweezy/cronstable GitHub Wiki
The canonical, exhaustive reference for the cronstable YAML configuration. It
documents the top-level structure and every per-job option, with the exact
strictyaml type, default, and load-time validation rule taken from
cronstable/config.py. Deep topics (schedules, reporting, the HTTP API, metrics,
logging, includes) have dedicated pages linked from each section.
A configuration is either a single YAML file or a directory of *.yml/*.yaml
files, selected with the -c flag (see the
command-line reference). The document is parsed and validated
against a fixed strictyaml schema (CONFIG_SCHEMA). An unknown key, a wrong
type, or a malformed value is a hard ConfigError at load time. An empty
document is valid.
Classic (Vixie-style) crontab files are accepted alongside YAML, recognized by
name (*.crontab, *.cron, or a file named crontab). Each entry is lowered
to an ordinary job definition and merged over the same DEFAULT_CONFIG
defaults documented later, so internally it is configured to cronstable's
standard behavior rather than an emulation of cron's. A crontab can only
define jobs. Every other section on this page (and any per-job option beyond
schedule, command, shell, timezone, and environment) is YAML-only.
See classic crontabs.
In the option tables that follow, "Required" means the strictyaml key is
mandatory (not wrapped in Opt(...)). Every other key is optional and falls
back to the default shown. Per-job defaults come from DEFAULT_CONFIG. A
defaults: block and any included files override DEFAULT_CONFIG, and an
individual job overrides both. See
includes, defaults, and multi-file config for the
precise merge order.
After validation, ${VAR} and ${VAR:-default} in any string value are
expanded from cronstable's environment ($$ is a literal $), so a listen
address, state path, time zone, or webhook URL can come from an environment
variable. Job and reporter command/shell fields are left for the runtime
shell, and the logging section is left for Python's logging.config. An
unset variable with no default is a hard ConfigError. See
environment-variable interpolation.
defaults: { ... } # optional: per-job defaults for this file
jobs: # optional: list of job definitions
- name: ...
command: ...
schedule: ...
dags: # optional: durable orchestration DAGs (needs state)
- name: ...
tasks: [ ... ]
include: [ ... ] # optional: list of other config files to merge
web: { ... } # optional: HTTP control API
mcp: { ... } # optional: Model Context Protocol server for AI agents
cluster: { ... } # optional: mTLS peer attestation / leader election
state: { ... } # optional: durable state store (history, catch-up, retries)
pools: { ... } # optional: shared capacity and durable admission queues
logging: { ... } # optional: Python logging dictConfig
notify: { ... } # optional: daemon/orchestration event notifications
push: { ... } # optional: E2E-encrypted push alerts (relay + device registry)| Key | Type | Required | Description |
|---|---|---|---|
defaults |
Map of the per-job common options |
No | Default values inherited by every job in the same file. May contain any per-job option except name, command, and schedule. See includes, defaults, and multi-file config. |
jobs |
Seq(Map) of job definitions |
No | The list of cron jobs. Each entry is validated against the per-job schema described later. |
dags |
Seq(Map) of DAG definitions |
No | Durable orchestration workflows: each DAG (directed acyclic graph) is a graph of tasks with dependsOn edges, run on a schedule. Requires a state section with jobApi enabled. See orchestration and DAGs. |
include |
Seq(Str) |
No | Paths (relative to the including file) of other config files to parse and merge. Include cycles raise a ConfigError. See includes, defaults, and multi-file config. |
web |
Map |
No | Enables the HTTP control API. See HTTP control API. |
mcp |
Map |
No | Enables the Model Context Protocol server (POST /mcp on the web listeners, plus the cronstable mcp stdio bridge) so AI agents can observe and, opt-in, control jobs/DAGs. Requires a web section. Off by default. See mcp later and MCP. |
cluster |
Map |
No | Enables mutual-TLS peer attestation and optional leader election across replicas. See clustering and leader election. |
state |
Map |
No | Enables the opt-in durable state store: restart-durable run history, missed-run catch-up, restart-surviving retries, and once-per-boot @reboot runs. Without it, cronstable is stateless: everything stays in memory. See durable state. |
pools |
MapPattern(Str, Map) |
No | Named resource pools (slots 1-10000, maxQueued default 1000, 1-10000). Requires state. See resource pools. |
logging |
Map (Python logging.config dictConfig) |
No | Custom logging configuration. See logging configuration. |
notify |
Map |
No | Daemon and orchestration event notifications (DAG failures, approval gates, leadership changes) through the standard reporter block. See notify later and reporting. |
push |
Map |
No | The daemon-global relay endpoint and paired-device registry behind the end-to-end encrypted push reporter. See push later and push notifications. |
| Option | Type | Default | Description |
|---|---|---|---|
listen |
Seq(Str) |
required | Listen URLs, such as http://127.0.0.1:8080, https://0.0.0.0:8443, or unix:///tmp/cronstable.sock. http:// and https:// listeners work everywhere, and a list may mix schemes, each address serving the same app over its own transport. An https:// entry needs the later tls.cert/tls.key keys, and both a host and a port. Windows does not support unix:// listeners (the Proactor loop lacks create_unix_server); they are skipped with the warning Ignoring web listen url <url>: unix-socket listeners are not supported on this platform. Use an http:// listener instead. See running on Windows and listener TLS. |
headers |
MapPattern(Str, Str) |
none | Extra HTTP response headers applied to all endpoints. A Content-Type entry (in any spelling) is ignored: every endpoint owns its own content type. Some routes set headers themselves, such as ETag, Vary, and Cache-Control, and an entry can't replace those; see response headers. |
allowedOrigins |
Seq(Str) |
[] |
Extra exact-match browser Origins allowed to call the mutating endpoints: every mutating (non-GET/HEAD/OPTIONS) route except /mcp, such as job start/cancel/pause/resume, DAG trigger/backfill/recover, task decisions, /push/devices, and /shutdown. Cross-site browser requests to them are refused 403 as a CSRF/DNS-rebinding defense. Same-origin requests (the served dashboard) and clients that send no Origin (curl, monitoring) always pass. /mcp keeps enforcing its own mcp.allowedOrigins. A headers entry of Access-Control-Allow-Origin: <origin> allow-lists that origin implicitly. Access-Control-Allow-Origin: "*" disables the check (logged loudly). |
authToken |
Map with value / fromFile / fromEnvVar (each EmptyNone() | Str) |
none | Opt-in bearer-token auth. When set but resolving empty, cronstable does not start the web API (it logs the error; scheduled jobs keep running). --validate-config does not resolve token sources, so it does not catch this. |
authTokens |
Seq(Map), each with the authToken source triple, scopes (Seq of view/control/approve), and an optional label
|
none | Multiple named bearer tokens with per-token scopes (control and approve imply view). label identifies a token in logs and lets you revoke it by dropping the entry and reloading. Tokens must resolve nonempty and be distinct from each other and from authToken. See HTTP control API. |
anonymousScopes |
Seq(Enum), accepting only view
|
none | Scopes granted to requests that present no credential at all, so the instance serves a public read-only board (dashboard, run history, log tails, calendar feeds, /metrics). The schema rejects control and approve, so no anonymous path to a mutating route can be configured. GET /push/devices is excluded as well, and a presented-but-wrong token is still 401 rather than a downgrade to this grant. Requires at least one authToken/authTokens entry, because a tokenless daemon already grants every scope to everyone. See public read-only access. |
socketMode |
Str |
none | Octal permissions applied to a unix:// listen socket. It only ever applies to unix sockets, so it is irrelevant on Windows, where unix:// listeners are unsupported. |
tls.cert |
EmptyNone() | Str |
none | Path to the certificate (chain) every https:// listener serves. Required together with tls.key: one without the other is a ConfigError at load, as is TLS material with no https:// address in listen, or an https:// address with no certificate. Whether the files exist is not checked at load (a mounted secret may not exist at first boot); an unloadable certificate keeps the web API down with a logged error and is retried on the next reload. Rotating these files in place restarts the listeners. See listener TLS. |
tls.key |
EmptyNone() | Str |
none | Path to the private key for tls.cert. |
tls.clientCa |
EmptyNone() | Str |
none | Path to a CA. When set, every https:// listener requires a client certificate signed by it (mutual TLS) and rejects the handshake otherwise. Unset, the listener does no client authentication. That CA file is the caller allowlist, because a server does no hostname verification. Point it at a dedicated CA issued only to cronstable clients, never a shared organizational one. Requires tls.cert/tls.key. Also satisfies the mcp.enabled no-token gate on an https:// listener (see mcp later). |
ui |
Bool |
true |
Serve the browser dashboard at /. ui: false omits the dashboard page while every JSON endpoint keeps working, for an API-only deployment. See web dashboard. |
metrics |
Bool | Map with enabled / public (each Bool) and durationBuckets (Seq(Float)) |
enabled | The Prometheus GET /metrics endpoint, served by default whenever the web API is on. metrics: false (the bool form) disables it. The map form sets enabled (default true), public (default false, exempting only /metrics from authToken), and durationBuckets (histogram bounds in seconds; must be finite, positive, and strictly increasing, else a ConfigError). See metrics with Prometheus. |
badges |
Bool | Map with enabled / public (each Bool) |
enabled | The SVG status badges at GET /badge.svg and GET /jobs/{name}/badge.svg, served by default whenever the web API is on. badges: false (the bool form) disables both routes. The map form sets enabled (default true) and public (default false). public: true exempts only the two badge routes from authToken, sends Cross-Origin-Resource-Policy: cross-origin in place of same-origin, and leaves out frame-ancestors and X-Frame-Options, so an image proxy can fetch the badges without a token and pages on other origins can load them. See status badges. |
nodeHistory |
Bool | Map with enabled (Bool), interval (Float), and points (Int) |
enabled | Background node CPU/memory sampling for the dashboard's node history chart, served by GET /node/history. On by default whenever the web API is on; nodeHistory: false disables the sampling task. The map form sets interval (seconds between samples, default 5.0, minimum 1.0) and points (ring size, default 720, the last hour at the default cadence; 10–50000). The ring is in-memory only and follows the web app's lifecycle. |
bonjour |
Bool | Map with enabled (Bool) and name (Str) |
off | Opt-in zero-config LAN discovery: advertises the web API as a _cronstable._tcp mDNS/Bonjour service so a companion app on the same network finds the daemon without a typed URL. Requires the discovery extra (python-zeroconf) and at least one TCP listen address; enabling it without the library is a ConfigError, never a silent no-advert. The map form overrides the advertised instance name. See LAN discovery. |
listen is the only required key. Full behavior, authentication, and endpoint
semantics are documented in HTTP Control API. The tls block,
certificate rotation, and the client-side flags are documented in
Listener TLS.
Opt-in Model Context Protocol server, so an
AI agent (Claude Desktop/Code, Cursor, VS Code Copilot, and other clients) can
drive cronstable the way an operator drives the dashboard. It is served as a
stateless Streamable-HTTP endpoint at POST /mcp on the existing
web.listen addresses (inheriting the same authToken / unix-socket auth) and
over the cronstable mcp stdio bridge for desktop clients. Every field is
optional. The server is off unless enabled: true, and requires a web
section (there is nowhere to serve it otherwise). Tools operate on the same data
as the REST API, so there is one source of truth.
| Option | Type | Default | Description |
|---|---|---|---|
enabled |
Bool |
false |
Serve POST /mcp and expose the cronstable mcp stdio bridge. |
readOnly |
Bool |
true |
Strip every mutating tool: the nine tools that run, cancel, pause, or resume a job, cancel queued work, or trigger, backfill, recover, or approve a DAG. On by default: agents get read-only access unless the operator opts into control. Takes precedence over toolsets (act stays suppressed while true). |
toolsets |
Seq(Enum) of observe / act / dags / state
|
[observe] |
Which tool groups to expose. observe = read-only job/cluster/metrics views. dags = DAG introspection (+ control when readOnly:false). state = durable-state inspector (redacted). act = mutating job control (only when readOnly:false). |
allowedOrigins |
Seq(Str) |
[] |
Exact-match browser Origins allowed to call /mcp. Empty serves non-browser clients only, so a present Origin not on the list is refused 403 (a DNS-rebinding defense). A nonempty list additionally answers CORS preflight with a scoped Access-Control-Allow-Origin. |
allowUnauthenticated |
Bool |
false |
Serve /mcp on a routable (non-loopback, non-socket) listener even with no web.authToken. Fail-closed default: with no token the web app has no auth middleware at all, so an enabled /mcp on a routable address raises a ConfigError at load. An https:// listener with web.tls.clientCa set is exempt without this flag, because mutual TLS authenticates the caller. Plain https:// is not. Set true only when the endpoint is protected by other means (an mTLS-terminating proxy, a network policy). |
resources |
Bool |
true |
Expose MCP resources: URI-addressable read-only context like cronstable://status and cronstable://jobs/{name} that mirrors the read tools for clients that consume resources. Their scope follows toolsets (a cronstable://dags/{name} resource is served only when the dags toolset is on). |
prompts |
Bool |
true |
Expose MCP prompts: canned triage playbooks (triage_job_failure, why_did_dag_run_fail, blast_radius, fleet_health_summary, backfill_plan) that chain the read tools into repeatable workflows. A prompt is served only when every tool it calls is: why_did_dag_run_fail needs the dags toolset, and backfill_plan needs dags with readOnly: false, so the default [observe] exposes the other three. |
instructions |
EmptyNone() | Str |
none | Optional free-text server instructions surfaced to the client at initialize. |
maxRows |
Int |
200 |
Ceiling on any list tool's limit; a larger request is capped (never an error) and a nextOffset, the integer offset of the next page, is offered for the rest. Must be >= 1 (a ConfigError at load). |
maxBodyBytes |
Int |
1048576 |
Cap on a single /mcp request body (tool arguments arrive from an LLM); an oversized POST is refused 413. Must be >= 1 (a ConfigError at load). |
web:
listen:
- http://127.0.0.1:8080
authToken:
fromEnvVar: CRONSTABLE_WEB_TOKEN # also gates /mcp
mcp:
enabled: true
readOnly: false # allow mutating tools
toolsets: # jobs + DAG reads/control
- observe
- dags
- actWire a client to it with, for example, claude mcp add --transport http cronstable https://host/mcp --header "Authorization: Bearer $TOKEN", or over
stdio with claude mcp add cronstable -- cronstable mcp --url http://127.0.0.1:8080.
This section is only the configuration schema. MCP documents the full tool catalog, the prompts and resources, client setup recipes, and the stdio bridge's flags.
Optional. A leadership backend coordinates scheduled jobs across replicas
using one configuration. Execution guarantees depend on the backend and each
job's clusterPolicy. cluster.backend selects the coordination mechanism:
- The default
gossipbackend attests, over mutual TLS, that a static list of peers is running the same job set, and runs a best-effort quorum election. - The
kubernetesandetcdbackends use a coordination store (aLease/ a lease-bound key) for fenced leadership with a single lease holder. - The
filesystembackend elects through a fenced TTL lease on a shared POSIX mount, with no coordination service at all. Its safety additionally rests on synchronized clocks; see its table later.
There can be at most one cluster block across the whole configuration; a
duplicate in an included file or a second config-directory file raises a
ConfigError. Defaults come from DEFAULT_CLUSTER (plus DEFAULT_K8S /
DEFAULT_ETCD / DEFAULT_FILESYSTEM for the lease backends) and are applied
only when a cluster section is present.
| Option | Type | Default | Description |
|---|---|---|---|
backend |
Enum(["gossip", "kubernetes", "etcd", "filesystem"]) |
gossip |
Which leadership backend gates jobs. gossip (default) is the embedded mTLS best-effort election; kubernetes/etcd/filesystem are fenced lease backends. The built-in Kubernetes and etcd transports use HTTP APIs through the core aiohttp dependency, with HTTPS supported. filesystem needs only a shared POSIX mount, so none of these transports adds a runtime dependency. |
Gossip backend (backend: gossip). listen, tls, and peers are
required only for this backend:
| Option | Type | Default | Description |
|---|---|---|---|
listen |
Str |
required (gossip) |
host:port the mTLS /peer listener binds to (for example, 0.0.0.0:8443). Served only here, never on the public web API. |
tls.ca |
Str |
required (gossip) | Path to the cluster CA (trust anchor for peer certificates). Must be nonempty: a blank or whitespace-only value is a ConfigError at load (cluster.tls.ca must be a non-empty path; a blank value would disable peer authentication). The same rule covers tls.cert and tls.key. |
tls.cert |
Str |
required (gossip) | Path to this node's certificate (used both to serve /peer and to authenticate as a client). Its SAN must match the host other nodes use to reach it. |
tls.key |
Str |
required (gossip) | Path to this node's private key. |
peers |
Seq(Map({"host": Str})) |
required (gossip) | Every other member as host:port. Cluster size is len(peers) + 1. |
nodeName |
Str |
system hostname | Stable, human-readable identity for this node; the leader is the lowest nodeName among agreeing members. Must be unique across the cluster: a duplicate is detected at runtime (status conflict) and pauses Leader jobs until resolved. The hostname default is already unique per host. (Also used as the lease backends' default identity / etcd key value.) |
interval |
Int |
30 |
Seconds between peer-attestation rounds. Must be > 0. |
driftAfter |
Int |
3 |
Consecutive reachable-but-mismatched rounds before a peer is reported drifted (debounce). Must be >= 1. |
connectTimeout |
Int |
10 |
Seconds per request (also the HTTP timeout for the lease backends). Must be > 0. |
electLeader |
Bool |
false |
When true, only the quorum-gated elected leader runs scheduled jobs (manual API triggers and retries are unaffected). Off by default, so a gossip cluster section is observe-only until opted in. The lease backends imply electLeader: true (configuring one is opting into leadership). |
distribution |
Enum(["single-leader", "spread"]) |
single-leader |
How leader-gated jobs spread across the quorate cluster. single-leader: one elected leader runs every Leader job. spread: per-job ownership through rendezvous hashing, so the work fans out across the quorate nodes (same quorum gate, same guarantee). Inert without electLeader (warns if set anyway). With a lease backend (kubernetes/etcd/filesystem) a non-default distribution is a hard ConfigError at load (a single lease holder cannot be a per-job owner), not a silent fallback. See clustering and leader election. |
Gossip load-time validation (in addition to the numeric ranges listed
earlier): every tls path (ca, cert, key) must be nonempty, so a blank
scalar (what a template renders for an unset variable) is a ConfigError at
load rather than a /peer listener that requires no client certificate. Every
address (listen and each peers[].host) must be host:port
with a numeric port in 1-65535, and an IPv6 host must be written
bracketed ([2001:db8::1]:8900). A bare IPv6 literal is a ConfigError at
load (looks like a bare IPv6 address; write it as [ipv6]:port), because
splitting it at the last colon would silently mis-read the host and, for a
peer, drop it from quorum with no error.
The same address checks apply to the
observability overlay's listen/peers, which are
built through the same code path.
With electLeader: true, a 2-node cluster (one peer) is rejected outright
with a ConfigError (a quorum of 2 needs both up, strictly worse than one
replica). An even cluster size greater than 2 is allowed but logs a
warning (an odd count is best for a clean majority).
Kubernetes backend (backend: kubernetes), under cluster.kubernetes. A
coordination.k8s.io/v1 Lease is the fence. Defaults from DEFAULT_K8S:
| Option | Type | Default | Description |
|---|---|---|---|
leaseName |
Str |
cronstable-leader |
Name of the Lease object the replicas contend for. Must be a valid RFC1123 subdomain (lowercase alphanumerics, - and .; <=253 chars), checked at load; it is spliced into the apiserver URL path, so a stray /, ?, #, or space is rejected. |
leaseNamespace |
Str or null |
null → in-cluster namespace | Namespace of the Lease; defaults to the pod's own namespace (the service-account namespace file). When set, must be a valid RFC1123 label (lowercase alphanumerics and -; <=63 chars), checked at load. |
leaseDurationSeconds |
Int |
15 |
How long a renewal keeps the lease valid. Must be > renewDeadlineSeconds. |
renewDeadlineSeconds |
Int |
10 |
Per-round renew/observe deadline: a round that exceeds it is abandoned and retried next round, so a stuck apiserver call cannot run out the full lease. Must be > 0 and < leaseDurationSeconds. |
retryPeriodSeconds |
Int |
2 |
Seconds between renew/observe rounds. Must be > 0 and < renewDeadlineSeconds (a holder must be able to attempt a renew before its own deadline). Additionally, renewDeadlineSeconds + retryPeriodSeconds < leaseDurationSeconds is enforced at load, so the worst-case interval between two successful refreshes still fits inside the lease. |
identity |
Str or null |
null → nodeName
|
The human-readable holder for this node (shown in the dashboard / GET /cluster). The daemon appends a per-process token to the holderIdentity it writes (<identity>#<token>), so two nodes sharing an identity/nodeName still write distinct holders and cannot both believe they hold the Lease. See node identity. |
kubeconfig |
Str or null |
null → in-cluster | Path to a kubeconfig file for local testing or use outside a cluster. If omitted, the backend uses in-cluster service account credentials. The built-in HTTP transport supports an inline bearer token, a tokenFile read before each request, or a client certificate and key supplied as files or embedded data. An inline token takes precedence over tokenFile. Relative paths resolve against the kubeconfig directory. If certificate-authority or certificate-authority-data is set, only that CA is trusted; otherwise, the system trust store applies. insecure-skip-tls-verify disables server certificate verification and logs a warning. Users that require an exec credential plugin or an auth-provider must use clientLibrary: library with cronstable[kubernetes]; the HTTP transport raises ConfigError for these configurations. |
apiServer |
Str or null |
null | Override the apiserver URL (else the in-cluster KUBERNETES_SERVICE_* env or the kubeconfig). When set, must be an https:// URL: a non-https value is a ConfigError at load, because the ServiceAccount bearer token must not travel in cleartext. |
clientLibrary |
Enum(["auto", "http", "library"]) |
auto |
Transport selection. auto uses the official kubernetes client when it is importable (install cronstable[kubernetes]) and otherwise falls back to a hand-rolled apiserver REST transport over aiohttp. library requires the native client (a ConfigError if absent). http forces the hand-rolled transport. |
etcd backend (backend: etcd), under cluster.etcd. A lease-bound key is
the fence; the backend uses etcd's v3 gRPC-gateway JSON/HTTP API directly (no
native client). Defaults from DEFAULT_ETCD:
| Option | Type | Default | Description |
|---|---|---|---|
endpoints |
Seq(Str) |
["http://127.0.0.1:2379"] |
etcd client URLs, tried in order for failover. Each must be http(s)://host[:port]. The port is optional (it defaults to the scheme's port, such as 443 behind an https ingress), and only an explicitly out-of-range port is rejected. Credentials embedded in the URL are refused. |
electionName |
Str |
cronstable/leader |
The etcd key contended for; its value is the holder's nodeName. There is no separate identity key for etcd (the holder identity is always cluster.nodeName), but leadership is fenced on the bound lease id, not this string, so a duplicate nodeName cannot make two nodes both lead. See node identity. |
ttl |
Int |
15 |
Lease time-to-live, seconds. Must be >= 3: the leader holds the key only until ttl minus a 1s clock-skew margin, so a smaller ttl would make a fresh winner treat its own lease as already expired (no Leader job would ever run). The keepalive cadence is ~ttl/3 against the effective ttl, which etcd may grant smaller than requested (a smaller granted TTL narrows the fence window). |
username |
Str or null |
null | etcd auth username (omit for an auth-less cluster). Pair it with a resolvable password. The auth token is re-fetched automatically when it expires (re-auth on a 401). |
password |
Map with value / fromFile / fromEnvVar (each EmptyNone() | Str) |
unset | etcd auth password source, resolved like web.authToken from the first set of value, fromFile, fromEnvVar; a configured-but-empty source fails closed. |
tls.ca / tls.cert / tls.key
|
Str or null |
null | Optional client TLS for https:// endpoints. tls.cert and tls.key are all-or-nothing (a client certificate needs its private key), enforced at load. |
etcd load-time guards: any TLS material (tls.ca/tls.cert/tls.key) requires
at least one https:// endpoint (otherwise it would be silently ignored and
traffic sent in cleartext, a ConfigError). Likewise a username or resolved
password requires every endpoint to be https://, so the credentials and
bearer token are never POSTed in cleartext. A username without a resolvable
password is also rejected.
One more etcd check is advisory rather than fatal: a small ttl also shrinks
the per-request timeout each renew POST gets (roughly
(ttl - max(1, ttl/3) - 1s) / 5), and when that budget falls below ~1 second
(any integer ttl from 3 to 8) load emits a one-time startup advisory.
If a single round-trip to etcd is slower than the budget (such as a cross-AZ or
cross-region endpoint), every renew round times out, the node treats a
reachable etcd as unreachable, and Leader jobs fail closed. It is the
operator's explicit ttl choice and a local, low-latency etcd is fine, so
cronstable warns rather than rejects. Raise ttl otherwise.
Filesystem backend (backend: filesystem), under cluster.filesystem. The
flock-guarded, fence-counted TTL lease of the durable state store's filesystem
backend is the fence, taken over a shared POSIX mount (Amazon EFS (NFSv4) / S3
Files). There is no coordination service: the mount is the store. Defaults from
DEFAULT_FILESYSTEM:
| Option | Type | Default | Description |
|---|---|---|---|
path |
Str |
required | Directory the election lease lives in, normally a shared mount. Must be present and nonempty: a missing, blank, or whitespace-only value is a ConfigError at load (cluster.filesystem.path is required and must be non-empty). When both are configured, sharing the state store's directory (same mount, same deploymentId) is legal and recommended: the namespaces are disjoint, and one mount stays the whole coordination surface. |
electionName |
Str |
cluster/leader |
Name of the lease the replicas contend for. Must be nonempty: a blank or whitespace-only value is a ConfigError at load (cluster.filesystem.electionName must be non-empty). |
ttl |
Int or Float
|
15 |
Lease time-to-live, seconds. Must be >= 3, for the same reason as etcd's floor: the leader holds the lease only until ttl minus a clock-skew margin and renews every max(1s, ttl/3), so a smaller ttl would make a node that wins the election immediately treat its own lease as expired (no Leader job would ever run). |
deploymentId |
Str or null |
null → namespace default
|
Stable namespace prefix inside the store, same semantics as state.deploymentId. Use the same value as state.deploymentId when sharing a mount with the state store. |
topology |
Enum(["auto", "single-node", "shared"]) |
auto |
Whether the mount's locks may be trusted across hosts; same semantics and probe as state.topology. auto probes /proc/mounts for a shared network mount. Windows and macOS cannot probe, so there auto resolves to single-node (the election then only excludes processes on this host, with a startup warning) unless overridden with an explicit shared. |
The lease-backend family rules apply exactly as for kubernetes/etcd:
electLeader is forced on (configuring the backend is opting into
leadership), distribution: spread is a hard ConfigError, a store block
belonging to a different backend (say an etcd: block under
backend: filesystem) is a ConfigError rather than silently ignored, and
gossip-only keys (listen, tls, peers, interval, driftAfter) draw an
emit-once startup advisory.
One filesystem-backend guard is deferred to startup, because it needs the live
mount. start() probes lock fidelity and hard-refuses a store whose locks
are no-ops, verbatim
cluster.backend filesystem: refusing to elect over <path>: <reason>. The
probe requires two descriptors of one file to contend on a non-blocking
exclusive lock; on Linux start() additionally refuses an NFS mount carrying
nolock or local_lock=flock/local_lock=all, because those honor locks
host-locally.
A refused start leaves the cluster manager unbuilt, so Leader jobs fail
closed (the safe direction). Both checks run on one host, so on platforms
without /proc/mounts (Windows, macOS) the residual risk rests on the
operator's topology: shared assertion.
Unlike the kubernetes/etcd fences, election safety here also rests on wall
clocks (two leaders need inter-host clock skew above roughly 2 seconds): run
NTP on every node, the same requirement durable state
documents for shared mounts.
Because the cluster schema has many load-time rejections (the ordering rules,
the RFC1123 and https guards, the credential-over-plaintext refusals, and the
lease-family rules listed earlier), check a cluster configuration before
deploying with cronstable --validate-config, which runs the full load path
and prints the first ConfigError without starting the scheduler. See the
command-line reference.
Full behavior, the trust model, quorum math, the lease backends' guarantees,
and per-job clusterPolicy are documented in
Clustering and Leader Election.
cluster.observability shares fleet data (each node's live CPU/memory, see
GET /node, and per-job run summaries) across the cluster
for the dashboard's fleet view,
independent of which backend owns election. It exists because the fleet view
depends on node-to-node gossip, which the lease backends do not have: it lets a
kubernetes/etcd/filesystem cluster run a second, election-inert gossip
mesh purely to carry that data.
| Option | Type | Default | Description |
|---|---|---|---|
shareNodeStats |
Bool |
true |
Gossip this node's whole-node CPU/memory for the fleet view. Set false to run the overlay mesh for job summaries only. |
listen, tls, peers
|
as backend: gossip
|
n/a | The overlay gossip transport. Required with a lease backend (the overlay is its own mesh); rejected with backend: gossip (redundant, because the election mesh already carries fleet data). |
nodeName, interval, driftAfter, connectTimeout
|
as backend: gossip
|
gossip defaults | Optional overlay mesh tuning. Lease backends only (they tune the dedicated overlay mesh); rejected with backend: gossip, where node stats ride the election mesh and the cluster-level keys of the same names already apply. |
Two shapes:
-
backend: gossip: the election mesh already exchanges/peerbodies, soobservabilityis an opt-in marker. Anobservabilityblock withshareNodeStats: trueadds node CPU/memory to what that mesh already gossips.listen/tls/peershere are aConfigError(redundant), and so are the overlay tuning keysnodeName/interval/driftAfter/connectTimeout(there is no overlay mesh to tune; the stats gossip atcluster.interval, so set the cluster-level keys instead). The stats ride each/peerresponse as anX-Cronstable-Node-Statsheader, never in the body, so sharing live load keeps the mesh's idle304optimization intact. -
A lease backend (
kubernetes/etcd/filesystem): election stays with the lease store.observabilityruns a dedicated gossip mesh (its ownlisten/tls/peers, all required) that never elects (it holds no leadership and gates no jobs), purely to carry fleet data.
cluster:
backend: kubernetes # election via a coordination.k8s.io Lease
kubernetes:
leaseName: cronstable-leader
observability: # a gossip mesh JUST for the fleet view
listen: "0.0.0.0:8140"
tls:
ca: /tls/ca.pem
cert: /tls/node.pem
key: /tls/node.key
peers:
- host: node-b:8140
- host: node-c:8140Requires psutil (a core dependency) for
the CPU/memory numbers; a node that cannot read its own load shares none.
Node stats are best-effort observability: a malformed or hostile peer payload
degrades to "no data for that node", never poisoning the view.
Optional. Enables the durable state store: restart-durable run history,
missed-run catch-up, restart-surviving retries, once-per-boot @reboot
dedupe, restart-durable Prometheus job counters, and durable output archival.
By default cronstable is stateless: absent this section, everything stays in
memory.
The store is a directory of immutable JSON records behind a single filesystem backend. A local path gives single-node durability, while a shared Amazon EFS (NFSv4) / S3 Files mount gives the same durability and coordination fleet-wide (the same code either way; the mount decides the reach).
There can be at most one state block across the whole configuration; a
duplicate in an included file or a second config-directory file raises a
ConfigError. Defaults come from DEFAULT_STATE and are applied only when a
state section is present. (The web dashboard's browser-side
IndexedDB run ledger is a separate, client-local feature, unrelated to this
store.)
| Option | Type | Default | Description |
|---|---|---|---|
path |
Str |
required | Directory the store lives in. A local path gives single-node durability; a shared Amazon EFS (NFSv4) / S3 Files mount gives the same durability fleet-wide. Must be nonempty: a blank or whitespace-only value is a ConfigError at load (state.path is required and must be non-empty). |
topology |
Enum(["auto", "single-node", "shared"]) |
auto |
Whether the store may offer cross-node coordination. auto probes /proc/mounts for a shared network mount (NFS/EFS/S3 Files) and otherwise assumes single-node; Windows and macOS cannot probe, so there auto resolves to single-node unless overridden with shared. |
deploymentId |
Str |
none (namespace default) |
Stable namespace prefix so several deployments can share one store/bucket without colliding or cross-reading. Unset means the default namespace. |
maxRunsPerJob |
Int |
1000 |
Durable finished-run retention per job (the durable analogue of the in-memory history ring); an append prunes the ledger to this, unless its row is recorded after a run of the same job that ended later (see the durable run ledger). <= 0 disables pruning (unbounded; rely on an external lifecycle rule). Durable retention is larger than the in-memory window on purpose. The daemon raises 1 to 2 and logs the adjustment once at startup, because the prune keeps the newest records by write order and a retention of one can leave the older run of an inverted pair; see durable state. |
onStoreUnavailable |
Enum(["degrade", "fail-closed"]) |
degrade |
What the stateful features do while the store is configured but unavailable (down, unreadable, unresponsive). degrade: durable-truth gates fail open to the in-memory state and failed writes are dropped with a warning (counted in cronstable_state_dropped_writes_total). fail-closed: prefer not running over possibly running wrong, so the onlyIfLastSucceeded gate blocks, a due durable retry defers until the store answers, and an unverifiable @reboot boot marker skips the boot run. Plain scheduled fires are never gated on the store under either policy. |
gcGraceSeconds |
Int |
604800 (7 days) |
Age past which durable state belonging to a job that no recent manifest references (no node's loaded config under this deploymentId has mentioned it for this long) is garbage collected. <= 0 disables automatic GC. Values between 1 and 86399 are a ConfigError at load: a grace below the manifest cadence would make live peers' manifests look stale and collect their state. |
maxOpsPerSecond |
Int or Float
|
0 |
Token-bucket cap on store operations per second (burst of one second's tokens), for request-rate/cost control on mounts that bill per request; throttled ops queue and are counted. 0 disables throttling. Must be >= 0 (a negative value is a ConfigError at load). Lease (coordination) operations bypass the bucket: a lease renew queued behind bulk writes could overshoot its TTL and double-run the very job the lease exists to fence. A cap of about 2 or lower slows workflow run retention and can delay scheduled workflow runs; see rate limiting. |
slotTtlSeconds |
Int or Float
|
30 |
TTL, in seconds, of the per-job concurrency slot lease taken for concurrencyScope: cluster jobs; the running holder renews it at a third of this, and a crashed holder's slot frees itself after at most this long. Must be >= 5 (a ConfigError at load, state.slotTtlSeconds must be >= 5 and finite): the renew cadence needs headroom, and below ~5s one slow renew on a network mount expires a still-running holder's slot and invites the cross-node double-run the lease exists to fence. |
jobApi |
Map |
(see the next table) | The job-facing state endpoint: a loopback HTTP server the daemon injects into every job's environment, backing the cronstable state|cursor|lock|artifact|idempotent|secret commands. A nested block (merged over its defaults, so a partial block keeps the rest). |
The state.jobApi sub-keys:
| Key | Type | Default | Meaning |
|---|---|---|---|
enabled |
Bool |
true |
Run the loopback endpoint and inject its address/token into every job. false keeps the durable scheduler features but exposes nothing to jobs. |
listen |
Str |
(ephemeral) | Override the bind, as an http://host:port or https://host:port URL, or a bare host:port (a unix:// URL is a ConfigError: the job CLI speaks TCP only). Unset binds an OS-assigned ephemeral port on 127.0.0.1. An explicit port must be an integer in 0-65535 (a ConfigError otherwise; 0 or omitting the port keeps the ephemeral bind), and a non-loopback host is a ConfigError unless allowNonLoopbackBind is also true. https:// requires the later tls.cert/tls.key keys and a named host: a wildcard (0.0.0.0, ::) is a ConfigError, because jobs dial the address they are handed and no certificate covers every interface. The address advertised to jobs as CRONSTABLE_STATE_URL is the configured host, not the bound one. |
maxValueBytes |
Int |
1048576 |
Cap (bytes) on one KV / cursor value; a larger set is refused (HTTP 413). Must be >= 0. |
maxArtifactBytes |
Int |
67108864 |
Cap (bytes) on one artifact payload; a larger put is refused (HTTP 413). Must be >= 0. |
lockTtlSeconds |
Int or Float
|
30 |
TTL of a job mutex/semaphore lease, renewed by the daemon at a third of this. Must be >= 5 (a ConfigError, state.jobApi.lockTtlSeconds must be >= 5 and finite), for the same reason as slotTtlSeconds. |
allowNonLoopbackBind |
Bool |
false |
Explicit opt-in for a non-loopback listen host. Without it, a non-loopback host is a ConfigError: the endpoint serves per-run bearer tokens and staged job secrets, so exposing it beyond this host needs a deliberate choice. With it set and a plaintext http:// listen, startup logs a warning naming the exposure (those bytes cross the network in the clear). Pair it with the tls keys that follow to serve the endpoint over TLS in-process, or terminate TLS in front of it. |
tls.cert |
EmptyNone() | Str |
none | Path to the certificate (chain) an https:// listen serves. Required together with tls.key: one without the other is a ConfigError at load, as is TLS material with a non-https:// listen, or an https:// listen with no certificate. This listener builds its TLS context once at startup and has no rotation detection, so replacing these files takes effect on the next daemon restart. Mutual TLS is not offered here: the per-run bearer token already authenticates the caller. See listener TLS. |
tls.key |
EmptyNone() | Str |
none | Path to the private key for tls.cert. |
tls.ca |
EmptyNone() | Str |
none | Path to the trust anchor the daemon hands its own jobs as CRONSTABLE_STATE_CACERT, so the job CLI can verify a certificate no public root signed (the normal case for an internally-issued one). This is the opposite direction from web.tls.clientCa: it verifies the endpoint rather than authenticating callers. Unset when the certificate chains to a root the jobs' hosts already trust. Counts as TLS material for the https://-listen guard described later. |
path is the only required key. Durable state documents the
full behavior: the store layout and durability model, restart-surviving
retries, missed-run catch-up, @reboot dedupe, the service level agreement
(SLA) trends endpoint, garbage collection, the state metrics, and the
cronstable state CLI subcommands (backup, restore, migrate, gc, check,
migrate-schema). The per-job knobs that build on this store are under
durable state and catch-up later.
A list of durable orchestration DAGs. Requires a state section with
jobApi.enabled (the default). Full guide: orchestration and DAGs.
Per-DAG keys:
| Key | Type | Default | Description |
|---|---|---|---|
name |
Str |
required | Unique DAG name. |
tasks |
Seq(Map) |
required | The task nodes (at least one). |
schedule |
Str or Map
|
none | Same grammar as a job's schedule, except it must parse to a cron expression: @reboot is a ConfigError (dag 'etl': schedule '@reboot' is not a cron expression; DAG schedules must be cron expressions (@reboot is not supported for dags)), while @daily/@hourly-style aliases still work. Omit for a manual-only DAG. |
timezone / utc
|
Str / Bool
|
as jobs | Schedule time base, as jobs. |
onMissed |
skip / run-once / run-all
|
skip |
Missed-run catch-up on restart, as jobs. |
startingDeadlineSeconds |
Int |
none | Bound how old a missed run may be to replay. |
catchupJitterSeconds |
Int |
0 |
Spread boot-time catch-up. |
clusterPolicy |
Leader / PreferLeader / EveryNode
|
Leader |
Which node schedules the DAG under leader election. |
enabled |
Bool |
true |
Disable without deleting. |
retainRuns |
Int |
50 |
Keep the newest N terminal runs (must be ≥ 1). On a rate-limited or slow store, a busy DAG can hold more between retention passes. |
Per-task keys:
| Key | Type | Default | Description |
|---|---|---|---|
id |
Str |
required | Unique task id within the DAG. |
command |
Str or Seq(Str)
|
required (not for approval) |
The command to run. |
type |
task / sensor / approval
|
task |
Node kind. |
dependsOn |
Seq(Str) |
[] |
Upstream task ids. |
triggerRule |
all_success / all_done
|
all_success |
When the task becomes ready. |
retries |
Int |
0 |
Per-task retry attempts (DAG-owned). Must be >= 0: the job-level -1 retry-forever sentinel is a ConfigError here. |
retryDelaySeconds |
Int/Float
|
0 |
Delay between attempts. |
expand |
Map{fromTask, key} |
none | Dynamic mapping: fan out over an upstream's XCom (cross-communication) list (a direct, non-mapped dependency). |
pokeIntervalSeconds |
Int/Float
|
30 |
Sensor: seconds between pokes. |
pokeTimeoutSeconds |
Int/Float
|
3600 |
Sensor: give up after this long. |
pokeJitterSeconds |
Int/Float
|
0 |
Sensor: jitter added to each poke. |
onReject |
fail / skip
|
fail |
Approval gate: what a rejection does. |
Plus the shared launch fields a job takes: shell, environment,
captureStdout / captureStderr, monitorResources, saveLimit,
maxLineLength, streamPrefix, failsWhen, executionTimeout,
killTimeout, priority, resourceLimits, statsd, user / group,
env_file, workingDirectory, secrets, stateAllowedScopes, verify,
pool, poolSlots, queuePriority, queueTimeout, and report-only
onFailure / onSuccess hooks (each accepts a report block that fires on
the task's runs; there is no onFailure.retry on a task, because a task's
attempts come from the node's retries field listed earlier). Where a task's
monitorResources numbers surface is covered under metrics later.
The graph is validated at load: unknown/duplicate ids, a cycle, a self-edge, or
an expand.fromTask that is not a direct non-mapped dependency are config
errors.
A standard Python logging.config dictionary-schema. version (Int) is
required; incremental, disable_existing_loggers, formatters, filters,
handlers, loggers, and root are optional. See
logging configuration.
Daemon and orchestration event notifications: events that belong to no single job run (so no per-job hook would fire) delivered through the same reporter machinery as the per-job hooks. See reporting for the delivery semantics and templates.
| Option | Type | Default | Description |
|---|---|---|---|
events |
Seq(Enum) |
all four | Which events to deliver: dag_failure (a DAG run reached a terminal failed state), approval_waiting (an approval gate began awaiting a decision), leader_change (this node acquired or lost scheduled-job leadership), quorum_loss (this node left quorum, so Leader jobs stand down). |
report |
_report_schema (mail/sentry/shell/webhook/push/eventlog) |
event-worded defaults | The reporter block fired per event, sharing the per-job report schema; its default subject/body templates are worded for daemon events rather than run completions. |
The daemon-global half of the end-to-end encrypted push reporter: where
sealed alerts are relayed and where device pairings are stored. Turning the
reporter on stays per-job/per-event (report.push.enabled /
notify.report.push); this section only says where alerts go. Requires the
push extra (PyNaCl). Without it, the daemon refuses to start on a
configuration that has a push: section. See
push notifications.
| Option | Type | Default | Description |
|---|---|---|---|
relay.url |
Str |
required | The relay endpoint alert ciphertext is POSTed to. Explicit and required: the daemon never posts anywhere the operator did not spell out. Treated as a secret in logs. |
relay.timeout |
Float |
10 |
Total relay request timeout, seconds. |
devicesFile |
EmptyNone() | Str |
none | Path of the paired-device registry for stateless installs. With a state: section the registry rides the durable store instead (cluster-visible); one of the two must exist. |
allowUnauthenticated |
Bool |
false |
Escape hatch for the fail-closed web-auth gate (the twin of mcp.allowUnauthenticated): serve the /push/devices pairing endpoints on a routable listener that has no web.authToken. |
Every key in this section comes from _job_schema_dict (jobs) / _job_defaults_common
(defaults and the per-file defaults). Defaults are from DEFAULT_CONFIG.
name, command, and schedule are required on a job; all other keys are
optional. The three keys name, command, and schedule are not allowed
in a defaults block (only the common keys are).
| Option | Type | Default | Description |
|---|---|---|---|
name |
Str |
required | Job identifier, unique across the whole assembled configuration (all files and includes together): the scheduler tracks jobs by name, so a duplicate is a ConfigError at load naming the duplicated job names. A job name must also differ from every DAG's schedule job name (dag:<dag name>). Other names beginning with dag: are valid. Used in logs, the stream prefix, reports, statsd, and the HTTP API. |
command |
Str or Seq(Str)
|
required | A shell command string (run through shell) or an argv list (run directly, no shell). The shell used for a string command is platform-specific: /bin/sh on POSIX, and cmd.exe through %ComSpec% on Windows when shell is left empty. An argv list bypasses the shell on every platform. See commands and environment and running on Windows. |
schedule |
Str or Map
|
required | A crontab string (5, 6, or 7 fields), an @yearly/@annually/@monthly/@weekly/@daily/@hourly nickname, the literal @reboot, or a mapping with second, minute, hour, dayOfMonth, month, year, dayOfWeek (each Str, all optional). The mapping is assembled into a crontab: 5 fields normally, 6 when year is set, 7 when second is set (second/year emitted only when used, the rest default to *). A second schedules at second granularity; year restricts to specific years. See schedules and timezones. |
shell |
Str |
/bin/sh (POSIX) / empty (Windows) |
Shell used to run a string command. Ignored when command is a list. The default is platform-specific: on POSIX a string command runs as ["/bin/sh", "-c", command]; on Windows the default is empty, which routes a string command through the Windows command processor %ComSpec% (cmd.exe) by using asyncio.create_subprocess_shell. For PowerShell or another interpreter set shell: explicitly, or pass command as a list to bypass the shell entirely (on every platform). The shell field itself works on all OSes. See running on Windows. |
enabled |
Bool |
true |
When false, the job is parsed and validated but never scheduled or runnable. |
| Option | Type | Default | Description |
|---|---|---|---|
captureStdout |
Bool |
false |
Capture the process's stdout for failure detection and reports. When false, the job's stdout passes through to cronstable's stdout. |
captureStderr |
Bool |
true |
Capture the process's stderr for failure detection and reports. When false, the job's stderr passes through to cronstable's stderr. |
saveLimit |
Int |
4096 |
Maximum number of captured lines retained per stream (split into the first half and the last half; lines in between are discarded and counted). 0 disables retention. |
maxLineLength |
Int |
16777216 (16 MiB) |
Maximum bytes buffered per line by the stream reader. A longer line is skipped with a warning. |
streamPrefix |
Str |
[{job_name} {stream_name}] |
Format string prefixed to every emitted output line. Supports {job_name} and {stream_name} placeholders; set to "" to disable. |
See output capturing for buffering, truncation, and the captured-output handoff to reporters.
| Option | Type | Default | Description |
|---|---|---|---|
utc |
Bool |
true |
When true, the schedule is interpreted in UTC. When false (and no timezone), local time is used. |
timezone |
Str |
none | IANA time zone name (for example, America/Los_Angeles) overriding utc. An unknown name raises a ConfigError. |
| Option | Type | Default | Description |
|---|---|---|---|
concurrencyPolicy |
Enum(["Allow", "Forbid", "Replace"]) |
Allow |
Behavior when a scheduled run overlaps a still-running instance. Allow: run concurrently. Forbid: skip the new run. Replace: cancel the running instance and start the new one. |
concurrencyScope |
Enum(["node", "cluster"]) |
node |
How far concurrencyPolicy reaches. node (default): an overlap is another instance in this cronstable process. cluster: Forbid/Replace also exclude the job's instances on other nodes sharing the state store, through a per-job TTL slot lease (slots/<job name>) there. It works with or without a cluster section. Without a state section somewhere in the assembled configuration, load fails with a ConfigError naming the offending job(s). cluster with concurrencyPolicy: Allow is likewise a ConfigError at load: Allow places no bound on concurrent instances, so widening its scope gates nothing. Enforcement is best-effort at-least-once, not exactly-once. state.slotTtlSeconds and state.onStoreUnavailable govern the edges. Part of the job-set ID only when set to cluster (existing configurations keep their digests; replicas disagreeing on it show as drift). See concurrency and timeouts. |
clusterPolicy |
Enum(["Leader", "PreferLeader", "EveryNode"]) |
Leader |
Where this job runs under cluster leader election. Inert unless cluster.electLeader is set (without election every job runs on every instance). Leader: only the quorum-gated leader runs it (at-most-once; may skip). PreferLeader: the lowest reachable agreeing node runs it, ignoring quorum (never skips; may double-run across a partition). EveryNode: every node runs it, independent of cluster health. Part of the job-set ID. See clustering and leader election. |
executionTimeout |
Float |
none | Seconds after which a still-running run is terminated. Unset means no timeout. Must be > 0 when set. Termination signals the job's whole process group/tree, so the timeout ends the descendants along with the root process. The timeout bounds a run until its captured output closes: a run that captures neither stream ends when its own process exits, and a process that it left behind is not signaled at the deadline (see execution timeout). killTimeout in the next row covers the platform-specific escalation. See running on Windows. |
killTimeout |
Float |
30 |
The longest a cancellation waits, after its graceful step, for the job's own process to exit before the forced step. Must be >= 0. The job's own process is the shell for a string command and the program for a list. The graceful step is a trappable signal to the job's process group: SIGTERM on POSIX and CTRL_BREAK_EVENT on Windows. The forced step (SIGKILL to the group on POSIX, a taskkill /F /T tree kill on Windows) follows when that process exits or killTimeout ends. On POSIX, a program that runs as a child of the shell therefore has the grace period only while the shell is alive. See cancellation and killTimeout and running on Windows. |
verify adds a post-command result check to jobs and executable DAG tasks.
Its required command is a string or sequence of strings; its optional
timeout is a finite positive float, defaulting to 60. The default
verify: null disables the check. See result verification.
The top-level pools map declares named pools. Each requires integer slots
and accepts integer maxQueued (default 1000); both range from 1 to 10000.
Pools require state. Names must be unique across the assembled configuration.
Jobs and executable DAG tasks accept pool (string, default null),
poolSlots (integer, default 1), queuePriority (integer, default 0), and
queueTimeout (finite positive float in seconds, default 3600). Slots must
be positive and fit the named pool. Nondefault queue options require a pool.
Approval gates reject pools. See resource pools.
failsWhen determines when a completed run is treated as a failure. In the
strictyaml schema only producesStdout is a required key inside failsWhen.
The others are optional. The DEFAULT_CONFIG defaults are:
| Option | Type | Default | Description |
|---|---|---|---|
failsWhen.producesStdout |
Bool |
false |
Fail if any stdout was captured. |
failsWhen.producesStderr |
Bool |
true |
Fail if any stderr was captured. |
failsWhen.nonzeroReturn |
Bool |
true |
Fail if the process exits with a nonzero return code. |
failsWhen.always |
Bool |
false |
Fail whenever the process exits, regardless of output or return code. |
jobs:
- name: example
command: echo "hi"
schedule: "* * * * *"
failsWhen:
producesStdout: false
producesStderr: true
nonzeroReturn: true
always: falseSee failure detection and retries.
Three lifecycle hooks each carry a report block (mail, sentry, shell,
webhook, push, eventlog); onFailure also carries a retry block. The report
blocks all share the same _report_schema and the same _REPORT_DEFAULTS
(deep-copied so the three blocks do not alias one another).
| Option | Type | Default | Description |
|---|---|---|---|
onFailure.retry.maximumRetries |
Int |
0 |
Max retry attempts after a failure. 0 disables retries; -1 retries forever. Must be >= -1. |
onFailure.retry.initialDelay |
Float |
1 |
Seconds before the first retry. Must be >= 0. |
onFailure.retry.maximumDelay |
Float |
300 |
Upper bound on the backoff delay. Must be > 0. |
onFailure.retry.backoffMultiplier |
Float |
2 |
Multiplier applied to the delay between retries (exponential backoff). Must be > 0. |
onFailure.report |
_report_schema (mail/sentry/shell/webhook/push/eventlog) |
defaults later | Reporters fired on every detected failure (including each failed attempt). |
onPermanentFailure.report |
_report_schema |
defaults later | Reporters fired only after all retries are exhausted. |
onSuccess.report |
_report_schema |
defaults later | Reporters fired on a successful run. |
When a retry block is present, the strictyaml schema requires all four keys
inside onFailure.retry: maximumRetries, initialDelay, maximumDelay, and
backoffMultiplier. See
failure detection and retries.
Reporting (mail, Sentry, shell, webhook) covers the report
blocks in full; their schema and _REPORT_DEFAULTS are summarized here.
from and to are required keys (each EmptyNone() | Str). Defaults from
_REPORT_DEFAULTS["mail"]:
| Option | Type | Default | Description |
|---|---|---|---|
from |
EmptyNone() | Str |
None |
Sender address (required key). |
to |
EmptyNone() | Str |
None |
Recipient address(es) (required key). |
smtpHost |
Str |
None |
SMTP server host. |
smtpPort |
Int |
25 |
SMTP server port. |
subject |
Str |
jinja2 default subject template | Email subject (jinja2). |
body |
Str |
jinja2 default body template | Email body (jinja2). |
username |
Str |
None |
SMTP login username (enables login with password). |
password |
Map with value/fromFile/fromEnvVar (each EmptyNone() | Str) |
all None
|
SMTP login password source. |
tls |
Bool |
false |
Use implicit TLS. |
starttls |
Bool |
false |
Use STARTTLS. |
validate_certs |
Bool |
true |
Validate TLS certificates. |
html |
Bool |
false |
Send the body as HTML. |
Defaults from _REPORT_DEFAULTS["sentry"]:
| Option | Type | Default | Description |
|---|---|---|---|
dsn |
Map with value/fromFile/fromEnvVar (each EmptyNone() | Str) |
all None
|
Sentry DSN source. |
fingerprint |
Seq(Str) |
["cronstable", "{{ environment.HOSTNAME }}", "{{ name }}"] |
Issue-grouping fingerprint (jinja2 per entry). Replaces, never appends, on merge. |
level |
Str |
unset (effective error) |
Sentry event level. When unset, events are captured at level error. |
extra |
MapPattern(Str, Str | Int | Bool) |
unset | Extra structured context. |
body |
Str |
default subject + body templates | Event message (jinja2). |
environment |
Str |
None |
Sentry environment. |
maxStringLength |
Int |
8192 |
Max string length before Sentry truncation. |
| Option | Type | Default | Description |
|---|---|---|---|
shell |
Str |
/bin/sh (POSIX) / empty (Windows) |
Shell used to run the reporter command. The default is platform-specific, same as the per-job shell field: on Windows the default is empty (the reporter command runs through %ComSpec%, which is cmd.exe). Set shell: explicitly for another interpreter, or pass command as a list. See running on Windows. |
command |
Str or Seq(Str)
|
None |
Reporter command (required key). Receives CRONSTABLE_* environment variables. |
timeout |
Float |
60 |
Hard bound, in seconds, on the reporter command; on expiry the daemon stops its whole process group. Reports run inline on the daemon's job-completion loop, so a reporter that never exits would otherwise stall completion handling for every job. |
Defaults from _REPORT_DEFAULTS["webhook"]:
| Option | Type | Default | Description |
|---|---|---|---|
url |
Map with value/fromFile/fromEnvVar (each EmptyNone() | Str) |
all None
|
Webhook URL source (treated as a secret; never logged). No URL means webhook reporting is disabled. |
method |
Str |
POST |
HTTP method. |
contentType |
Str |
application/json |
Content-Type header value. |
headers |
MapPattern(Str, Str) |
{} |
Extra request headers, sent verbatim (not templated). |
body |
Str |
default webhook body template | Request body (jinja2). The default is a Slack-compatible {"text": ...} JSON payload of the default subject + body text. |
timeout |
Float |
10 |
Total request timeout, seconds. |
The end-to-end encrypted push channel. Enabling it anywhere requires the
daemon-global push: section (relay + device registry) and the
push extra (PyNaCl); both requirements fail closed at config load. See
push notifications.
| Option | Type | Default | Description |
|---|---|---|---|
enabled |
Bool |
false |
Opt this job/event into the push channel. |
priority |
Enum of time-sensitive / passive
|
time-sensitive |
Relayed to APNs as the interruption level: time-sensitive breaks through scheduled summaries, passive does not. |
includeLogTail |
Bool |
true |
Carry the last captured output lines inside the sealed payload (trimmed oldest-first to fit the APNs size cap). |
Windows Event Log records, and a no-op on every other platform (the load warns once, naming each hook that enabled it). Needs no extra and no dependency. See Windows Event Log for the event-ID and insertion-string tables.
| Option | Type | Default | Description |
|---|---|---|---|
enabled |
Bool |
false |
Write Event Log records for this hook. |
source |
Str |
cronstable |
Event source name. Refused when empty, when it contains \ or /, or when it is one of the log names Application, System or Security; the check applies only to enabled blocks. |
includeOutput |
Bool |
false |
Carry a bounded output tail as the last insertion string. Off by default because the Application log is readable by every local account. |
Per-job thresholds for the runs that did not happen, evaluated once per
minute by the in-process monitor. A breach fires the onLate reporters once.
The full semantics live on late-run detection; this is
the schema summary.
| Option | Type | Default | Description |
|---|---|---|---|
sla.maxTimeSinceSuccessSeconds |
EmptyNone() | Int |
None (off) |
Breach when this many seconds pass without a successful finish. Must be > 0 when set. |
sla.lateAfterSeconds |
EmptyNone() | Int |
None (off) |
Breach when a due scheduled slot has not started a run within this many seconds. Must be > 0 when set. |
sla.maxRuntimeSeconds |
EmptyNone() | Int |
None (off) |
Breach while a running instance exceeds this runtime. Observes only, and never stops the run. Must be > 0 when set. |
onLate.report |
_report_schema |
overdue-variant defaults | Reporters fired once per latched breach. Same schema as the earlier hooks; the defaults swap in an overdue subject/body, a Slack-compatible overdue webhook body, and the sentry fingerprint ["cronstable", "sla", "{{ name }}"]. |
If all three sla keys are unset, configuring an onLate reporter (a mail
to/from, a sentry dsn, a shell command, a webhook url, or an enabled
push or eventlog block) is a load-time
ConfigError (onLate requires sla). Both keys merge normally under
defaults: and are excluded from the job-set ID fingerprint,
like the catch-up options. The monitor does not evaluate disabled and
paused jobs.
These options build on the top-level state store and only take
full effect when a state section is configured. Without one they parse and
validate normally but change nothing. The one exception is
onlyIfLastSucceeded, which also works without a state section from the
in-memory history alone (the gate then resets on restart). See
durable state for the full semantics.
| Option | Type | Default | Description |
|---|---|---|---|
onMissed |
Enum(["skip", "run-once", "run-all"]) |
skip |
Job catch-up after downtime or same-process sleep/stalls. Startup uses the durable last-run watermark; resume uses the skipped scheduling window. skip keeps classic cron behavior. run-once coalesces missed slots into one attempt (a normal start after resume also satisfies it). run-all replays each missed occurrence, bounded. Inert without a state section. |
startingDeadlineSeconds |
Int or null |
none | Only occurrences missed within this many seconds are caught up; unset means no deadline. Bounds run-all to a recent window so a long outage cannot stampede (like the Kubernetes CronJob field of the same name). Also invalidates a persisted retry sequence older than the deadline. Must be > 0 when set. Only meaningful with a state section. |
catchupJitterSeconds |
Int |
0 |
Spread catch-up launches of different jobs over [0, N) seconds, deterministic per job name, on restart or resume. 0 fires them together. Must be >= 0. Only meaningful with a state section. |
onlyIfLastSucceeded |
Bool |
false |
Depends-on-past gate: skip a scheduled fire when the job's most recent finished run did not succeed, or when a previous instance is still running or has yet to be recorded (unless concurrencyPolicy: Replace). The last real outcome is the newest success or failure in the in-memory history and the durable run ledger, judged by the instant each run ended, whatever order the daemon recorded the runs in. Canceled and skipped runs are ignored, and retries, catch-up backfills, and manual API triggers deliberately bypass the gate. Works without a state section from the in-memory history alone (resetting on restart); with one, the gate's memory survives restarts. See depends-on-past. |
archiveOutput |
Bool |
false |
Persist each finished run's captured output durably to the state store (the job's logs/ stream). Encryption at rest is the mount's job (EFS/S3 SSE, an encrypted volume). Inert without a state section (a startup warning notes it archives nothing). |
redactArchivedSecrets |
Bool |
true |
Scrub recognizable secrets (tokens, passwords, keys, auth URLs) from archived output before it is written. Applies only when archiveOutput is set, so it too has no effect without a state section. |
| Option | Type | Default | Description |
|---|---|---|---|
environment |
Seq(Map({"key": Str, "value": Str})) |
[] |
Environment variables set for the process. Both key and value are required per entry. Merged by key with defaults and with env_file (config values win). |
env_file |
Str |
none | Path to a KEY=VALUE file; blank lines and # comments are ignored. Variables in environment override file values. A read error or a line without = raises a ConfigError. |
workingDirectory |
Str or null |
none | Directory the job's process starts in, the equivalent of the "Start in" box on a Task Scheduler action. Unset inherits cronstable's own working directory; under a defaults: block that sets it, a bare workingDirectory: on a job opts that one job back out to inheriting. The daemon expands ~ and ${VAR} and makes the result absolute at load, so a relative value settles against cronstable's working directory once rather than per run. It deliberately does not check that the directory exists at load, because a load also happens on hosts that are not the target. The OS checks at spawn, and a missing directory records the run as a launch failure (exit 127) whose log line names it. Not part of the job-set ID. See commands and environment and running on Windows. |
secrets |
Seq(Map({"name": Str, "value"/"fromFile"/"fromEnvVar": Str})) |
[] |
Run-scoped secrets staged for the job over the job-facing state endpoint rather than placed in the environment, so they never show in /proc/<pid>/environ. Each needs a name and at least one source; when several are set, value wins over fromFile, which wins over fromEnvVar (a nameless or sourceless entry is a ConfigError; a same-named entry merges last-wins, like environment). A job reads one with cronstable secret get NAME. Requires a state section with jobApi enabled, else load fails naming the offending job(s). |
stateAllowedScopes |
Seq(Str) |
[] |
Extra scope names (besides the job's own name and global) this job's cronstable state|cursor|lock|artifact calls may explicitly name through --scope. Naming any other scope (most dangerously another job's own name, which IS that job's private scope) is refused (403). See scopes. |
jobs:
- name: example
command: env
schedule: "* * * * *"
env_file: .env
environment:
- key: PATH
value: /bin:/usr/bin| Option | Type | Default | Description |
|---|---|---|---|
user |
Str or Int
|
none | User name or numeric uid the process runs as. A numeric uid derives its primary gid and login name from the passwd database. An unknown name raises a ConfigError. |
group |
Str or Int
|
none | Group name or numeric gid the process runs as. If only user is set, the group defaults to that user's primary group. An unknown name raises a ConfigError. |
This section is POSIX-only (the setuid/setgid model). On POSIX, setting user
or group requires cronstable to run as root (euid 0); otherwise load raises a
ConfigError. Privilege switching is not supported on Windows: a job with
user or group set raises a configuration error, verbatim
Job <name>: changing user/group is not supported on Windows. See
production and container deployment and
running on Windows.
| Option | Type | Default | Description |
|---|---|---|---|
priority |
Enum(idle, below-normal, normal, above-normal, high) |
normal |
Scheduling priority of the job's process. normal is the one level that is never applied: on POSIX the job keeps cronstable's own nice, and on Windows it keeps cronstable's own class only when cronstable is at idle or below-normal, otherwise NORMAL. On Windows the level becomes the process's priority class at creation. On POSIX the daemon renices the job's process group right after the spawn, to an absolute value: idle 19, below-normal 10, above-normal -5, high -10. Descendants inherit lowered levels. Raised ones apply to the job's own process, because Windows resets an unflagged child of an above-normal or high parent to NORMAL. POSIX renices the whole group, so it has no such split. Any other value is a ConfigError listing the five that are accepted. Part of the job-set ID only when set. |
Raising a priority (any level whose nice sits below cronstable's own; from the
usual nice 0 that means above-normal and high) needs CAP_SYS_NICE or
RLIMIT_NICE headroom on POSIX. The daemon says so once, at config load,
naming the job. If the kernel then refuses the renice, cronstable does not
fail the run. The job's processes keep the nice value they inherited, and the
refusal is logged at DEBUG. Windows hands every one of these classes to an
unprivileged account, so nothing is refused there.
On Linux, a nice value orders a job only against the processes in its CPU
group. Under the packaged systemd unit, that group holds the system's
services. A unit with Delegate=yes makes cronstable a group of its own.
With the cpu controller delegated that way, each run is a group of its own
while a job has a resourceLimits.cpu cap, and its CPU weight follows its
priority, even when the kernel refuses the renice. That weight orders the
run only against cronstable's other runs, within cronstable's own share. See
priority and Linux CPU groups.
realtime is deliberately not on the list. On Windows that class outranks the
threads that service disk, keyboard, and mouse, so a runaway job at REALTIME
can put the host out of reach of the operator who has to stop it.
See commands and environment.
Running on Windows shows how the levels
line up with Task Scheduler's -Priority numbers.
| Option | Type | Default | Description |
|---|---|---|---|
resourceLimits |
Map |
none | Per-run caps on CPU, memory, and process count, enforced by the kernel through a cgroup per run on Linux (cgroup v2) and a Job Object per run on Windows. Takes the four keys in the following rows. Merges key by key under defaults:. A map that sets no cap, or sets every cap to max, means no caps. Part of the job-set ID only when set, with normalized values. See resource limits. |
resourceLimits.cpu |
Str (a number, or max) |
none | CPU cap in CPUs, like docker run --cpus: 1.5 allows one and a half CPUs' worth of CPU time on any number of cores. Python float() syntax with ASCII digits, from 0.01 to 1000000. A run at the cap is throttled and does not fail for it. |
resourceLimits.memory |
Str (a size, or max) |
none | Memory cap. A byte count, or a number with a B, K, M, G, or T unit; K, M, G, and T can be followed by i, B, or iB. Units are case-insensitive powers of 1024, such as 512M or 1.5GiB. At least 1 MiB and below 8 EiB. Linux counts cgroup memory and Windows the commit charge. A run that goes over the cap is killed with all its processes and fails with memory limit exceeded (<cap>), whatever failsWhen says. |
resourceLimits.processes |
Str (a whole number, or max) |
none | The most processes the run can have at once, from 1 to 4194304. On Linux, threads count too. |
resourceLimits.required |
Bool |
false |
Fail a run whose caps the host cannot enforce, with resource limits could not be enforced: <reason> (resourceLimits.required is set), instead of running it uncapped. |
Every cap accepts max in any letter case, which also lifts a cap inherited
from defaults:. A bare resourceLimits: cannot clear an inherited map. The
schema reads each cap as text, so a ${VAR} value follows the same rules as a
literal; required is a boolean and takes no ${VAR}. An unreadable value
is a ConfigError that names the key, such as
Job <name>: resourceLimits.memory: cannot read '512X' as a size; ....
Approval gates reject resourceLimits and ignore one inherited from
defaults:.
On Linux the caps need cgroup v2 and a cgroup delegated to cronstable; the
packaged systemd unit sets Delegate=memory pids and OOMPolicy=continue,
and CPU caps need Delegate=yes in a drop-in. Windows needs no setup. macOS,
the BSDs, and illumos cannot enforce caps, and each config load there logs
one warning that names the capped jobs. Where the host cannot enforce a cap,
the run goes ahead without it, its record says so, and the daemon logs one
warning for each change. Full feature guide:
resource limits.
| Option | Type | Default | Description |
|---|---|---|---|
statsd |
Map({"prefix": Str, "host": Str, "port": Int}) |
none | When set, emit start/stop/success/duration metrics over UDP. All three keys are required. |
monitorResources |
Bool or Map
|
false |
Sample each run's CPU time and peak resident memory (RSS) by polling the job's process tree while it runs. Observability only: the numbers travel with the run record into the dashboard, GET /metrics, and statsd, but never change a run's success/failure verdict. Off by default, and a per-instance sampling task is spawned only when it is on. Set it under defaults: to enable it fleet-wide. The map form tunes it: enabled (Bool, default true), interval (Float seconds between samples, default 1.0), history (Int chart-series points kept per run, default 240; 0 keeps summary numbers only). Full feature guide: resource monitoring. |
See metrics with statsd. Prometheus metrics are not
configured per job: the GET /metrics endpoint is global, tuned under
web.metrics in the earlier web section. See
metrics with Prometheus.
Resource accounting (monitorResources). With it on, a run is sampled by
psutil (a core dependency) over its
whole process tree, so a job that launches child processes through a shell is
accounted too.
The result (total user/system CPU seconds and the peak RSS observed) appears in
the dashboard run history and stats, in the durable run record's resources
object (so it survives a restart), and as the metrics listed in
metrics with Prometheus. Report templates and the
shell reporter also receive cpu_seconds / max_rss_bytes
(CRONSTABLE_CPU_SECONDS / CRONSTABLE_MAX_RSS_BYTES). This section covers
the configuration surface; the full feature guide is
resource monitoring.
DAG tasks accept monitorResources too, but surface the result differently:
a finished task instance's usage is recorded in the resources object of its
task record inside the durable dag_run document (returned by
GET /dags/<name>/runs/<run_key>), and sent to the task's statsd sink when
one is configured. DAG task instances are ephemeral and do not appear in the
per-job Prometheus families on GET /metrics.
The numbers are sampled, so a run that finishes between two samples is measured approximately. The long, heavy runs whose resource use matters are sampled many times. It is best-effort: if psutil cannot read a process (already exited, permission denied), the run carries no resource stats, and monitoring never fails a job.
Sampling cadence and chart series. The map form controls how monitoring behaves:
monitorResources:
interval: 0.5 # seconds between samples (default 1.0, minimum 0.1)
history: 240 # chart points kept per run (default 240; 0 = summary only)interval sets the process-tree polling cadence: shorter intervals catch
sharper RSS spikes at the cost of more wakeups (each sample walks the process
table, hence the 0.1s floor).
Alongside the summary numbers, each monitored run records a downsampled
CPU%/RSS time series for the dashboard's Resources tab: one point per
sample until history points accumulate, after which adjacent points merge
(mean CPU%, peak RSS; spikes are never averaged away) and the effective
resolution halves, so a run of any length stays within history points (at
most 2000).
The series is embedded in the durable run record's resources.series, so
charts survive restarts and are bounded by the same state.maxRunsPerJob
retention as run records. It is served by GET /jobs/<name>/resources and
deliberately excluded from the polled /jobs and /jobs/<name>/runs payloads.
The dashboard's node-level CPU/memory history chart is configured separately,
under web.nodeHistory (see the web section): interval (Float seconds,
default 5.0, minimum 1.0) and points (Int ring size, default 720, an
hour at the default cadence), or nodeHistory: false to disable the background
node sampler entirely.
strictyaml enforces only the type (Int/Float). After type validation,
JobConfig._validate_numeric_ranges enforces value ranges (plus one
cross-field rule) and raises a ConfigError (prefixed Job <name>:) on
violation. JobConfig._validate_resource_limits checks the resourceLimits
rows the same way after it reads each cap from its text. These checks run at
load time, not at run time.
| Rule | Condition |
|---|---|
saveLimit >= 0 |
always |
maxLineLength > 0 |
always |
killTimeout >= 0 |
always |
concurrencyPolicy is Forbid or Replace
|
only when concurrencyScope: cluster (widening Allow gates nothing, so it is rejected rather than ignored) |
executionTimeout > 0 |
only when executionTimeout is set |
catchupJitterSeconds >= 0 |
always |
startingDeadlineSeconds > 0 |
only when startingDeadlineSeconds is set |
onFailure.retry.maximumRetries >= -1 |
only when a retry block is present |
onFailure.retry.initialDelay >= 0 |
only when a retry block is present |
onFailure.retry.maximumDelay > 0 |
only when a retry block is present |
onFailure.retry.backoffMultiplier > 0 |
only when a retry block is present |
sla.maxTimeSinceSuccessSeconds > 0, sla.lateAfterSeconds > 0, sla.maxRuntimeSeconds > 0
|
each only when set |
an onLate reporter requires an sla threshold |
when a reporter is configured (a mail to/from, a sentry dsn, a shell command, a webhook url, or an enabled push or eventlog block) while all three sla keys are unset |
monitorResources.interval >= 0.1 |
always (a sub-100ms cadence would busy-loop the process-table walk) |
0 <= monitorResources.history <= 2000 |
always (bounds what one run adds to a durable ledger record) |
0.01 <= resourceLimits.cpu <= 1000000 |
only when cpu is a number (the floor is the kernel's smallest cpu.max quota, 1 ms per 100 ms period) |
1 MiB <= resourceLimits.memory < 8 EiB |
only when memory is a size (the ceiling keeps the cap inside a signed 64-bit field) |
1 <= resourceLimits.processes <= 4194304 |
only when processes is a number (4194304 is the largest pids.max a 64-bit Linux kernel accepts; a 32-bit kernel accepts up to 32768) |
defaults:
shell: /bin/bash # POSIX path; on Windows omit shell (uses cmd.exe) or set a Windows interpreter
utc: false
jobs:
- name: nightly-backup
command: /usr/local/bin/backup.sh
schedule: "0 3 * * *"
captureStdout: true
captureStderr: true
executionTimeout: 3600
onFailure:
retry:
maximumRetries: 3
initialDelay: 5
maximumDelay: 60
backoffMultiplier: 2