Use the optional dags: section to define workflows whose tasks run in dependency order. Each workflow is a DAG (directed acyclic graph) backed by durable state, allowing it to resume after restarts and coordinate across a fleet. DAGs use the existing job runner and state store:

• a dag_run (one execution of a DAG) is a single mutable document in the state store, holding every task's state;
• a task is an ordinary job invocation: the same command/shell/env/ timeout machinery, launched the same way, with the same loopback state endpoint injected, so a task can call cronstable xcom|artifact|state|lock|...;
• cross-task data (XCom, cross-communication) rides the artifact store, scoped per dag_run;
• the scheduler advances each run under a single lease, so a healthy fleet does not launch a task twice or double-advance a run (crash recovery is at-least-once; see crash-resume and the fleet).

Opt-in and store-backed. DAGs require a state section with the loopback endpoint (state.jobApi.enabled, on by default). Without dags: none of this exists. Adding it leaves plain scheduled jobs unchanged.

On this page: A first DAG · Tasks and dependencies · The task state machine · XCom: passing data between tasks · Fan-out: dynamic mapping · Sensors · Approval gates · Scheduling, catch-up, and backfill · Crash-resume and the fleet · Retention and GC · Inspecting and controlling runs

state:
  path: /var/lib/cronstable/state      # DAGs need a state store + jobApi
dags:
  - name: nightly-etl
    schedule: "0 2 * * *"
    tasks:
      - id: extract
        command: "echo '[1,2,3]' | cronstable xcom push --key ids"
      - id: transform
        dependsOn:
          - extract
        command: "cronstable xcom pull --task extract --key ids"
      - id: load
        dependsOn:
          - transform
        command: "echo loading"

At 02:00 the daemon creates a dag_run and advances it: extract runs first, then transform (after extract succeeds), then load. Every transition is durable, so a restart resumes the run from exactly where it was.

Each task has an id (unique within the DAG) and, except for an approval gate, a command (the same string-or-list command a job takes). You declare edges with dependsOn:, a list of upstream task ids. The graph must be acyclic and every dependency must resolve. A cycle or a dangling edge is a configuration error at load, never a run that stops responding at runtime.

A task's triggerRule governs its readiness:

Per-task launch fields mirror a job: shell, environment, captureStdout / captureStderr, executionTimeout, killTimeout, user / group, workingDirectory, priority, failsWhen, run-scoped secrets, monitorResources, resourceLimits, and the rest of the shared launch keys. The configuration reference has the complete list, with types and defaults.

Because a task is a job invocation, its launch fields inherit the file's defaults: block the same way a job does: a global shell, environment, monitorResources, run-scoped secrets, or reporter block covers DAG tasks too, and the task's own value wins on any key it sets.

A task's onFailure / onSuccess reporters (set per-task or inherited) fire on each of its runs, every failed attempt included. Per-task the two hooks accept a report block only, because a task's retries come from the node's retries field, not a job-level onFailure.retry policy (an inherited one is ignored for tasks).

Only the launch fields inherit. The DAG-node fields that shape the graph (dependsOn, triggerRule, retries, retryDelaySeconds, expand, onReject, the poke settings) are never touched by a defaults: block.

The DAG's own schedule frame is separate too: the synthetic trigger job that fires the DAG on schedule stays on the built-in defaults, so a global onSuccess/onFailure reporter does not alert on every DAG tick.

A monitored task instance's sampled CPU time and peak resident set size (RSS) land in the resources object of its task record in the dag_run document, and in the task's statsd sink if one is configured. Task instances do not appear in the per-job Prometheus families.

Each instance of a task with resourceLimits runs in a cgroup or Job Object of its own under the task's caps, and the instance's limits record lands in its task record in the dag_run document. An approval gate runs no process: setting resourceLimits on one is a configuration error, and a gate ignores caps from defaults:.

Per-task retries are DAG-owned (independent of a job's onFailure.retry):

      - id: load
        command: "..."
        retries: 3                # up to 3 retries -> 4 attempts
        retryDelaySeconds: 30     # wait between attempts

Each task instance moves through:

pending ─▶ running ─▶ success
                   ├▶ up_for_retry ─▶ running (after retryDelaySeconds)
                   └▶ failed              (retries exhausted)
pending ─▶ upstream_failed   (an upstream failed, all_success)
pending ─▶ skipped           (an upstream was skipped, all_success)

A dag_run is success after every task is terminal and none failed. It is failed if any task ended failed or upstream_failed. skipped is not a failure.

A task publishes a small output under a key. A downstream task reads it. XCom is a thin, task-keyed convention over the artifact store, scoped to the dag_run, driven by the cronstable xcom CLI the daemon makes reachable in every task:

# in an upstream task:
echo '{"rows": 42}' | cronstable xcom push --key summary
cronstable xcom push --key summary producer_output_file.json    # or from a file

# in a downstream task:
cronstable xcom pull --task upstream_id --key summary            # -> stdout
cronstable xcom pull --task upstream_id --key summary -o out.json
cronstable xcom list                                            # keys in this run

Outputs are content-addressed and versioned (newest wins by key). The daemon injects the run's identity so the CLI needs no arguments beyond the key: CRONSTABLE_DAG_NAME, CRONSTABLE_DAG_RUN_ID, CRONSTABLE_DAG_RUN_KEY, CRONSTABLE_DAG_TASK, CRONSTABLE_DAG_TASKKEY, CRONSTABLE_DAG_MAP_INDEX, CRONSTABLE_DAG_MAP_ITEM, CRONSTABLE_DAG_XCOM_SCOPE.

A task can expand into N parallel instances, one per item of an upstream's XCom list (Airflow's .expand()):

      - id: list-work
        command: "echo '[\"a\",\"b\",\"c\"]' | cronstable xcom push --key items"
      - id: process
        dependsOn:
          - list-work
        expand:
          fromTask: list-work      # a direct, non-mapped dependency
          key: items               # its XCom list
        command: "echo processing $CRONSTABLE_DAG_MAP_ITEM (#$CRONSTABLE_DAG_MAP_INDEX)"

When list-work succeeds, the scheduler reads its items list and materializes process#0, process#1, process#2, each with its own state, retries, and XCom, and its item in $CRONSTABLE_DAG_MAP_ITEM. A downstream task that lists process in dependsOn waits for all the mapped instances (fan-in). An empty list resolves the mapped task to success immediately.

The expanded item set is recorded once in the dag_run and never recomputed, so a crash-resumed run reconstructs the identical set of mapped instances rather than re-deriving it from a possibly-changed upstream output.

Because the expansion is permanent, the read that derives it is strict about store trouble. A store that cannot answer (an I/O error or timeout on a shared mount, a record only a newer node's schema can read) leaves the fan-out unknown: the task stays unexpanded and the scheduler retries the read on a later pass, regardless of onStoreUnavailable. This deliberately overrides the store's usual skip-on-read-error rule: one blip must not freeze the task into a permanently empty, vacuously successful fan-out.

The empty fan-out is reserved for a definitive answer:

• the upstream finished without publishing the key;
• the published value is not a usable JSON list (invalid JSON, not a list, or carrying a non-portable value); or
• the record survives but its payload blob is gone (410), possible only through external interference with the store, such as a partial restore, because GC never sweeps a blob a surviving record references.

A warning names each mapping-to-empty that indicates a problem.

A version that a peer's publish removed from the scope is not store trouble for this read. The read lists the scope again, up to three more times, when a version that it listed has left the store. For that cause the task stays unexpanded only when versions keep leaving during every one of those reads, and the scheduler then logs dag <name>: xcom '<key>' from '<task>' could not be read (stream '<stream>': records kept leaving during the read); leaving the task unexpanded to retry. On a mount whose client serves directory reads from a cache, the scope's directory read comes back with the same entries, and the read answers from it: the task expands from the version of the key that the read found, or to an empty fan-out when the read found no version of the key. On such a mount an empty fan-out can therefore also come from a cached directory read that lacks the key's current version. See artifact store for these reads.

A fan-out is capped at 1000 items. A larger XCom list fails the mapped task with an explanatory reason instead of materializing that many instances (its all_success downstream sees upstream_failed). A single scheduler pass also launches at most 32 instances at a time, so a large fan-out ramps up in bounded bursts rather than one burst of subprocesses.

A type: sensor task polls an external condition on a bounded, jittered, durable schedule instead of running once. Its command's exit code is the verdict: 0 = condition met (the task succeeds); nonzero = not yet, poke again after pokeIntervalSeconds plus a random 0 to pokeJitterSeconds until pokeTimeoutSeconds elapses, after which the sensor fails.

      - id: wait-for-file
        type: sensor
        command: "test -f /data/$(date +%F).ready"
        pokeIntervalSeconds: 60
        pokeTimeoutSeconds: 7200
        pokeJitterSeconds: 10

The poke schedule (nextPokeAt, pokeCount) is durable, so a restart resumes polling on time rather than restarting the timeout window.

A type: approval task blocks the graph until a human or an API call decides it. It runs no command. Approve or reject it over the control API (or the dashboard):

curl -X POST .../dags/nightly-etl/runs/<run_key>/tasks/publish-gate/decision \
     -H 'Content-Type: application/json' \
     -d '{"decision": "approve", "by": "alice"}'

approve succeeds the gate and the graph proceeds. reject fails it (or, with onReject: skip, marks it skipped, cascading skipped to its all_success downstream). The decision (by, timestamp) is recorded durably.

You can be paged when a gate begins waiting: configure the notify: block with the approval_waiting event. The daemon then fires a reporter (webhook, mail, …) the first time each gate parks awaiting a decision. A whole DAG run reaching failed similarly fires the dag_failure event.

A scheduled DAG reuses the job schedule grammar with one restriction: the schedule must parse to a cron expression, so @reboot is rejected at config load (dag 'd': schedule '@reboot' is not a cron expression; DAG schedules must be cron expressions (@reboot is not supported for dags)), although @daily / @hourly-style aliases still work.

A scheduled DAG follows the catch-up discipline: onMissed (skip / run-once / run-all) and startingDeadlineSeconds bound how many missed logical dates a restart replays, capped like a job's catch-up.

catchupJitterSeconds spreads the replays, with the same checkpointed at-least-once resume the job engine has. The owed watermark goes into a catchup-dag/<dag> stream (the twin of the job's catchup/<job>) before the jitter offset starts, so a restart during the offset resumes the backfill rather than losing it. A DAG with no schedule is manual-only.

Backfill replays a DAG across a historical range on demand. It is a deliberate operation that ignores the automatic deadline but is still bounded and idempotent (each date's run is create-if-absent, so re-running a backfill never duplicates runs):

curl -X POST .../dags/nightly-etl/backfill \
     -d '{"from": "2026-01-01T00:00:00+00:00", "to": "2026-01-07T00:00:00+00:00"}'

While the state store cannot answer, the request answers 503 with backfill state is unavailable. The runs written before a fault stay, and the same request creates the rest. See POST /dags/{name}/backfill for the cases.

The durable per-task state, not memory, is the source of truth. A dag_run is advanced only by the node holding that run's advance lease, a time-to-live (TTL) lease on the shared store renewed while the run is active, so in a healthy fleet only one node advances a given run and a task does not launch twice (crash recovery is at-least-once; see the delivery contract below). The claim that flips a task pending → running is a single atomic compare-and-set on the run document, a correctness backstop underneath the lease.

The owner also keeps an in-memory record of every task instance it claimed and was handed to launch. A store write that lands after the owner's own timeout stopped waiting for it (a stalled disk or network mount) leaves the task claimed with nothing launched. On its next pass the owner recognizes that claim, because the record has no entry for it, releases the task back to pending (a sensor back to its idle shape between pokes) and re-claims it at once. Nothing ran, so the task's attempt count is unchanged. A launch the owner cancels before its subprocess starts (a state section reload while the task waits behind the spawn gate) is released the same way. So is the launch of a task with resourceLimits that is canceled while its process joins its cgroup or Job Object, once cronstable has killed that process. Either way, a pooled task gives back its pool slot. The record forgets a run once the store shows it finished or collected, whichever node finished it.

If a node crashes, its lease lapses and a peer adopts the run, reconciling from the durable state:

• a task recorded running whose process is gone (a dead pid, or a foreign owner proven dead by the lease lapse) is retried if attempts remain, else failed;
• a sensor mid-poke is re-poked;
• an approval gate keeps waiting.

This mirrors the job-level crash reconciliation seam. Like every cronstable coordination primitive it is at-least-once, not exactly-once: a task whose process outlives a crashed daemon may run again on resume. A task that must be exactly-once should guard its side effect with an idempotency key.

A dag_run document is durable and, while its DAG is configured, is not swept by the record garbage collector (GC). Instead each DAG keeps its newest retainRuns terminal runs (default 50) and prunes the rest, along with their XCom, on a periodic DAG-owned pass.

Each node runs the pass hourly and at daemon start. It deletes runs in batches of up to 32, each run under its own lease, and keeps a run that a recovery references. The pass deletes a run only while the document is still the finished run that it listed, which it identifies by run ID and creation time. A run that a backfill creates again under the key of a deleted run therefore stays, and an adopt scan picks it up. A recovery run that is accepted again after another node's pass deleted it has the same key and run ID and a later creation time, so the pass keeps it and its XCom. When the pass finds another run under a key that it listed, this node's GET /dags/{name}/runs, the latestRun and runCounts of GET /dags, and the MCP run listings show the run that the key holds on their next read.

A batch starts a run's delete within 30 seconds of leasing the run and leaves the runs it cannot reach for the next pass. On a rate-limited or slow store, a busy DAG can therefore hold more than retainRuns terminal runs. The node creates no scheduled run while the pass is in flight. See rate limiting for the figures.

A delete that times out or is canceled can still land, so the pass keeps that run's lease until it lapses, 60 seconds after it was taken, and releases the other leases of the batch. Until the lease lapses, a recovery execution that names the run answers 409 with source run is busy; retry shortly, and no node can take the advance lease of that run key.

A DAG removed from every config ages out like a removed job. After it has been absent from every config and recent manifest for a full state.gcGraceSeconds, the daemon's GC pass deletes its terminal run documents (an active run is never touched, so a re-added DAG resumes it) and its aged XCom streams.

Artifact payload blobs are content-addressed. A blob any surviving record still references is never swept, so a retained run's XCom can never dangle. Only blobs no surviving record references, and older than the grace, are reclaimed.

Over the HTTP control API:

• GET /dags: the configured DAGs and their tasks
• GET /dags/{name}/runs: recent runs and their per-task state counts
• GET /dags/{name}/runs/{run_key}: one run's full document
• POST /dags/{name}/trigger: start a manual run now
• POST /dags/{name}/backfill: replay a date range
• POST /dags/{name}/runs/{run_key}/tasks/{taskkey}/decision: approve/reject a gate

The web dashboard drives the same endpoints from a DAG orchestration UI: a DAG card and a per-DAG drawer (runs, tasks, graph, XCom, logs) with trigger, backfill, and approval decisions. Its page documents that UI.

See example/dag/ for a complete configuration exercising every node type.