concurrencyPolicy controls overlapping runs of the same job. executionTimeout limits each run's duration, and killTimeout sets the longest that cancellation waits for the job's own process before forcing termination. These settings apply separately to each job and can also be set in defaults. They do not coordinate different jobs.

By default, the concurrency policy applies within one daemon process. concurrencyScope can extend Forbid and Replace to all nodes sharing a durable state store.

On this page: Overview · Option summary · Concurrency policy · Concurrency across a cluster · Execution timeout · Cancellation and killTimeout · Scope and interaction

A job is identified by its name. The daemon tracks, per name, a list of currently-running instances. When a scheduled time arrives (or a manual start is requested through the HTTP control API), the daemon checks whether any instance of that job is already running and consults concurrencyPolicy before launching a new one.

That tracking is local to one daemon process. By default (concurrencyScope: node) the local list is all that counts. concurrencyScope: cluster makes Forbid and Replace additionally consult a slot lease in the shared state store, extending their reach to instances on other nodes (see concurrency across a cluster). The local check always runs first, and the cluster gate is additive.

Independently, each running instance carries a deadline derived from executionTimeout. On expiry it is cancelled, and killTimeout controls the SIGTERM-then-SIGKILL escalation used during any cancellation. Both signals go to the job's whole process group, and each job is spawned in its own session, so a cancellation takes down the command's descendants too, not only the process cronstable spawned.

The SIGTERM-then-SIGKILL escalation is the POSIX spelling. On Windows the same two-step runs with the platform's own primitives: the graceful step delivers a trappable CTRL_BREAK_EVENT to the job's process group (each job is spawned in its own group), and the forced step kills the live process tree with taskkill /F /T. killTimeout bounds the wait between the two. See running on Windows, including the one degradation (a daemon with no console cannot deliver the break, so the graceful step becomes the tree kill immediately).

A run of a job with resource limits that is inside its own cgroup (Linux) or Job Object (Windows) takes the same graceful step. Its forced step kills that container, which also reaches descendants that left the process group or the process tree. On Linux the group SIGKILL follows the cgroup kill, so a process that left the cgroup but kept the group ends too. On Windows the Job Object kill takes the place of the tree kill.

Types come from the strictyaml schema: concurrencyPolicy is Enum(["Allow", "Forbid", "Replace"]), concurrencyScope is Enum(["node", "cluster"]), and executionTimeout and killTimeout are Float(). Defaults come from DEFAULT_CONFIG. All four options are optional (Opt(...) in the schema).

Numeric ranges are enforced after parsing: killTimeout >= 0 and, when set, executionTimeout > 0. A violating value raises a ConfigError at config load. Two concurrencyScope: cluster combinations are likewise refused at load rather than left silently inert. See concurrency across a cluster.

See the configuration reference for where these options sit in the document and how defaults apply.

When maybe_launch_job runs for a job and one or more instances of that name are already running in this process, it logs a warning (Job <name>: still running and concurrencyPolicy is <policy>) and then acts according to concurrencyPolicy. This local check always runs first. For a concurrencyScope: cluster job, a launch that clears it must then also claim the job's cluster slot (see concurrency across a cluster).

An instance counts as running until the daemon has collected its command's exit status and captured output and has finished any result verification, or until the daemon has finished canceling it. The instance has then ended, and an instance whose command could not start has ended as well. The daemon reads captured output to the end of the stream, so a process that the command started and that holds the stream open keeps the instance running after the command exits, until the stream closes or the instance is canceled. An instance that has ended does not count, even before the daemon has recorded the run. Every policy passes such an instance over, and the run keeps the outcome that it ended with. Until the run is recorded, GET /jobs reports the job as running and keeps the instance's process ID in pids.

The new instance is started immediately alongside the existing one(s). Multiple instances of the same job can run concurrently with no bound on their number. Each instance is tracked and reaped independently.

jobs:
  - name: ingest
    command: ./ingest.sh
    schedule: "* * * * *"
    concurrencyPolicy: Allow

If any instance is still running, the new launch is skipped entirely. No new process is started. The already-running instance continues unaffected. This applies equally to scheduled launches and to retry-triggered launches. A scheduled fire of a job with no pool that is dropped this way leaves the job's retry sequence as it is. The running instance keeps its retries and schedules the next one when it fails, a retry that is already pending stays pending, and the fire creates no retry state. This holds whichever launch started the running instance: a scheduled fire, a retry attempt, a manual start, or a catch-up run. The instances considered are this process's own. With concurrencyScope: cluster, an instance running on another node that shares the state store also forbids the launch (see concurrency across a cluster).

jobs:
  - name: ingest
    command: ./ingest.sh
    schedule: "* * * * *"
    concurrencyPolicy: Forbid

Every currently-running instance of the job is cancelled, then a new instance is started. Before canceling, the scheduler sets replaced = True on each outgoing instance. This flag changes how the finished run is reaped:

• The replaced run is not treated as a failure: _handle_finished_job returns early when replaced is set, logging Job <name> was replaced by a newer instance.
• Because it is not a failure, it is not reported (no Mail/Sentry/Shell/Webhook reporters fire for it) and it does not trigger retries. cancel() itself does not set a return code, so the run's own wait() task records whatever it happened to see: the signal-derived code, or -100 had its own executionTimeout expired first. That value is irrelevant: the reaper short-circuits on replaced before inspecting it.

Cancellation of the outgoing instance uses the same process-group SIGTERM/killTimeout/SIGKILL sequence described under cancellation and killTimeout. maybe_launch_job awaits each cancel() before starting the replacement, so the replacement starts only after the old instance has terminated.

This inline cancel-then-launch applies to instances running in this process. With concurrencyScope: cluster, replacing an instance on another node means asking that node to cancel it, and the replacement launch waits until that node yields. See replace across the cluster.

jobs:
  - name: sync
    command: ./sync.sh
    schedule: "* * * * *"
    concurrencyPolicy: Replace
    killTimeout: 10

concurrencyPolicy on its own reaches only the instances tracked by one daemon process. A fleet of nodes sharing a durable state store can widen Forbid and Replace to the whole fleet with the per-job concurrencyScope option (settable in defaults; see the configuration reference):

state:
  path: /mnt/shared/cronstable-state   # the same store on every node
  topology: shared

jobs:
  - name: ingest
    command: ./ingest.sh
    schedule: "* * * * *"
    concurrencyPolicy: Forbid
    concurrencyScope: cluster

concurrencyScope: node (the default) is the classic behavior described earlier: only this process's running instances are considered. concurrencyScope: cluster makes Forbid and Replace also exclude instances of the job on other nodes sharing the state store. The local check still runs first and is unchanged, and the cluster gate is additive. It works with or without a cluster: section: the shared store, not leader election, is what coordinates the nodes.

Two combinations are refused at config load rather than left silently inert:

• concurrencyScope: cluster requires a state section somewhere in the final assembled config (a config directory may keep state and jobs in different files). Without one, parsing fails with concurrencyScope: cluster requires a `state` section (the shared store is what coordinates the nodes), but none is configured; offending job(s): ..., naming every offending job.
• concurrencyScope: cluster with concurrencyPolicy: Allow raises Job <name>: concurrencyScope: cluster has no effect with concurrencyPolicy: Allow (the default); set Forbid or Replace, or drop concurrencyScope. Allow places no bound on concurrent instances, so there is nothing for the cluster to gate.

Every launch of a cluster-scoped job (scheduled, retry, catch-up backfill, deferred @reboot, or a manual API start) first claims a TTL slot lease named slots/<job name> in the state store (the cron state store, not the leadership store). The claim is a single choke point in maybe_launch_job, and each store operation in it is bounded (10 seconds), so an unresponsive mount cannot stall the scheduler pass.

• The lease TTL is state.slotTtlSeconds (default 30; a value below 5 raises state.slotTtlSeconds must be >= 5). While the job runs here, the holder renews the lease every third of the TTL. A node that crashes mid-run stops renewing, and its slot frees itself after at most one TTL.
• Lease operations bypass the state.maxOpsPerSecond token bucket: a renew queued behind bulk writes could overshoot its TTL and double-run the job the lease fences.
• The slot is released when the job's last local instance finishes (claims are refcounted, so overlapping instances share one lease). If the release fails, a state: failed to release the concurrency slot for <name> (...); it frees by TTL message is logged. TTL expiry is always the fallback.
• The lease is held under a process-unique identity (log messages show the node's display name), so a restarted daemon can never adopt its predecessor's slot.
• A state-section reload that names the same store (the same path and deploymentId as the backend resolves them, so a respelled path or an explicit deploymentId: default is the same store) rebuilds the backend under the live run and keeps its slot lease and renewer (a changed slotTtlSeconds applies from the next renew). A reload that moves the store leaves the lease behind to lapse by TTL and logs Job <name>: its cluster concurrency slot stays in the previous state store while the run continues here; a peer may launch it once that lease expires; the next launch claims in the new store.

Manual starts through the HTTP control API go through the same gate, consistent with the local behavior: a manual start was already subject to a node-local Forbid.

If a live instance on another node holds the slot, the launch is skipped with a warning naming the holder: Job <name> skipped: its cluster concurrency slot is held by <node> (concurrencyPolicy: Forbid, concurrencyScope: cluster). Nothing further happens for that occurrence, exactly like a local Forbid skip.

Replace cannot signal a process on another machine, so the requester asks the holder to yield instead of canceling it directly:

1. The requester appends an immutable cancel record to the slots/<job name> stream, targeted at the holder's exact lease fence, and logs Job <name>: cluster Replace: asking the current slot holder (<node>) to yield; the launch is re-attempted when the slot frees. Fence targeting makes a stale request inert: a takeover always bumps the fence.
2. The holder's renew task observes the cancel within about a third of the slot TTL and logs Job <name>: node <host> requested this instance be replaced (concurrencyPolicy: Replace, concurrencyScope: cluster); cancelling. Its instances are then marked replaced (the same not-a-failure treatment as a local Replace: no reports, no retries) and cancelled in the background, and the finish path releases the slot. The holder keeps renewing the lease while the old instance drains (up to killTimeout), so the slot frees on release rather than by expiry. A drain longer than about twice slotTtlSeconds outlasts the pursuit, which then skips the launch (no-run over double-run), so raise slotTtlSeconds for jobs with a large killTimeout.
3. The requester waits in a background pursuit task, never inline on the scheduler pass: waiting a holder out takes up to two slot TTLs and would stall every other due job. When the slot frees, by release or TTL expiry, it re-attempts the launch through every normal gate and logs Job <name>: launched after the previous cluster slot holder yielded (concurrencyPolicy: Replace) on success.
4. The pursuit is bounded at twice the slot TTL. If the holder never yields, the requester abandons this launch: Job <name>: the foreign holder (<node>) did not yield its cluster concurrency slot within <N>s; skipping this launch (no-run over double-run).

A store that is down, unresponsive, or whose denied claim cannot even be confirmed by a follow-up read leaves the gate unanswerable, and state.onStoreUnavailable decides:

• degrade (the default) launches anyway, enforcing concurrencyPolicy on this node only for that run: Job <name>: cannot claim its cluster concurrency slot (...); enforcing concurrencyPolicy on this node only for this run (onStoreUnavailable: degrade).
• fail-closed skips the launch: Job <name> skipped: cannot claim its cluster concurrency slot (...) and onStoreUnavailable is fail-closed.

A lock-fidelity probe, run once per backend and latched, catches a store whose file locks are demonstrably no-ops (some FUSE filesystems grant two exclusive locks on one file). That store's claims are then treated per onStoreUnavailable, with an error logged once: state: the store's file locks cannot be trusted for cluster-wide concurrency (...); concurrencyScope: cluster claims degrade per onStoreUnavailable.

The contract is at-least-once, not exactly-once. The gate closes the routine overlap windows, but these remain open by design:

• A holder that loses its slot keeps running. The daemon never cancels work over a store blip. If a store outage outlasts the slot TTL and another node takes the slot over, the original holder logs Job <name>: its cluster concurrency slot was taken over by <node> while it is still running here (a store outage outlasted the slot TTL?); the run continues -- the overlap is the documented at-least-once trade. A Forbid peer that then wins the slot overlaps the still-running original.
• degrade trades the gate for availability. While the store cannot answer, only node-local enforcement applies to launches made under onStoreUnavailable: degrade.
• Windows locks are same-host only. On Windows the store's file locks have no cross-host reach, so a cluster slot claim there only fences daemons on the same host (topology: auto resolves to single-node on Windows and logs an advisory; see one backend, two topologies).
• Replace can skip the launch. The pursuit abandons the launch after twice the slot TTL rather than risk a double run: the bias is no-run over double-run.
• Slot expiry compares wall clocks across hosts, so the shared-mount clock discipline in durable state applies: run NTP on every node.

Winning a slot whose previous holder went silent also reconciles that holder's interrupted run into the durable ledger as an unknown outcome. An expired slot proves the holder stopped renewing, not that its process died. See durable state for the in-flight records behind that.

executionTimeout bounds the wall-clock duration of a single run. It is unset by default (null), meaning a run may take arbitrarily long.

When a run starts and executionTimeout is set, RunningJob.start records an absolute deadline using a monotonic clock:

execution_deadline = time.perf_counter() + executionTimeout

The deadline uses time.perf_counter() rather than wall-clock time, so it is immune to system clock adjustments while the job runs.

When the run is awaited (RunningJob.wait):

• If no deadline is set, cronstable waits indefinitely for the process to exit.
• If a deadline is set, the remaining time is execution_deadline - time.perf_counter(). If that remaining time is > 0, cronstable awaits the process exit under asyncio.wait_for(..., timeout). If it is already <= 0, the timeout path runs immediately.
• When the process exits inside the deadline, cronstable waits for the rest of the deadline for the run's captured output to close (stdout under captureStdout, stderr under captureStderr). When a process that the job started still holds a captured stream open as the deadline passes, the timeout path runs as it does for a run whose process is still running.

executionTimeout therefore bounds a run until its captured output closes, on every supported Python version and on both stock asyncio and uvloop. A run whose process has exited and whose captured output has closed by the deadline keeps its own exit code. A run that captures neither stream ends when its own process exits: the deadline has no effect after that exit, and a process that the job left behind is not signaled at the deadline.

On timeout (the remaining time elapses, or was non-positive), cronstable:

1. Logs Job <name> exceeded its executionTimeout of <N> seconds, cancelling it... at INFO.
2. Sets the run's return code to -100.
3. Calls cancel() to terminate the run and its process group (see the next section).

When the job's own process has already exited, the wait for killTimeout in that sequence ends at once: on POSIX the group SIGKILL follows the group SIGTERM without a grace period, and the daemon does not log did not gracefully terminate. Each captured stream then has 30 seconds to reach end-of-file, as for any run that its executionTimeout ends. On Windows the graceful step sends CTRL_BREAK_EVENT to the job's process group and the tree kill finds no process. See job termination semantics for the processes that the break ends.

A -100 return code is therefore the marker of a timeout-induced termination. It replaces the process's own exit code, so it also marks a run whose own process exited by itself (for example with code 0) while a process that it started held a captured stream open past the deadline. For a normal (non-replaced) run, retcode = -100 is nonzero, so a job with the default failsWhen.nonzeroReturn treats the timeout as a failure, which is then reported and may be retried. See failure detection and retries for what happens after a timeout-induced failure. When the timed-out run was a Replace victim, the replaced flag suppresses failure handling regardless of the -100 code.

A deadline can also pass after an operator cancel, a Replace launch, or a lost pool lease has canceled the run, while the run still waits on output that another process holds. The daemon then logs the exceeded its executionTimeout line and the run's exit code is -100, on every runtime. It sends no further signal, and the run is recorded within the same 30-second bound that started when the cancel finished (see cancellation and killTimeout). What canceled the run still decides how it is handled: a replaced run is not reported or retried, an operator-canceled run is recorded as canceled, and a run that lost its pool lease is a failure with exit code -100 under the default failsWhen.nonzeroReturn, including one whose own process had exited with code 0.

jobs:
  - name: maybe-hangs
    command: |
      echo "starting..."
      sleep 2
      echo "all done."
    schedule:
      minute: "*"
    captureStderr: true
    executionTimeout: 1   # seconds; cancel the run if still alive after 1s

An executionTimeout expiry, concurrencyPolicy: Replace, an operator cancel (POST /jobs/{name}/cancel or the MCP cron_cancel_job tool), and a lost resource pool lease all invoke cancellation (RunningJob.cancel). It terminates the run and everything the run spawned. On POSIX each job is started in a fresh session (start_new_session), so the job and every descendant share one process group (the child's own pid), which cancellation then signals as a unit (os.killpg in cronstable/platform.py). Windows has no equivalent at spawn time, so descendants are reached through the live process tree instead (taskkill, described later). The sequence is:

1. Send SIGTERM to the whole process group. This reaches the descendants even when the process cronstable spawned has already exited. If the group cannot be signaled (it is already empty, or killpg failed), fall back to SIGTERM on the direct child with proc.terminate(). A ProcessLookupError (process already gone) is ignored. A signal that the operating system refuses is logged, and the sequence continues, as the paragraph after these steps describes.
2. Wait up to killTimeout seconds for the direct child to exit. The daemon waits on the child and also reads its exit status every 0.25 seconds, so the wait ends within about 0.25 seconds of the exit, including while a descendant holds one of the job's output pipes open. If the child is still running at the end of killTimeout, log Job <name> did not gracefully terminate after <N> seconds, killing it....
3. Send SIGKILL to the whole process group, unconditionally, whether or not the direct child exited within killTimeout. The child exiting says nothing about descendants sharing its group, and those are what hold the job's stdout/stderr pipes open. A group that is already empty makes this a no-op. Where the group cannot be signaled, fall back to proc.kill() on the direct child. For a run inside its own cgroup (resource limits), this step first writes 1 to the cgroup's cgroup.kill, and the kernel sends SIGKILL to every process in the cgroup, including a descendant that called setsid. The group SIGKILL then follows as for any run, so a process that left the cgroup but kept the group ends too. A kernel before 5.14 has no cgroup.kill: the daemon then sends SIGKILL to each process listed in the cgroup.

The operating system can refuse the daemon's signal to a job's own process, for example with EPERM for a process that another user owns, such as a command that sudo runs under a daemon that does not run as root. The daemon then logs Job <name>: could not signal its process (pid <pid>): <error> at WARNING, once for the graceful step and once for the forced step, and completes the sequence: it waits killTimeout for the process, logs Job <name> did not gracefully terminate after <N> seconds, killing it..., and the run has ended when the sequence finishes. On POSIX each step first logs could not signal the process group of pid <pid> (<error>); falling back to signalling that process alone at WARNING. The process keeps running until it exits by itself.

Signaling the group is what makes executionTimeout a bound on the run's work rather than on its root process alone. A command like sh -c 'helper & main' leaves helper behind holding the write-ends of the job's stdout/stderr pipes. Killing only the shell would leave the pipes open, so the run would never finish draining, its slot would never be released, and under concurrencyPolicy: Forbid the job would never run again. Killing the group takes the helper down with the shell. The deadline applies while the run lasts. Once the job's own process has exited and its captured output has closed, the run has ended, and the deadline signals no process that the job left behind (see deadline mechanism).

Both group signals address the group by the process ID of the direct child. After the daemon has collected the child's exit status, the operating system can give the ID to another process, which POSIX permits only once the job's group is empty. On POSIX the daemon therefore checks, before each of the two group signals, whether a process exists under the ID. A process that another user owns counts. When one exists, the job's group is empty, and the daemon sends neither SIGTERM nor SIGKILL to that group ID. When none exists, the daemon sends the signal, which reaches the descendants that share the group. Until the daemon has collected the exit status, it sends both signals without the check. Windows makes no such check. One reuse goes undetected: a process that took the ID, exited, and left members in a process group of its own. The daemon then sends its signals to that group.

On Windows each job is likewise spawned in its own process group (CREATE_NEW_PROCESS_GROUP), and the steps map onto the platform's own primitives. Step 1 delivers CTRL_BREAK_EVENT to the group, a trappable request the job can handle (signal.SIGBREAK in Python, SetConsoleCtrlHandler natively) to flush and exit. Step 3 shells out to taskkill /F /T, which force-kills the live parent/child process tree (the taskkill run itself is bounded at 10 seconds; on failure the fallback is proc.kill() on the direct child). For a run inside its own Job Object (resource limits), step 3 calls TerminateJobObject instead. That ends every process in the job with exit code 1, including one in a process group of its own and one whose parent has exited.

One degradation: delivering a break needs a console shared with the daemon, so where there is none (a service context) step 1 becomes the tree kill immediately and no graceful signal reaches the job. See running on Windows.

killTimeout defaults to 30 seconds and must be >= 0. A value of 0 is valid and means the group SIGKILL follows almost immediately after the group SIGTERM: the daemon lets the event loop run once in between, so the process gets no grace period. The daemon logs the did not gracefully terminate line only when the process has not exited by then.

killTimeout is the time that the job's own process has to exit after the graceful signal. The job's own process is the one that the daemon started (the direct child in the preceding steps): the program itself when command is a list, and the shell when command is a string. The wait ends when that process exits, and the forced step follows within about 0.25 seconds, whatever remains of killTimeout. On POSIX the forced step ends every process that is left in the group. On Windows the tree kill finds no process once the job's own process has exited. For a run with resource limits in place, the forced step ends every process left in its cgroup or Job Object, on both platforms. Raise killTimeout for a job whose own process needs longer to shut down, and lower it for a job that may ignore the graceful signal and must be force-killed quickly.

On POSIX the SIGTERM reaches the shell and the programs that it started at the same moment. A shell that sets no trap for SIGTERM exits at once, and the SIGKILL then ends the rest of the group. A program that runs as a child of the shell therefore has the grace period only while the shell is alive. A workload that needs the whole killTimeout must be the job's own process, or the shell must wait for it:

• Pass command as a list. The program runs with no shell.
• End a shell command with exec, which replaces the shell with the program. Write the exec for a single command too, because shells differ: bash replaces itself with a lone command, and dash 0.5.12, the /bin/sh of Ubuntu 24.04, does not.
• Set a trap that runs a command, such as trap 'exit 143' TERM, ahead of the workload. The shell runs the trap only after the program that it is waiting for has exited.

jobs:
  - name: queue-worker
    command: |
      cd /srv/queue
      exec ./worker --drain   # the worker replaces /bin/sh
    schedule: "*/15 * * * *"
    killTimeout: 120          # the worker has 120 seconds to exit

On Windows the graceful signal is CTRL_BREAK_EVENT, and cmd.exe keeps running until the program that it is running exits, so a workload under a string command has the whole killTimeout. A program that the command launches with start and does not wait for is the exception: once cmd.exe has exited, the tree kill does not reach that program, and it keeps running. In a run's own Job Object, the forced step ends that program too. When the interrupted program exits, cmd.exe goes on to the next step of a command joined with &. That step starts after the break, so it runs until it exits or until killTimeout ends and the tree kill stops it. A step joined with && starts only when the interrupted program exits with status 0. A daemon without a console cannot deliver the break, and there killTimeout has nothing to bound because the tree kill runs at once. See running on Windows.

As defense in depth, a descendant that escaped the group entirely (it called setsid itself, or on Windows it was already orphaned when taskkill walked the tree) cannot strand the run either. A killed run's wait for its stdout/stderr streams to reach EOF is bounded at 30 seconds, a fixed constant deliberately independent of killTimeout, which is legitimately 0 for jobs that must be force-killed at once but whose already-captured output should not be discarded. For a run with resource limits in place, the container kill reaches such a descendant as well.

When that bound expires, cronstable keeps the output captured so far, closes its end of the job's pipes, logs a warning, and records the run. The run loses only the output that the escaped descendant produces afterward. How the bound is counted depends on what canceled the run:

• For an operator cancel, a Replace, or a lost pool lease, one 30-second bound covers both streams and starts when the cancel sequence finishes. The warning reads job <name>: output did not reach end-of-file within 30.0 seconds of the job being killed. The bound applies whether the job's process was still running when the cancel arrived or had already exited by itself. When the process needed the forced kill, the daemon looks for its exit 0.25 seconds after the sequence and counts the 30 seconds from there. While the process outlives the forced kill, as when the operating system refuses the daemon's signal, the pipes stay open: the daemon looks again every 30 seconds and closes them 30 seconds after the look that first finds the process gone. The daemon records that run, with the process's own exit code, when the process exits and its captured output closes. If the job has an executionTimeout and its deadline passes first, the run is recorded with exit code -100 after the 30-second bound for each captured stream. A cancel of this kind that arrives during a verification check counts the same bound for the check's streams, and its warning names verify.output.
• For an executionTimeout, and for a verification check that times out, each captured stream has 30 seconds to reach EOF after the cancel sequence, stderr first and then stdout. A run whose two captured streams are both held open is recorded within 60 seconds. The warning names the stream: job <name>: <stream> did not reach end-of-file within 30.0 seconds of the job being killed. A check's streams are named verify.stderr and verify.stdout. A run whose process outlives the forced kill is recorded with exit code -100 after those bounds, or at once when no stream is captured, while its process is still running. The daemon closes its end of the captured pipes then.

On a Python runtime that reports a process's exit while one of its pipes is still open, a cancel of the first kind can also log the per-stream warning for the first held stream beside the output warning, in either order, for a job with no executionTimeout. A job with an executionTimeout whose process exits during the cancel logs the output warning alone, as on the other runtimes. The run is recorded at the same time in both cases and on every runtime.

A run that a cancel or an executionTimeout stops has ended when the cancel sequence finishes, and the wait for held output comes after that instant. The run record's finished_at is that instant, so its duration excludes the wait (see GET /jobs/{name}/runs). A run of the same job that started after that instant can be recorded first, and it stays the job's last run. A verification check that times out is part of its run: the run ends, and the daemon takes its finished_at, when the check's streams have closed or their bounds have expired.

An instance that the daemon has finished canceling has ended (see concurrency policy), even when one of its processes is still alive: a descendant that left the process group or the process tree (for a run with resource limits in place, one outside its container, which on Linux must also have left the process group), or a process that outlives the forced kill, as when the operating system refuses the daemon's signal. A concurrencyPolicy: Forbid job's next fire then starts a new instance beside that process, and a Replace launch starts the replacement beside it. A job that only ignores the graceful signal causes no such overlap, because its fires are dropped for the whole killTimeout and the forced kill ends it.

A cancel that arrives while a run's verification check runs applies the sequence to the check's process group, with the job's killTimeout. The run has ended when that sequence finishes, even while a process that the check started still holds the check's output open. The daemon records the run when that output closes or its bound expires. A cancel that arrives while the daemon is starting the check, before the check has a process, leaves the instance running. The daemon terminates the check right after it starts (for a workflow task, after the task's verifying status is published), or when verify.timeout expires if that comes first, and the instance has ended from then on.

jobs:
  - name: ignores-sigterm
    command: |
      trap '' TERM   # ignore SIGTERM; sleep inherits the ignored disposition
      echo "starting..."
      sleep 10
      echo "all done."
    schedule:
      minute: "*"
    captureStderr: true
    executionTimeout: 1
    killTimeout: 0.5   # SIGKILL 0.5s after the (ignored) SIGTERM

SIGTERM goes to the job's whole process group. A trap that only runs a handler does not keep the job alive, because sleep dies of the signal and the shell exits after the handler runs. trap '' TERM sets the ignore disposition, which sleep inherits, so the SIGKILL 0.5 seconds later ends the run.

This example's trap spelling is POSIX (sh trapping SIGTERM). The Windows equivalent traps SIGBREAK: the graceful step there delivers CTRL_BREAK_EVENT to the job's process group, and the escalation timing killTimeout illustrates is the same (the forced step is the taskkill /F /T tree kill). See running on Windows, including the no-console case where the graceful step cannot be delivered.

• Per run. executionTimeout and killTimeout apply to a single instance of a job. The deadline is established at that instance's start and is not shared across instances. With concurrencyPolicy: Allow, each concurrent instance has its own independent deadline.
• Replace + timeout. A Replace victim is cancelled regardless of its own executionTimeout. killTimeout governs its termination, and that termination is not reported as a failure (the replaced flag).
• Manual starts. Launches through the HTTP control API (POST /jobs/{name}/start) take the same maybe_launch_job path and so honor concurrencyPolicy, including the cluster slot gate for concurrencyScope: cluster jobs.
• Node first, cluster second. concurrencyScope: cluster never changes the per-node behavior documented earlier. The local check runs first, and the cluster gate is an additional gate behind it.
• Resource limits. A run that reaches its memory cap is killed whole, with no graceful step and no killTimeout wait: by the kernel's OOM kill of its cgroup on Linux, and by a TerminateJobObject call on Windows. It fails with memory limit exceeded (<cap>) and the kill's exit code, -9 on Linux and 1 on Windows. A cancellation or an executionTimeout of a capped run follows the sequence on this page, and its forced step kills the run's cgroup, followed by the process group, on Linux, or its Job Object on Windows.
• startup failures vs. timeouts. A -100 return code specifically denotes a timeout-induced cancellation. A command that could not be launched at all (not found, for example) is assigned 127 instead, on the normal failure path. See commands and environment and failure detection and retries.