Internal design reference for developers reading or extending cronstable. It maps the modules, describes the asyncio event loop, the scheduler main loop and hot reload, the running-job lifecycle, the retry state machine, concurrency handling, and signal-driven shutdown. It references functions by name rather than repeating the option reference. For option semantics, see the configuration reference.

cronstable is an asyncio Python daemon that runs natively on Linux, macOS, and Windows. cronstable/platform.py isolates all OS-specific behavior (DEFAULT_SHELL, DEFAULT_CONFIG_PATH, supports_unix_sockets, encode_argv, install_shutdown_handlers, plus the IS_WINDOWS flag). grp/pwd and per-job user/group switching remain POSIX-only and are runtime-gated on IS_WINDOWS rather than blocking import. For the operator-facing walkthrough, see running on Windows.

The following table is a partial map of the core modules. The package also holds smaller helpers (for example, _json, _gzip, tlsutil, and redact) and the modules behind individual features.

The dependency direction is __main__ -> cron -> (config, job, fingerprint, leadership) -> (statsd, config). cron.py imports make_backend from leadership, giving this dependency chain: cron -> leadership -> {cluster (gossip), backends.kubernetes, backends.etcd, backends.filesystem}. make_backend imports those backend modules lazily, so backends/ never enters the import graph unless cluster.backend selects a lease backend.

cluster.py depends on config, fingerprint, leadership, tlsutil, _gzip, and _json, and has no dependency on cron.py. The lease backends depend on config, leadership, and backends._common; backends.filesystem also depends on state and platform, because it embeds a private FilesystemStateBackend as its lease store. config.py has no dependency on cron.py or job.py.

prometheus.py imports cronstable.version, cronstable.config, and cronstable.cronexpr at module scope, cron.py imports it, and its renderer late-imports cronstable.cron helpers at scrape time to break the cycle. platform.py is a leaf module with no cronstable dependencies, imported by __main__, config, cron, and job wherever per-OS behavior is needed.

state.py depends on config, platform, and _json. cron.py imports it and builds the backend with make_state_backend inside start_stop_state (only when a state: section is configured). __main__ imports state_admin.py (which depends on config and state) lazily for the cronstable state ... subcommands, so state_admin never enters the daemon's import graph.

main() in __main__.py builds no event loop at all. It calls main_loop() with no argument, and the loop is created and closed inside _run_daemon, on the daemon branch only. That is deliberate: asyncio is a ~50 ms import (and several MB of RSS), and every branch that exits earlier (--version, --third-party-licenses, --validate-config, --job-set-id, and the job-spawned thin clients like state get, lock and xcom pull) must not pay it. main_loop(loop) still accepts a caller-supplied loop, which the test suite relies on. A loop passed in stays the caller's to close, while one built by _run_daemon is closed there.

The daemon runs on Windows too. On POSIX the daemon runs on uvloop when the speedups extra is installed, and on stock asyncio otherwise; Windows uses the Proactor loop. POSIX also wires SIGHUP to an immediate reload. _run_daemon also installs an explicitly sized, named default thread pool (see executor_workers()).

main_loop parses arguments (-c/--config, -l/--log-level, -v/--validate-config, --version). The two pure config questions answer without constructing a Cron: --validate-config and --job-set-id call config.parse_config_with_sources() (and, for the latter, fingerprint.job_set_id()) directly, so neither imports the daemon graph nor builds the metrics/sampler/advertiser objects a Cron carries.

A ConfigError from that parse is caught and turned into exit code 1. A clean parse logs "Configuration is valid." or prints the id, and exits 0. Only the daemon branch reaches Cron(args.config), whose construction calls Cron.update_config() once eagerly and whose ConfigError is handled the same way. With --version, the version is printed and the process exits 0 before any config is loaded.

Scheduling logic runs on one event-loop thread, which drives the scheduler loop, the reaper task, all running-job subprocess waits, the reporters, and the aiohttp web server. Concurrency is cooperative through await. Blocking work (config reparse, file-backed secrets, calendar and metrics rendering) runs on the sized default thread pool, output mirroring runs on its own writer thread, and the per-job launch and slot mutexes are asyncio.Locks.

__main__ delegates shutdown-handler registration to platform.install_shutdown_handlers, which returns a cleanup function called in the finally. On POSIX the helper registers SIGINT and SIGTERM with loop.add_signal_handler(...) bound to cron.signal_shutdown. On Windows loop.add_signal_handler raises NotImplementedError, so the helper falls back to signal.signal for SIGINT (Ctrl-C) and SIGBREAK (Ctrl-Break / console close), marshaling onto the loop thread with call_soon_threadsafe and ticking a ~0.25s heartbeat timer so the handler runs while the loop is blocked in IOCP. loop.run_until_complete(cron.run()) blocks until the main loop returns. See running on Windows.

Cron.run first starts the reaper task:

self._wait_for_running_jobs_task = asyncio.create_task(self._wait_for_running_jobs())

It then enters while not self._stop_event.is_set():. Each iteration performs the following steps in order. Steps 1 to 4 are housekeeping, which runs at most once per wall-clock minute, even when a sub-minute schedule wakes the loop more often.

1. Hot reload. config = await self.reload_config() reparses the configuration off the loop, skipping the parse when the config files' stat signature is unchanged, and rebuilds self.cron_jobs (an OrderedDict of name -> JobConfig). On ConfigError, the error is logged and self.cron_jobs is left unchanged, because reload_config only assigns self.cron_jobs on a successful parse. The loop keeps running the previously loaded jobs. config is initialized to None at the top of each iteration so a failed parse does not dereference an unbound config later in the body. Any other exception is logged as "please report this as a bug (1)".

2. Cluster start/stop. await self.start_stop_cluster(config.cluster_config) runs inside the same try as reload_config, immediately after it, and reconciles the leadership backend (self.cluster_manager, typed Optional[LeadershipBackend], so it may be the gossip ClusterManager or a KubernetesBackend/EtcdBackend) against the (possibly changed) cluster config, mirroring the web-app reconcile: the backend is stopped when the section is removed or changed, and (re)started when present.

   It also records _elect_leader_configured up front so the leader gate can fail closed even when the backend is absent or failed to start. A start failure (bad cert/credential files, a bad listen address, a port in use, or a lease store the backend cannot reach or authenticate to) is logged and the reload continues. The id the backend reports tracks reloads on its own (it calls self.job_set_id each round), so only a change to the cluster section itself needs a restart. (See "Cluster manager" and clustering and leader election.)

   The same try block then reconciles the observability overlay (start_stop_observability), the state backend (start_stop_state), and push (start_stop_push).

3. Web app start/stop. await self.start_stop_web_app(config.web_config, config.mcp_config) reconciles the running aiohttp server against the (possibly changed) web config. It runs only when the config parsed, and after the cluster reconcile, under its own error handling. A ConfigError (for example, an authToken that resolves empty) logs "Error in the web configuration, so not starting the web API" and leaves only the web API down. The rest of the new config (jobs, cluster, logging) is still applied, so a web misconfiguration can never skip the cluster gate (which would fail open). Any other exception here is logged as "please report this as a bug (4)". (See "Web control app".)

4. Logging config. If the reloaded config has a non-None logging_config that differs from applied_logging_config, logging.config.dictConfig(...) is applied. It is recorded as applied only on success, so a logging section that was broken on a previous reload is re-tried and picked up after a fix, without a restart. A failure logs an error pointing at the Python logging.config dictionary-schema docs and the offending config.

5. Service due work. await self._service_slots(startup) reads the clock once, after housekeeping, then runs spawn_jobs, catch-up, the DAG scheduler, and pool servicing.

6. Sleep until the next wake. _sleep_interval() returns the smallest of the soonest job fire, the next housekeeping minute boundary (next_sleep_interval()), and the next DAG wake. The loop then waits on _wake_event with that timeout, so a shutdown or a SIGHUP reload request wakes it immediately.

startup is True only on the first iteration and is set to False immediately after the first _service_slots. This drives the @reboot startup pass (see the next section).

spawn_jobs(startup, now) first logs any leadership transition through _log_cluster_role. At startup only @reboot jobs are considered. Afterwards due jobs come from a next-fire index, advanced with bounded catch-up, then gated by _cluster_allows and launched in config order through launch_scheduled_job.

_cluster_allows is always True unless cluster.electLeader is configured. Then it consults the job's clusterPolicy against the elected-leader state. EveryNode always runs. Leader/PreferLeader fail closed when no backend is running, except PreferLeader, which is never-skip and runs anyway. Otherwise, under the default distribution: single-leader, it gates on is_available_leader() for PreferLeader and, for Leader, checks has_conflict() first (fail closed if true, see later) and then is_leader(). Under distribution: spread the same shape applies to the per-job variants (is_available_job_owner(name) for PreferLeader, then has_conflict()/is_job_owner(name) for Leader), so leader-gated work fans out across the quorate nodes by rendezvous hashing.

The has_conflict() fail-closed-first case stands Leader jobs down whenever a reachable peer makes the election unsafe (a duplicate nodeName, a cluster-size disagreement, or a coordination-policy conflict). PreferLeader is left running because it already accepts double-runs. A backend read is wrapped in a try, and any exception fails the gate closed (skip this cycle) so a backend bug cannot stop the unguarded spawn_jobs path.

Manual (API) triggers and retries go through maybe_launch_job and are not gated (though schedule_retry_job re-checks the gate before relaunching a scheduled-job retry). See clustering and leader election.

job_should_run:

• returns False for any disabled job (job.enabled is False);
• when startup is True, returns True only for jobs whose schedule is the string "@reboot", and False for all others;
• when startup is False, evaluates CronTab schedules with crontab.test(schedule_slot(job)) (the current second for a second-level job, otherwise the current minute) and returns True when that slot matches. Non-CronTab schedules (that is, "@reboot") return False on non-startup iterations, so a @reboot job runs once at daemon start and never on the regular cadence.

Without a state: section, a @reboot job runs each time the daemon starts.

When a state: section is configured, a non-cluster-deferred @reboot job additionally passes through _reboot_boot_gate before launching: a durable boot marker (stream reboot/<job>) turns the one-shot into once per OS boot per host rather than once per daemon start. The ordering is record-before-launch. The marker is appended (bounded, STATE_OP_TIMEOUT) before the spawn, the same at-most-once bias as the cluster's mark_reboot_ran path, so a crash between record and spawn errs toward not re-running.

Boot identity comes from platform.os_boot_id() (Linux's /proc/sys/kernel/random/boot_id, an exact per-boot UUID), falling back to platform.os_boot_time() (boot time derived from uptime: GetTickCount64 on Windows, /proc/uptime on POSIX) compared within BOOT_TIME_TOLERANCE (60 s). Where neither source exists (macOS/BSD) the behavior is unchanged (runs every daemon start). The marker also carries the host and the job's per-job digest (fingerprint.job_digest), so a redefined @reboot job runs again.

An unreadable or unwritable marker runs the job anyway under the default onStoreUnavailable: degrade (at-least-once, the stateless behavior) and skips it under fail-closed. Cluster-deferred Leader/PreferLeader @reboot jobs never reach this gate. Their dedupe remains the leadership backend's reboot_ran path.

When _stop_event is set the while loop exits and Cron.run logs "Shutting down (after currently running jobs finish)...", then:

1. Drains pending retries: while self.retry_state is nonempty, it cancel_job_retries(name) for every entry concurrently with asyncio.gather, passing settle=None. With a state: section configured, a graceful stop leaves each pending durable retry record in place for the next boot's re-arm (see "Retry state machine").

2. If a leadership backend is running, logs "Stopping cluster manager" and await self.cluster_manager.stop() (which releases leadership best-effort for fast failover: gossip cancels the poll loop and tears down the mTLS /peer listener; a lease backend cancels its renew loop and releases its lease).

   Leadership is released before the running-job drain in step 3: the drain is unbounded, and holding leadership through it would stall every Leader job cluster-wide until the slowest local job finishes. The trade-off is confined to the still-draining jobs: the new owner may start one of those while it finishes here, the same overlap a crash produces. (Retries were all canceled in step 1, so no retry task consults the stopped manager.)

3. await self._wait_for_running_jobs_task awaits the reaper, which returns only after self.running_jobs is empty and the stop event is set.

4. If a web server is running, logs "Stopping http server" and await self.web_runner.cleanup().

signal_shutdown calls self._stop_event.set(). The platform shutdown handler invokes it (SIGINT/SIGTERM on POSIX, Ctrl-C/Ctrl-Break on Windows; see "The event loop"). handle_job_failure early-returns when _stop_event.is_set(), so jobs that finish during shutdown are not reported as failures and do not schedule new retries.

The run loop reloads through reload_config, which reparses off the loop only when the config files' stat signature changed; update_config is the synchronous twin used at construction. When config_arg is None (the unit-test path) both return an empty CronstableConfig. Otherwise they call parse_config_with_sources, which runs parse_config(config_arg) and reports the files the parse read. parse_config dispatches on whether the argument is a directory (_parse_config_dir) or a single file (parse_config_file). On success the reload overwrites self.cron_jobs with a fresh OrderedDict keyed by job name and returns the full CronstableConfig (jobs, web config, job defaults, logging config).

The stat signature holds each source file's path, modification time in nanoseconds, and size, plus a configuration directory's own modification time. For a source that is not a regular file, such as a pipe or a FIFO, it holds the path alone, because reading such a source moves its modification time. reload_config takes it before the parse starts and again when the parse returns, and records the first value only when the two are equal. An edit that lands while a parse runs, including one between the signature and the read of the file, is therefore parsed again at the next check. This holds for a single configuration file as well as for a directory or an include tree. When an edit and its reversal both land between the two signatures and restore the file's size and modification time, the daemon keeps what the parse read until the file changes again or a reload is requested. After a reload that reads a different set of source files (a configuration file, an include, or an env_file added or removed), the daemon records no signature and parses once more at the next check. update_config parses once at startup, then reads every source that is a regular file again and compares it with the bytes that it parsed. When a file changed in between, the first scheduling pass parses the configuration again. A source that cannot be read twice (a pipe, a FIFO, a terminal, /dev/stdin, or /dev/fd/N) is read once at startup and left out of that comparison. A later load reads such a source again only when it parses the file that names it (see the per-file parse cache later in this section). A drained pipe then gives no bytes, and the read of a FIFO waits until a writer opens it.

When the configuration that -c names is itself a pipe, a FIFO, or a terminal (by its own path, or as /dev/stdin or /dev/fd/N), the once-a-minute check never reads it again, so the check does not wait on a FIFO that has no writer. When the size or modification time of another source changes (an included file, an env_file, or the cluster.etcd.password.fromFile file), the daemon logs at WARNING A configuration source changed. <config path> is not a regular file, so only a reload request reads it again; keeping the loaded configuration. It keeps the loaded jobs and the cluster, web, and state sections, and cronstable_config_last_reload_successful reads 0. The daemon logs once for each change. For a FIFO, the next reload request reads the configuration and the changed source again. A drained pipe gives that request no bytes, as described later in this section. At startup, when a regular source file differs from the bytes that the daemon parsed, the first scheduling pass logs the same warning and does not parse again.

A reload request (SIGHUP, cronstable service reload) reads a pipe or FIFO configuration once for each request, and requests that arrive before one pass share one read. For a FIFO the read waits until a writer opens it, and scheduling waits with it, so write the configuration to the FIFO and request a reload. A request that arrives while that read runs gets a read of its own. After a reload that installs a configuration from a FIFO, the daemon reads each regular source file a second time and compares it with the bytes that it parsed, as at startup. It does not parse again at the next check, including when the reload read a different set of files. When a regular source file differs from the bytes that the reload parsed, the next check logs the A configuration source changed warning. A reload of a FIFO configuration that fails to parse logs the error, cronstable_config_last_reload_successful reads 0, and the next check does not try it again. The next reload request reads the FIFO again.

When a reload reads zero bytes from a configuration that cannot be read twice (a drained pipe, a standard input at end-of-file, a FIFO whose writer wrote nothing), the daemon installs nothing. It logs at WARNING Reload read no configuration from <config path>, which gives its bytes once; keeping the loaded configuration, keeps the loaded jobs and the cluster, web, and state sections, and cronstable_config_last_reload_successful reads 0. Each request and each change is answered once. Restart the daemon to load another configuration from a pipe or standard input. An empty regular file named by its own path is a valid empty configuration. An empty regular file reached through /dev/stdin or /dev/fd/N is refused with the same warning, and so is a reload request with -c /dev/null.

With -c /dev/stdin or -c /dev/fd/N where the descriptor is a regular file, the once-a-minute check reads the configuration again when the file's size or modification time changes. On Linux that read returns the file from its start, and the edit is applied. Where the open shares the descriptor's offset (macOS, BSD), the read starts where the first read ended, so it returns nothing or only the bytes that were appended. The daemon installs neither: it logs the Reload read no configuration warning and keeps the loaded configuration. When the appended bytes do not parse, that reload first fails with the parse error, and the next check logs the warning.

For a configuration directory, update_config and reload_config also list the directory again when the parse returns. They compare the names that the loader takes (YAML by extension or a crontab by name, without the names that start with _ or .) with the names that the parse listed. When a file that the loader takes joined or left the directory in between, the daemon records no signature, and the next check parses again. A file that joins the directory while a parse runs is therefore loaded at the next check, including when the directory's modification time did not move.

The files of a configuration directory and the targets of include go through a per-file parse cache. Each entry is signed with a digest of the bytes that its parse read: the file's own bytes, those of every file it includes, and those of every env_file that its jobs and workflow tasks read. A load reuses an entry only while each of those files on disk still hashes to the signature, so a file that is replaced after the daemon read it is parsed again by the next load that reaches the parser. When one load reads a file in two different states, the entry of the file that includes it gets a signature that matches no content, and the next load parses that entry again. An edit to one file of an include tree reparses that file, each file that includes it, and the entry file, which is parsed on every load. The cache has room for every file of the loaded configuration, and for at least 1,024 files.

The signature covers the sources that can be read again. A source that cannot be read twice (a pipe, a FIFO, a terminal, /dev/stdin, or /dev/fd/N) is left out of it, and a load never opens such a source to check an entry. The entry of the file that names the source stays in use while its other sources are unchanged, so jobs keep the environment read from a piped env_file across a reload request and across an edit to another file. An edit to the file that names a piped env_file, or to another of that file's sources, parses the file again and reads the env_file again. The entry file is parsed on every load, so every load reads a piped env_file that the entry file names. An included file that is read from a pipe has an entry of its own, which keeps the file's jobs across a reload request and across an edit to another file, the file that includes it among them. When a FIFO named by its own path is replaced by a regular file or deleted, the file that names it is parsed again at the next load. A /dev/stdin or /dev/fd/N path counts as read-once whatever it names: closing the descriptor does not make the file that names it parse again, and an edit to a regular file reached through such a path is not read until the file that names the path is parsed again.

When a regular file that an entry read (the file itself, an included file, an env_file, or the cluster.etcd.password.fromFile file) is replaced by a FIFO, the next load does not open the FIFO to check the entry. It parses the file that names it again, and that parse is the one read of the FIFO, so a FIFO that is written once is enough. On Linux, macOS, and BSD, the check of an entry takes one os.stat of each of its sources before it opens them.

The file that cluster.etcd.password.fromFile names is a source file like an env_file. It is part of the stat signature and the startup comparison, and of the cache signature of the file that holds the cluster section and of each file that includes it. The once-a-minute check therefore parses again when the password file's size or modification time changes, unless the configuration is a pipe, a FIFO, or a terminal (see earlier in this section). A reload request reads a password that was rewritten in place. The check that reads a new password restarts the cluster connection once. Change the password on the etcd server before you replace the file: when the file changes first, the restarted connection fails to authenticate until the server has the new password. The other fromFile secrets are read outside the parse and are not source files.

Because housekeeping checks for a reload at most once per wall-clock minute, and immediately on SIGHUP, configuration edits take effect within a minute without a restart. SIGHUP clears the recorded signature, so the next pass parses whatever the files' stats say. A request that arrives while a parse is running gets a parse of its own on the next pass, and a request that arrives before the parse starts is served by that parse. Schedule parsing, include merging, and defaults application all happen inside parse_config*. See includes, defaults, and multi-file config.

start_stop_web_app(web_config) reconciles a single aiohttp AppRunner:

• If a runner exists and the new web_config is None or differs from the currently applied self.web_config, the old server is cleaned up and self.web_runner is reset to None.
• If a web_config with a nonempty listen list exists and no runner is running, a new web.Application is built. When authToken or scoped authTokens entries are configured, _resolve_web_tokens resolves each bearer token from its source, in the precedence value, then fromFile, then fromEnvVar (the first set source wins), and raises ConfigError if one resolves to empty (fail-closed). _make_auth_middleware then enforces a case-insensitive Bearer scheme and a constant-time hmac.compare_digest token comparison. Routes come from the WEB_ROUTES table (see HTTP API). web_site_from_url turns each listen URL into a site (TCP for http://host:port, Unix socket for unix://path). On a malformed URL or a bind error, the daemon logs a warning and skips that address rather than stopping the reload. _apply_socket_mode applies socketMode to unix:// sockets.

On Windows, unix:// listeners are unsupported (the Proactor loop lacks create_unix_server / aiohttp's UnixSite), so such a URL is skipped (gated by platform.supports_unix_sockets) with the warning "Ignoring web listen url <url>: unix-socket listeners are not supported on this platform". Use an http:// listener instead. Because socketMode only ever applies to unix sockets, it is irrelevant on Windows. See running on Windows.

For the request/response contract, see the HTTP control API.

The scheduler checks whether a node may run a job through the LeadershipBackend abstract base class in cronstable/leadership.py. make_backend(cluster_config, get_job_set_id) constructs the implementation selected by cluster.backend. It imports backend modules only when needed, so using gossip does not load the lease implementations:

• gossip (default) -> cluster.ClusterManager, the mTLS, no-shared-state, best-effort quorum election (detailed later). Zero new dependencies.
• kubernetes -> backends.kubernetes.KubernetesBackend, a coordination.k8s.io/v1 Lease. Provides fenced leadership with a single lease holder while the store is reachable.
• etcd -> backends.etcd.EtcdBackend, a lease-backed key/election against an etcd cluster, same fenced guarantee.
• filesystem -> backends.filesystem.FilesystemBackend, a flock-guarded, fence-counted TTL lease on a shared POSIX mount, embedding the durable store's lease machinery (state.FilesystemStateBackend) as its store. Fenced, but lease expiry is judged across the participating hosts' wall clocks, so unlike the kubernetes/etcd backends it requires NTP-bounded skew on every node. Stdlib-only.

The LeadershipBackend interface has three groups of methods, allowing lease backends to reuse common behavior:

• The core abstract methods every backend implements: start, stop, is_leader, leader_name, is_quorate, and view_dict.
• The defaulted bodies a single-holder lease backend inherits unchanged: per-job ownership collapses to the leader, there are no gossip-style conflicts, the cluster is logically size 1, the view is never mid-convergence, and TLS rotation does not apply.
• The never-skip available_* family, defaulted to the locked lease semantics: a node that currently cannot reach the store runs a PreferLeader job anyway, while a node that can see the holder defers, and Leader stays fail-closed.

The @reboot "already ran" defaults (reboot_ran false, mark_reboot_ran a no-op) are the one pair LeaseBackend replaces rather than inherits. It persists the ran-set in the lease store (a Lease annotation / etcd sibling key under REBOOT_RAN_KEY), scoped to the job-set ID, so a failover holder does not re-run a one-shot (see clustering and leader election). Gossip overrides these defaults to preserve its existing behavior.

The lease backends share LeaseBackend, which pins distribution to "single-leader" and provides the common lease status fields in view_dict(). The built-in Kubernetes and etcd transports use HTTP APIs through the core aiohttp dependency, with HTTPS supported. The filesystem backend uses the standard library. None requires grpc/protobuf wheels, preserving the supported architectures. For the operator-facing model and the lease backends' fencing, see clustering and leader election.

start_stop_cluster(cluster_config) reconciles the single backend held in self.cluster_manager (typed Optional[LeadershipBackend]), mirroring the web-app reconcile: the backend is stopped when the cluster section is removed or differs from the running one, and (re)started (with make_backend inside a try) when present and none is running.

A start failure is caught as OSError, ssl.SSLError, ValueError, ConfigError, aiohttp.ClientError, or asyncio.TimeoutError, logged, and the reload continues (jobs keep running). OSError/ssl.SSLError/ValueError cover the gossip case (bad cert files, a bad listen address, a port already in use). ConfigError, aiohttp.ClientError, and asyncio.TimeoutError additionally cover a lease backend that cannot reach or authenticate to its store at start() (a ClientResponseError on a rejected token is an aiohttp.ClientError, not an OSError), so these operational misconfigurations are logged rather than escaping to the run loop's generic "please report this as a bug" handler. _elect_leader_configured is set first, so the leader gate is correct even if the backend is absent.

The gossip concrete (ClusterManager in cronstable/cluster.py) owns two things:

• The mTLS /peer listener: its own aiohttp AppRunner on the cluster listen address, with a server SSL context that requires a CA-signed client cert (ssl.CERT_REQUIRED). Its GET /peer returns the full attestation payload (see "The peer attestation payload" later). It also accepts POST /reboot-ran (the eager @reboot push). This listener is entirely separate from the public web app.

• The poll loop: every interval seconds it polls each peer's /peer over mTLS (a client SSL context with check_hostname=True, so the peer cert's SAN must match the configured host), and feeds each observation into the pure ClusterView. The poller classifies the peer as untrusted after a TLS or cert failure, unreachable after a connect or timeout failure.

  The exchange is conditional. Every /peer response carries a strong ETag (a content hash of the payload, with the live per-job countdown normalized to the absolute next-fire time so the tag is stable between fires), the poller echoes it back as If-None-Match, and an unchanged peer answers with a bodyless 304. The poller then replays its cached observation with a fresh timestamp, so a converged, idle cluster's steady-state round costs headers rather than the full O(members + jobs) JSON. A 304 is still a fresh, mutually-authenticated round trip, so agreement, conflict detection, and the drift debounce advance exactly as if the identical body had been re-sent. Full bodies large enough to be worth it are gzip-compressed.

ClusterView is pure (no I/O): it holds the per-peer table and the rules that update it (the agreed/syncing/drifted/unreachable/untrusted/self state machine and the driftAfter debounce), which keeps the logic trivially testable. Leader election is likewise pure. quorum_size, elect_leader (quorum-gated, returns None for a minority), and elect_available_leader (no quorum gate, for PreferLeader) take a node name, the agreeing peer names, and the cluster size, and the manager wraps them as is_leader() / is_available_leader() / leader_name(). The /cluster web endpoint serializes view_dict().

Under distribution: spread the single leader is replaced by per-job ownership. The equally pure elect_job_owner (quorum-gated) and elect_available_job_owner pick the owner of a given job by rendezvous (highest-random-weight) hashing (_hrw_score / _hrw_owner), and the manager exposes is_job_owner(name) / is_available_job_owner(name). Cron._cluster_allows branches on manager.distribution to call the leader or the per-job variant. The choice is purely about which node runs a job, so the quorum gate and the guarantee are unchanged. Because the owner is a deterministic function of the job name and the agreeing member set, all quorate nodes with a converged view agree, and a membership change only reassigns the affected jobs.

A gossip node's GET /peer handler (ClusterManager._handle_peer) returns a single JSON object that a polling peer feeds into its ClusterView and its election. The full body:

{
  // this node's stable, human-readable identity (the configured nodeName,
  // defaulting to the hostname). The election's key: elect_leader picks the
  // minimum live name, so two nodes sharing this is a conflict_names conflict.
  "node_name": "cronstable-a",
  // the order-independent job-set fingerprint (fingerprint.canonical_job);
  // peers with a different value are "drifted" (running a different config).
  "job_set_id": "3f2a...c9",
  // the fingerprint scheme version (fingerprint.SCHEME_VERSION), so a peer on a
  // newer hashing scheme is recognised rather than silently treated as drifted.
  "scheme_version": "v1",
  // a per-process-boot random token: lets a poller distinguish a genuine
  // restart of this node from a second node fraudulently reusing node_name,
  // and dedupes a node reachable at two addresses.
  "instance_id": "b17e...",
  // this node's declared cluster size (len(peers)+1). The election's safety
  // assumes every node shares one N; a peer declaring a different one is a
  // size_conflict (conflicting_sizes) and fails Leader closed.
  "cluster_size": 3,
  // this node's own per-peer observations (itself first, always agreeing), each
  // tagged with node_name, instance_id, and whether it is currently seen
  // agreed, so the poller can confirm the edge is two-way (mutual agreement)
  // and spot a duplicate nodeName transitively (one name, two instance_ids).
  "members": [
    { "node_name": "cronstable-a", "instance_id": "b17e...", "agreed": true },
    { "node_name": "cronstable-b", "instance_id": "6c40...", "agreed": true }
  ],
  // the peers this node MUTUALLY agrees with (witnessed two-way edges): the
  // poller uses this as sound evidence that a transitively-reached node is
  // itself quorate, driving bridge-discovery deferral.
  "mutual_agreeing": ["cronstable-b", "cronstable-c"],
  // @reboot one-shots already run in the cluster (this node's, plus any it
  // learned from agreed peers), so a poller can retire its matching deferred
  // job without re-running it. Capped to MAX_ADVERTISED_REBOOT_JOBS.
  "ran_reboot_jobs": ["nightly-migrate"],
  // this node's coordination policy: neither is in the job-set fingerprint, so
  // a peer declaring a different distribution or elect_leader is a
  // policy_conflict (conflicting_policies) and fails Leader closed.
  "distribution": "single-leader",
  "elect_leader": true,
  // the nodes THIS node can confirm are themselves quorate (its eligible
  // candidates): stronger than mutual_agreeing. A poller folds these into its
  // spread Leader-path owner set, so it only ever defers a job to a node
  // vouched able to run it.
  "quorate_vouched": ["cronstable-a", "cronstable-b", "cronstable-c"],
  // per-job run summaries for the fleet view (GET /fleet). Observability
  // only, never an election input; capped, and job_summaries_truncated is
  // true when the cap dropped jobs.
  "job_summaries": {
    "hourly-report": {
      "running": false, "enabled": true, "scheduled_in": 1800.0,
      "last": { "outcome": "success", "finished_at": "...", "duration": 1.2, "exit_code": 0 }
    }
  },
  "job_summaries_truncated": false
}

The distribution, elect_leader, and cluster_size fields make coordination policy and cluster size part of the attested payload precisely so a disagreement becomes a detectable conflict rather than a silent split-brain: a differing distribution/elect_leader surfaces as policy_conflict: true, a differing cluster_size as size_conflict: true, and both (like a duplicate node_name) stand Leader jobs down through the umbrella conflict flag.

Threat model. The listener requires a CA-signed client cert (ssl.CERT_REQUIRED) and the poller pins the peer cert's SAN (check_hostname=True), so only a node holding a cert your CA issued can join the gossip.

But mTLS authenticates the transport, not the claims: every field in that body is self-asserted, so a validly-signed node can lie about its own state (advertise a different job_set_id, claim quorate_vouched peers it cannot reach, or reuse another node's node_name). This is why the election is best-effort and why the conflict gates fail Leader closed on any disagreement rather than trusting a single peer's word. The instance_id blunts the crudest fabrication (a second node reusing a node_name shows a different instance_id, surfacing the duplicate).

Responses are size-capped (ran_reboot_jobs truncated to MAX_ADVERTISED_REBOOT_JOBS, the whole body bounded by MAX_PEER_RESPONSE_BYTES) so a peer cannot exhaust a poller's memory with an inflated payload, and poll failures are bounded by timeouts (classified unreachable) so a slow or unresponsive peer cannot exhaust file descriptors or wedge the loop. For the operator-facing trust boundary, see clustering and leader election.

A single long-lived task started by Cron.run waits for running jobs and dispatches their completions. It maintains a local wait_tasks map from RunningJob to an asyncio.Task wrapping job.wait(). It runs until shutdown is signaled and self.running_jobs is empty.

The reaper scans self.running_jobs once when it starts, including when Cron.run invokes it for the final shutdown drain. Regular jobs and DAG tasks register subsequent launches through _add_running_instance, which adds the instance to _reaper_pending and sets _jobs_running. Pending registrations are keyed by instance identity, so concurrent runs of the same job have separate entries. Removing an instance also removes its pending registration.

Each cycle drains pending registrations into wait_tasks. A wait task's done callback files the completed job and sets a completion event. The reaper clears _jobs_running after registration, in the same event-loop turn. When idle, it waits on _jobs_running; while jobs are running, it waits on the launch and completion events with asyncio.wait(..., return_when=FIRST_COMPLETED). These waits have no polling timeout. Registration work is proportional to new instances, and each completion callback takes constant time.

The reaper handles completed jobs in batches. It removes each job from wait_tasks, checks task.cancelled(), reads task.result() when the task was not canceled, and passes the job to _handle_finished_job. A wait task that ended canceled is logged at ERROR with the line The wait on job <name> was canceled; please report this as a bug (9). Its job still goes to _handle_finished_job, so the instance leaves running_jobs, and the reaper continues with the batch. An unexpected wait exception is logged as bug (2). A handler exception is logged as bug (6), and processing continues with the rest of the batch. If that instance remains in running_jobs, it is queued for another completion attempt. Each batch flushes buffered DAG completions in a finally block, including when cancellation interrupts the batch.

CancelledError is re-raised. Other unexpected exceptions in the outer loop are logged as bug (3), followed by a one-second sleep.

_handle_finished_job removes the job from self.running_jobs[name] (deleting the key when the list becomes empty), then:

• If job.replaced is True, the run was a deliberate concurrencyPolicy: Replace termination: it is logged as replaced and not reported and does not trigger retries.
• Otherwise it inspects job.fail_reason. None -> handle_job_success; non-None -> handle_job_failure.

maybe_launch_job constructs RunningJob(job, self.retry_state.get(job.name)), calls await running_job.start(), and registers the instance with _add_running_instance. The lifecycle inside RunningJob:

1. start() chooses the spawn function from the command form: create_subprocess_exec for a list command, or when shell is set the command is run as [shell, "-c", command] through exec. A string command with no shell uses create_subprocess_shell. Windows naming cmd/cmd.exe is the one exception (shell_spawn): cmd.exe requires /c, and it parses its command line by its own rules rather than the CommandLineToArgvW rules argv rendering assumes, so it takes the create_subprocess_shell path too and the command string reaches %ComSpec% /c unrendered.

   The default shell is platform-specific (platform.DEFAULT_SHELL). POSIX defaults to /bin/sh, so a string command runs as ["/bin/sh", "-c", command]. On Windows the default is empty, so a string command with no shell goes through create_subprocess_shell to the native command processor %ComSpec% (cmd.exe). A list command bypasses the shell on every platform.

   The spawn kwargs start from platform.new_process_group_kwargs(). On POSIX start_new_session=True puts the job in a fresh session/process group (the child's pid is the pgid), so cancel() can take the whole descendant tree down as a unit. On Windows CREATE_NEW_PROCESS_GROUP does the same for console process groups, which both shields the job from the daemon console's Ctrl-C and makes it a CTRL_BREAK_EVENT target for cancel().

   start() also assembles env (when the job has environment entries, when the daemon injects CRONSTABLE_* run variables, or on a PyInstaller build, layering them over os.environ after fixup_pyinstaller_env), sets preexec_fn=self._demote when a uid/gid is configured, requests stdout/stderr PIPEs per captureStdout/captureStderr, sets the pipe flow-control limit to 64 KiB (_READ_CHUNK; the reader enforces maxLineLength), and records execution_deadline = time.perf_counter() + executionTimeout when a timeout is set. Arguments are encoded with platform.encode_argv before spawning (UTF-8 bytes on POSIX; passed as str unchanged on Windows for CreateProcessW).

   The preexec_fn=self._demote path is POSIX-only: per-job user/group switching is not supported on Windows, where a job with user or group set is rejected up front with the configuration error "Job <name>: changing user/group is not supported on Windows" (gated on IS_WINDOWS in config.py), so _demote/preexec_fn never applies there.

   If the spawn raises SubprocessError, ValueError (an unencodable argv, an embedded NUL), or OSError (a bad argv[0] or cwd, plus the resource-exhaustion and permission cases), the error is logged, self.start_failed = True is set, and start() returns without a process. On success _on_start() emits the statsd job_started metric (best-effort; OSError is caught) and a StreamReader task is started for each captured stream.

   The job's priority reaches the spawn through the same two-platform split. On Windows the Ctrl-C/CTRL_BREAK creation flags also carry it as a priority class, OR-ed in by new_process_group_kwargs itself. That is why the helper is the only writer of creationflags in the tree: CreateProcess is the only race-free moment to set a class, and a caller assembling the flags itself could drop one of the two bits.

   POSIX has no spawn-time equivalent, so platform.apply_priority renices the job's process group (setpriority with PRIO_PGRP) as the first thing after a successful spawn. It is not a preexec_fn: that hook runs between fork and exec, where only async-signal-safe calls are sound, and wiring one would put a fork-time hook on every spawn, including the jobs that have nothing to drop. A raised Windows class covers the spawned process and not the tree it goes on to launch, for the CreateProcess reason set out in priority. See running on Windows.

2. _demote() runs in the child (still privileged) and drops privileges in the security-critical order: supplementary groups first (os.initgroups(username, gid) when both are known, else os.setgroups([])), then os.setgid(gid), then os.setuid(uid). Any failure raises RuntimeError. (uid/gid resolution and the "must be root" check happen earlier in JobConfig._resolve_user_group.) This whole privilege-drop path is POSIX-only: there is no setuid/setgid model on Windows. There, user/group config is rejected up front with a configuration error (see start() earlier), so _demote is never reached.

3. wait() is what the reaper awaits. If there is no process but start_failed is set, it synthesizes retcode = 127 (conventional "command not found"), reads streams, and returns, so a failed launch is a normal job failure, not a reaper bug. With no deadline it awaits proc.wait() and then _on_stop(). With a deadline it awaits asyncio.wait_for(proc.wait(), timeout) and, once the process has exited, waits until the deadline for the captured streams to close (_output_open_at_deadline). When the deadline passes with the process still running or a captured stream still open, it logs the executionTimeout, sets retcode = -100, and calls cancel(). Finally it reads the stream buffers.

4. cancel() signals the job's whole process group, not only the spawned process. It sends SIGTERM to the group with platform.kill_process_group(pid, force=False) (falling back to proc.terminate() on the direct child where the group cannot be signaled), waits up to killTimeout for the direct child to exit, then forcibly stops the group with kill_process_group(pid, force=True), unconditionally even when the child exited in time, because descendants sharing its group are what hold the job's stdout/stderr pipes open (fallback: proc.kill() on the direct child). When the operating system refuses a fallback signal (for example EPERM for a process that another user owns), cancel() logs Job <name>: could not signal its process (pid <pid>): <error> at WARNING and completes the sequence.

   On POSIX both group signals are os.killpg (SIGTERM, then SIGKILL). On Windows the graceful call delivers CTRL_BREAK_EVENT to the job's process group (trappable as SIGBREAK); where the break cannot be delivered (no shared console, as in a service context) that call degrades to the forced tree stop immediately, while the root is still alive to anchor the walk. The forced call shells out to taskkill /F /T, which walks the live process tree (the taskkill run is bounded at 10s).

   On POSIX, once asyncio has reaped the direct child, cancel() checks before each group signal whether a process exists under the child's pid (platform.pid_reused, a signal 0 probe). POSIX keeps a pid out of use while a process group with that id has a member, so a process that exists there is an unrelated one and the job's own group is empty. In that case cancel() sends the group no signal. Windows makes no such check.

   A run whose spawn failed (proc is None) makes cancel() a logged no-op rather than an error, because callers (the Replace branch, the cluster slot-renewer) run outside the scheduler loop's try/except. On a run that cancel() stopped, the stream drain in _read_job_streams is bounded (30s, fixed) so a descendant that escaped the group cannot strand the run in running_jobs; cronstable then closes its end of the job's pipes. Finally cancel() calls _on_stop(), which is idempotent (guarded by self._stopped) because cancel() and wait() can both reach it for one run (for example, under Replace); it emits the statsd job_stopped metric once. A cancel that arrives while the run's verification check runs goes to the check, and cancel() returns without calling _on_stop(). wait() calls it once it has collected the check, so the stop metrics are sent then and the success gauge carries the check's outcome. Full user-facing semantics: cancellation and killTimeout.

5. Failure classification. fail_reason is a property evaluated against failsWhen: always, then nonzeroReturn (retcode != 0), then producesStdout/producesStderr (true when captured output is nonempty or lines were discarded). failed is fail_reason is not None.

6. Reporting. report_failure, report_permanent_failure, and report_success each delegate to _report_common, which calls _fan_out_reports to run all six REPORTERS (SentryReporter, MailReporter, ShellReporter, WebhookReporter, PushReporter, EventLogReporter) concurrently with asyncio.gather(..., return_exceptions=True). An exception from any one reporter is logged and does not stop the others. Each reporter reads the relevant sub-key of the report config (onFailure["report"], onPermanentFailure["report"], or onSuccess["report"]) and self-disables when its required fields are unset (for example, mail with no to/from, sentry with no resolvable dsn, shell with no command). Templates render against template_vars. See reporting with mail, Sentry, shell, and webhook.

Each captured stream gets a StreamReader whose _read task reads stream.read(_READ_CHUNK) chunks and splits them on newlines. Each line is decoded by _decode_output_line (UTF-8, then the OEM code page on Windows, then UTF-8 with replacement). For live output it prefixes each line with streamPrefix (formatted with job_name/stream_name) and hands the lines in batches to a mirror writer thread, which writes them to the daemon's own sys.stdout/sys.stderr with _emit (bytes write with an ASCII-replacement fallback).

For retention it keeps the first saveLimit // 2 lines in save_top and the last saveLimit - saveLimit // 2 lines in a bounded save_bottom deque, incrementing discarded_lines for every evicted or never-saved line. A line longer than maxLineLength bytes is dropped with the warning ignored a very long line. join() awaits the reader task and returns the assembled text (top lines, an optional "[.... N lines discarded ...]" marker, then bottom lines) plus the discard count. See output capturing.

Retries are driven by JobRetryState (in job.py), the Cron.retry_state map (name -> JobRetryState), and the schedule_retry_job coroutine.

JobRetryState holds delay (current, initialized to initialDelay), multiplier (backoffMultiplier), max_delay (maximumDelay), a count of retries performed, an optional task (the pending schedule_retry_job task), and a cancelled flag. next_delay() returns the current delay, then advances it to min(delay * multiplier, max_delay) and increments count (exponential backoff capped at maximumDelay).

Flow:

• Arming. A scheduled fire ends the job's retry sequence and stores a fresh state through _begin_retry_sequence. That method calls cancel_job_retries(name) until no retry state is left for the job and, if maximumRetries is truthy, creates a JobRetryState(initialDelay, backoffMultiplier, maximumDelay) and stores it under the job name. For a job with no pool, launch_scheduled_job calls maybe_launch_job, and the launch calls _begin_retry_sequence under the per-job launch lock (_launch_job_locked), after the Forbid check and before a Replace launch cancels the running instances. For a pooled job, launch_scheduled_job calls _begin_retry_sequence before it queues the fire. A scheduled fire of a job with no pool that arrives while another launch of the same job holds the launch lock, such as a retry attempt that is starting its process, waits for the lock and leaves the retry state alone until it holds the lock. The concurrency policy then rules on the fire with that launch's run in place: Forbid drops the fire and the retry sequence stays, and Allow and Replace end the sequence and give the fire's run a fresh one. The RunningJob is given the state through the retry_state constructor argument.
• On failure (handle_job_failure, only when not shutting down): the job's output is logged, report_failure() runs, then the retry state is examined. If there is no state or it is cancelled, report_permanent_failure() runs and the flow stops. Otherwise any prior pending retry task is awaited (if done) or canceled, and the cap is checked. When state.count >= maximumRetries and maximumRetries != -1, retries are canceled (cancel_job_retries) and report_permanent_failure() runs. A maximumRetries of -1 means retry forever (the documented sentinel, enforced by JobConfig._validate_numeric_ranges). Otherwise a new retry is scheduled: state.task = asyncio.create_task(schedule_retry_job(name, state.next_delay(), state.count)).
• schedule_retry_job(name, delay, retry_num) logs the planned retry, await asyncio.sleep(delay), then re-fetches the job from self.cron_jobs. If the job has disappeared from the (possibly reloaded) config, it pops the stale retry state and returns; otherwise it calls maybe_launch_job(job).
• On success (handle_job_success): cancel_job_retries(name) clears any pending retry, then report_success() runs. A retry state that another run's launch installed and that has scheduled no retry stays in place. The record of an operator-canceled run applies the same test (_ends_retry_sequence) before it settles the retries as cancelled.
• cancel_job_retries(name) pops the state (no-op if absent), sets cancelled = True, and awaits or cancels the pending task. It takes a settle reason (default "superseded") for the durable retry state described next; the shutdown drain passes settle=None so a graceful stop settles nothing.

With a state: section configured, the machine gains a durable half (without one, the preceding flow is complete and retries die with the process):

• Durable retry records. schedule_retry_job asynchronously persists a pending record (stream retries/<job>, _persist_retry_pending) carrying the attempt number, the absolute notBefore deadline, and the job's fingerprint.job_digest. Every resolution appends a settled record on top (_persist_retry_settled) with a reason: launched, succeeded, superseded, cancelled, exhausted, owner-moved, job-removed, or a re-arm-time invalidation reason.

  When cross-node retry resume is active (a shared-topology store plus leader election) the stream carries a third kind: an ownership move writes a handoff record (_abandon_retry) so another node can resume the retry sequence. The claiming node's new pending record carries a claimedFrom field naming the previous owner. With a single-node store, an ownership change instead settles the retry as owner-moved.

  Just before the relaunch, _retry_consume_ok settles the pending record with reason launched: record-before-run, so a crash right after the launch cannot re-arm the attempt that already ran. Under onStoreUnavailable: degrade an unsettleable record does not block the launch (at-least-once, a bounded one-attempt replay window); under fail-closed the launch defers and re-checks, exactly like a closed cluster gate. A graceful shutdown deliberately does not settle: the surviving pending record is the restart handoff.

• Boot re-arm (_rehydrate_retries). When the backend first comes up, each configured retry-enabled job whose newest retry record is pending (and which has no live retry state or running instance; live activity takes precedence over the ledger) is re-armed at the persisted position. A fresh JobRetryState is replayed to the recorded attempt (count and the next delay as if the process never restarted) and the ordinary schedule_retry_job is scheduled with only the time remaining until notBefore (zero if it passed while down), so the cluster-gate re-check and cleanup for removed jobs behave just as they would without a restart.

  A record is settled instead of re-armed on:

  • a malformed record;
  • a per-job digest mismatch (config-changed, covering only that job's fingerprinted settings, so unrelated config edits do not discard the retry);
  • a disabled job;
  • an exhausted budget;
  • a record older than the job's startingDeadlineSeconds;
  • an @reboot job whose boot marker does not cover the current OS boot (superseded-by-reboot: the new boot run supersedes the previous retry sequence).

  Conversely, a covered marker re-arms the pending retry, which is what lets an @reboot maximumRetries: -1 keep-alive survive daemon restarts. Every ambiguous case settles: the bias is no-run over double-run.

For the operator-facing options, see failure detection and retries.

self.running_jobs is a defaultdict(list) mapping job name to the list of currently running RunningJob instances, so multiple concurrent runs of one job are tracked. maybe_launch_job consults concurrencyPolicy when an instance is already running:

• Allow: launch another instance (the list grows).
• Forbid: return without launching.
• Replace: for each currently running instance, set running_job.replaced = True before await running_job.cancel(), so the reaper recognizes the forced termination as a replacement rather than a failure (_handle_finished_job skips reporting/retries for replaced runs), then a fresh instance is launched.

With concurrencyScope: cluster, one more gate sits between this local check and the spawn: _claim_cluster_slot takes a TTL slot lease (slots/<job name>, state.slotTtlSeconds) in the shared state store, so Forbid and Replace also exclude instances on other nodes. maybe_launch_job is the single choke point, so every launch flavor (scheduled, retry, catch-up backfill, deferred @reboot, manual API start) is gated.

Against a live foreign holder, Forbid skips the launch (a warning names the holder) and Replace appends a fence-targeted cancel record to the slots/<name> stream and hands the relaunch to a background pursuit task: never waited on the scheduler path, bounded at twice the slot TTL, then it gives up with a warning (no-run over double-run). A store that cannot answer follows onStoreUnavailable: degrade (the default) launches with node-local enforcement only for that run, fail-closed skips.

The contract is at-least-once, not exactly-once: a holder that loses its lease to a store outage keeps running, so a peer that then wins the slot overlaps it. The slot releases when the job's last local instance finishes; a crashed holder's slot frees by TTL expiry.

The replaced flag is the single mechanism distinguishing an operator/scheduler-initiated replacement from an actual job failure. See concurrency and timeouts.

When a job has a statsd config, RunningJob builds a StatsdJobMetricWriter. _on_start sends {prefix}.start:1|g; _on_stop sends {prefix}.stop:1|g, {prefix}.success:{0|1}|g, and {prefix}.duration:{ms}|ms, where success is 0 if self.job.failed else 1 and duration is wall time between start and stop in milliseconds. For a monitored run the stop datagram also carries {prefix}.cpu:{ms}|ms and {prefix}.max_rss:{bytes}|g.

Each send_to_statsd call is one fire-and-forget UDP datagram, so there are two datagrams per run: job_started sends the single .start line, and job_stopped sends the stop metrics concatenated (newline-delimited) into one datagram. job_stopped also early-returns without sending if start_time is None (that is, job_started never ran). send_to_statsd sends through a pooled per-target UDP endpoint (StatsdClientProtocol), rebuilt every 60 s or after an error. _on_start catches OSError and _on_stop catches any Exception, and both log a warning so telemetry never crashes the scheduler. See metrics with statsd.

The pull-side sibling of statsd is the PrometheusMetrics registry in cronstable/prometheus.py, owned by the Cron object rather than the web app, so counters survive web-app restarts and cluster-manager rebuilds across reloads (they reset only on process restart; _apply_reload prunes series for jobs removed from the config).

Synchronous in-memory hooks in cron.py record cumulative state: _record_run (run outcomes and the duration histogram), schedule_retry_job (retries actually launched), the permanent-failure branches, reload_config and update_config (reload success/failure), and the leadership- and quorum-transition latches. Gauges are not stored at all: Cron._web_metrics calls the renderer, which computes them at scrape time from cron_jobs, running_jobs, last_run, and cluster_manager.view_dict(). If a backend read fails during a scrape, the cluster block degrades to cronstable_cluster_enabled and the transition counters instead of failing the whole scrape. See metrics with Prometheus.

• One process, with scheduling logic on one event-loop thread. Blocking work runs on the sized default thread pool, and output mirroring on its own writer thread.
• One long-lived scheduler coroutine (Cron.run) and one long-lived reaper coroutine (_wait_for_running_jobs).
• One short-lived wait() task per running job, plus one schedule_retry_job task per armed-and-pending retry.
• Reporters run concurrently per job but failures are isolated.
• Shutdown is cooperative: a signal sets _stop_event, the scheduler stops spawning, pending retries are canceled, and the reaper drains in-flight jobs before the process exits. The trigger is platform-specific: SIGINT/SIGTERM on POSIX vs Ctrl-C/Ctrl-Break (SIGINT/SIGBREAK) on Windows, both routed through platform.install_shutdown_handlers into signal_shutdown -> _stop_event.set(). But the finish-running-jobs-first behavior is identical on every OS.

For the broader operational picture, see production and container deployment. For the CLI surface, see the command-line reference.