Without clustering, instances using the same configuration each run every job independently. Adding a replica therefore duplicates scheduled work.

The optional cluster section lets instances verify over mutual TLS that they share a job set. Enabling leader election uses that agreement and quorum to select which replica runs scheduled jobs. It builds on the job-set ID and is implemented in cronstable/cluster.py (the ClusterManager, ClusterView, and the pure elect_leader/quorum_size functions).

The default gossip backend provides best-effort coordination. It keeps no shared state and needs no coordination service. During leadership changes or network partitions, a scheduled run may be skipped or duplicated. For fenced leadership, use cluster.backend: kubernetes or etcd: a shared store controls a single lease holder. If nodes already share a POSIX mount, cluster.backend: filesystem uses a lease file on that mount, with fencing dependent on NTP-bounded clock skew. Fencing controls leadership; it does not guarantee that every scheduled job executes exactly once. See choosing a backend and guarantees and trade-offs.

Terms used on this page. A job-set ID is an order-independent fingerprint of selected fields in the node's effective job configuration. Matching IDs indicate agreement on those fields, including jobs that are not currently executing. A quorum is a strict majority of the cluster, ⌊N / 2⌋ + 1 nodes. A node is quorate when it currently sees a quorum of agreeing members. Fenced means a shared store allows one lease holder at a time: Kubernetes and etcd, and filesystem while clock skew stays under about 2 s. A lease is a short-lived, auto-expiring claim on that store that the holder keeps renewing. A bridge is a set of members that two mutually-unreachable nodes can both still reach (the sides see the bridge, not each other). A thin bridge is one of fewer than quorum - 1 shared members, too thin for the two sides to confirm each other through it (see guarantees and trade-offs).

On this page: Quickstart · From one node to a cluster · Choosing a backend · At a glance · The job-set ID foundation · Cluster peer attestation · Leader election · Per-job policy · Distribution · Observing the cluster · Guarantees and trade-offs · Certificate rotation · Operating the lease backends · Trying it locally · Cluster sizing math

The fastest way to see a leader-electing cluster is the bundled three-node demo, which mints a throwaway CA and per-node certs for you:

docker compose -f example/cluster/docker-compose.yml up --build
# then open http://localhost:8080/ , :8081 , :8082 and watch the cluster panel

See trying it locally for the fuller walkthrough (failing the leader, losing quorum, drift, and per-policy behavior).

To build one by hand, each node gets the same job set plus a per-node cluster block. This is a complete leader-electing gossip node (cronstable-a of three):

cluster:
  listen: "0.0.0.0:8443"                  # this node's mTLS /peer listener
  tls:
    ca:   /etc/cronstable/cluster-ca.pem     # shared cluster CA (trust anchor)
    cert: /etc/cronstable/cronstable-a.pem       # this node's cert (SAN = cronstable-a)
    key:  /etc/cronstable/cronstable-a.key
  peers:                                  # every OTHER member -> size = 3
    - host: cronstable-b.internal:8443
    - host: cronstable-c.internal:8443
  nodeName: cronstable-a                      # unique, stable per node
  electLeader: true                       # only the elected leader runs jobs

(The certificate paths are yours to choose; the bundled compose demo, for example, mounts its generated certs at /certs/ca.pem / /certs/cronstable-a.pem.)

The quorum rule in one paragraph. List every other node in peers (never this node itself), so the cluster size is len(peers) + 1 and every node computes the same N. A node acts as leader only while it sees a quorum, a strict majority (⌊N / 2⌋ + 1), of agreeing members. So a 3-node cluster tolerates one node down. Give each node a distinct nodeName and a matching peer list.

The default single leader suits most clusters. Enable distribution: spread only when one leader cannot carry all the scheduled work.

On Kubernetes, skip the certs and peer list entirely and set cluster.backend: kubernetes so a Lease fences leadership instead. Nodes that already share a POSIX mount can likewise skip them with cluster.backend: filesystem and elect through a lease file on the mount (see choosing a backend).

To grow a single instance into a cluster without a double-run flag day, add attestation first, confirm the peers agree, then turn on election:

1. Pick a unique, stable nodeName per replica (the orchestrator's stable hostname, a StatefulSet ordinal, or an explicit value). Reusing one across nodes silently double-runs. See unique node names.

2. Provision the coordination material. For gossip, issue per-node certs from a dedicated cluster CA (see cluster peer attestation). For a lease backend, set up the Lease RBAC or etcd credentials, or point cluster.filesystem.path at the shared mount (operating the lease backends).

3. Add the cluster block with electLeader: false first (attestation only). Every replica still runs every job, so nothing changes operationally. You can confirm the peers reach agreed on GET /cluster before trusting the topology.

4. Run cronstable --validate-config to catch a bad peer list, blank TLS paths (the files themselves are read when the listener starts), or lease ordering at rest. See the command-line reference.

5. Roll the replicas one at a time, letting each converge to agreed before the next (change membership incrementally so majorities always overlap; see consistent cluster size).

6. Set electLeader: true. Enabling election only after the peers already agree makes the switch a clean transition, not a flag day.

To collapse back to one instance, scale to replicas: 1, or set electLeader: false (keep attestation) or remove the cluster block entirely. The gossip backend keeps no shared state, so there is nothing to clean up. A lease backend releases its lease on a graceful stop, so a survivor (if any) takes over at once.

cluster.backend selects how leadership is decided. The decision rule:

• Stay on the default gossip when you want zero-dependency replicas and can tolerate an occasional skip or double-run in narrow windows.
• Pick kubernetes or etcd when you already run that service and want fenced leadership through a shared store with a single lease holder.
• Pick filesystem when the nodes already share a POSIX mount (Amazon S3 Files / EFS / NFS) and you want fenced leadership with zero extra services.

All four support the same per-job policy (clusterPolicy), so switching backends does not require rewriting jobs. The coordination mechanism, failure behavior, and cluster status fields differ.

The per-backend config keys (cluster.kubernetes.*, cluster.etcd.*, cluster.filesystem.*) are in the configuration reference. Deployment, RBAC, auth/TLS, failure modes, and monitoring are in operating the lease backends. Runnable samples are in example/kubernetes/ and example/etcd/.

The rest of this page documents the gossip backend (the default) in depth. Its trust model and quorum math are specific to it. The clusterPolicy semantics in per-job policy, however, apply to every backend.

A job-set ID is an order-independent fingerprint of selected fields in each job's effective (post-merge) configuration. It excludes secret values and certain host-specific settings, and has a v1: prefix so ids are compared only within one scheme. See job-set ID for field coverage, secret handling, and where to find the id.

The cluster compares these ids to check agreement on the fingerprinted settings. The id is available on the standalone GET /job-set-id endpoint and in the dashboard header, and the daemon logs it at startup and whenever a reload changes it. clusterPolicy (described later) is part of the id, so two replicas that disagree on a job's policy show up as drift rather than silently coordinating differently.

With a cluster section but without electLeader, the cluster is observe-only: every instance still runs every job, and attestation tells you whether the peers agree. Each node serves a small GET /peer endpoint on a dedicated mTLS listener and periodically polls every configured peer, comparing job-set IDs.

cluster:
  listen: "0.0.0.0:8443"                  # the mTLS listener for this node
  tls:
    ca:   /etc/cronstable/cluster-ca.pem     # trust anchor for peer certificates
    cert: /etc/cronstable/this-node.pem      # this node's certificate
    key:  /etc/cronstable/this-node.key
  peers:
    - host: cronstable-b.internal:8443
    - host: cronstable-c.internal:8443
  nodeName: cronstable-a                      # optional; defaults to the system hostname
  interval: 30                            # optional; seconds per round (default 30)
  driftAfter: 3                           # optional; rounds before "drifted" (default 3)
  connectTimeout: 10                      # optional; seconds per peer request (default 10)
  electLeader: false                      # optional; run jobs on the leader only (see below)

Run cronstable --validate-config before deploying to catch a bad cluster section (peer list, blank TLS paths, lease ordering) at rest rather than at startup. The TLS files themselves are read when the listener starts. See the command-line reference. Address formats are part of that load-time validation: listen and every peers[].host must be host:port with a port in 1-65535, and an IPv6 literal must be written bracketed ([2001:db8::1]:8443). The bare form is rejected with a ConfigError up front rather than mis-splitting at the last colon and failing opaquely at connect time. Every tls path must be nonempty: a blank ca: (what a template renders for an unset variable) is a ConfigError at load rather than a /peer listener that requires no client certificate. The same checks apply to the cluster.observability mesh.

The trust model is deliberately small and keeps no shared state:

• mTLS is the membership boundary. A peer's certificate must chain to the configured ca, and (client side) match the host it was reached at, so only nodes the CA vouches for are ever attested. Standard TLS hostname verification provides that SAN pinning: the cert presented by cronstable-b.internal:8443 must carry cronstable-b.internal as a Subject Alternative Name.

  • The CA is the whole authentication boundary (cronstable trusts any cert it signs to assert its identity and gossip state), so it must be a dedicated, closed CA issued only to cronstable nodes, not a shared service-mesh or organization-wide CA. Any cert that CA admits can otherwise fabricate the /peer payload described later: fake agreement, trip the conflict gate, or suppress an @reboot job.
  • Provision the certificates from your own dedicated PKI (a private cert-manager issuer, an internal CA); cronstable only consumes them. The node uses the same per-node cert/key both to serve /peer and to authenticate as a client when it polls peers.
  • The daemon detects an in-place renewal of these files (same paths, new bytes) and applies it automatically, with no restart (see certificate rotation).

• Each node keeps its own view. No node is authoritative: two responsive nodes converge to the same picture, and any disagreement is itself the signal.

• Drift is debounced. A reachable peer whose id differs is only reported as drifted after driftAfter consecutive rounds, so a rolling deploy (a brief, legitimate mismatch) does not raise a false alarm.

Every peer in the GET /cluster view and the dashboard panel carries one of these statuses (the constants live in cronstable/cluster.py):

A failed round (unreachable or untrusted) neither advances nor resets the drift streak: the streak counts reachable-but-mismatched rounds. Only a confirmed agreement (an agreed round, or the benign self case) clears it, so a genuinely drifting peer cannot postpone its drifted label by flapping in and out of reach.

The /peer endpoint is served only on the separate mTLS listen address, never on the public web API. It returns a small JSON document with everything a polling peer needs:

• the reporter's node_name and job_set_id (the agreement key);
• a per-process instance_id, so a duplicate nodeName is distinguishable from a self-listing;
• the declared cluster_size and the distribution / elect_leader descriptors (the conflict gates);
• its own members view and the mutual_agreeing / quorate_vouched sets (bridge discovery and spread owner deferral);
• ran_reboot_jobs (deferred-@reboot de-duplication).

The full annotated payload, the per-field safety role, and the trust-model notes (any CA-admitted peer can read and could fabricate the whole member and agreement graph, so the cluster CA must be a dedicated, closed boundary, and the listener caps request size but not concurrent connections) are in architecture and internals.

Setting electLeader: true enables quorum-gated leader election using the same attestation. Replicas can then coordinate scheduled jobs from one shared configuration, subject to the guarantees and trade-offs described later:

cluster:
  listen: "0.0.0.0:8443"
  tls:
    ca:   /etc/cronstable/cluster-ca.pem
    cert: /etc/cronstable/this-node.pem
    key:  /etc/cronstable/this-node.key
  peers:
    - host: cronstable-b.internal:8443
    - host: cronstable-c.internal:8443
  electLeader: true

Each node independently elects, as leader, the lowest nodeName among the members it currently sees agreeing on the job-set ID, but only if that set is a quorum (a strict majority) of the cluster. Only the leader runs scheduled jobs. Manual runs through the API (POST /jobs/{name}/start) are deliberately not gated, so you can still trigger a job on any node.

Automatic retries re-check the gate before every relaunch. A transient fail-closed denial (lost quorum, a detected conflict, a rebuilt manager's still-converging view) defers the retry and re-checks it, while a confirmed ownership change stops the local retry sequence so this node does not launch the attempt after learning that another node owns it.

What happens to the pending attempt then depends on the state store. On a shared durable state store with leader election, the retry sequence is handed off: the old owner writes a durable handoff record, and the new owner resumes the remaining attempts. No cancelled run-history record is written because the sequence continues on the new owner. Without a shared store the retry is abandoned (a WARNING plus a cancelled run-history record). Retries for EveryNode and @reboot jobs never move between nodes. The conditions for deferring, abandoning, or transferring retries are documented in failure detection and retries.

• List every other member in peers, not this node itself. The cluster size is therefore len(peers) + 1, and the quorum is ⌊size / 2⌋ + 1. The peer lists must be consistent across nodes for every node to compute the same size and quorum. This is enforced at runtime, not an assumption.

• If you accidentally list a node's own address in its own peer list:

  • An entry the config load can prove is local (a loopback address on this node's port under a matching listen) is rejected or warned about up front.
  • Anything else (for example, the node's own routable IP) is recognized at runtime as self after its self-poll succeeds, never counts toward agreement, and is excluded from the cluster size.

  In a genuinely 3+-node cluster that is benign (logged once at INFO): it neither changes the effective N/quorum nor (because N stays equal to what other nodes declare) trips the size-consistency check described later. It is not harmless at the boundary.

  A self-padded "3-node" config that is really 2 nodes escapes the electLeader 2-node refusal and runs as the degenerate quorum-2-of-2 cluster (both nodes must be up, and any single failure stops all Leader jobs cluster-wide), which cronstable flags at runtime with a prominent WARNING. Remove the self entry rather than relying on the exclusion.

• An exact duplicate peers entry is likewise dropped at config load (the first occurrence is kept): the cluster size, and therefore the quorum, is derived from the peer list, so a copy-pasted peer would otherwise inflate N and cost fault tolerance.

• The computed size, quorum, elected leader, and whether this node is the leader are all shown at GET /cluster and in the dashboard panel.

Without shared state, the quorum gate gives best-effort at-most-once under one shared N. Two strict majorities of N cannot be disjoint, so under a clean network partition at most one side is quorate, and therefore (within about one poll interval) at most one leader exists.

That qualifier matters. A leader newly cut off from the majority keeps acting on its last, now-stale view until its own next poll, so for up to one interval a clean partition can briefly double-run a Leader firing rather than only skip one. The single-leader property reasserts after the cut-off node re-polls and stands down (see guarantees and trade-offs).

The price is liveness: a node that cannot see a majority deliberately stands down (runs nothing) rather than risk a second leader. A Leader job therefore runs on a given firing only while a majority of the cluster is up and mutually reachable.

The preceding safety argument assumes every node has a distinct nodeName. If two nodes shared one, each would compute itself as the lowest name in its live set and both would elect themselves: a silent double-run, exactly what election is meant to prevent. So nodeName uniqueness is a correctness requirement, not a convenience.

The daemon enforces it at runtime. Each process mints a random instance id at startup and reports it on /peer alongside its nodeName. That lets a node distinguish two cases that otherwise look identical:

• a benign self-listing (an operator put this node's own address in its own peer list), where the peer reports this node's name and its own instance id (status self); from
• a duplicate nodeName (a different process announcing this node's name), where the instance id differs (status conflict). A third node can likewise spot two distinct instances claiming one name.

This detection is best-effort. It relies on some node being able to see both copies, directly, or transitively by unioning peers' reported member lists (one hop). Two copies of a duplicated nodeName that sit in disjoint partitions (no single node can observe both, even transitively) cannot be reconciled, so each side stays quorate and both lead, the same residual class as a same-N membership swap (see consistent cluster size). So treat unique nodeNames as something to enforce (distinct cert SANs, the orchestrator's stable hostnames), not to detect at runtime alone.

While any conflict is visible, this node's Leader jobs fail closed (stand down) instead of risking a double-run, and the conflict is surfaced as a conflict flag on GET /cluster, a banner in the dashboard cluster panel, and an ERROR log line. It clears automatically after the duplicate is renamed: the gate is self-healing. PreferLeader is not gated on conflicts: it already accepts double-runs as the price of never skipping. The default nodeName (the system hostname) is already unique per host. Set an explicit, unique nodeName when several nodes might share a hostname (for example, identical container images without distinct hostnames).

The safety argument also assumes every node uses the same cluster size N. "Two strict majorities of N cannot be disjoint" is only true for a single N: two majorities of different sizes can be disjoint. But N is each node's own len(peers) + 1, and the job-set fingerprint deliberately covers job definitions only, not the peer list. So two nodes with divergent peer lists still see each other agreed, each reaches a quorum under its own N, and both elect themselves: a silent double-run. An ordinary cluster resize (say rolling 3 → 5 nodes) triggers exactly this: mid-roll, the old nodes still carry N = 3 (quorum 2) while the new ones carry N = 5 (quorum 3), so the old {a, b} and new {c, d, e} groups are each quorate and each run the Leader jobs.

The daemon closes this the same way as a duplicate nodeName. Each node reports its declared N on /peer, and any reachable peer that declares a different N is treated as a first-class conflict, whatever job set that peer runs: this node's Leader jobs fail closed until the cluster reconverges on one N. A node stands down when it can reach a peer that declares a different N, so a resize rolled out on a connected network does not double-run. Old-size and new-size groups that cannot reach each other can each still lead.

The comparison ignores job-set agreement on purpose. A resize rolled out together with a job change (a peers edit and a job edit in the same deploy) puts the old and new config generations on different job-set IDs, so the two sides see each other as syncing or drifted rather than agreed, while each side can still reach a quorum under its own N. Comparing every reachable peer's N catches that roll too: it stands every Leader job down on every node until every member of the new peers list declares the new N. A removed member that is still running sees the new N on the nodes it polls, so its own Leader jobs stay stood down until it stops. A pure job change at a fixed N declares one N everywhere and raises no conflict.

The conflict is surfaced as the size_conflict / conflicting_sizes fields on GET /cluster, a banner in the dashboard cluster panel, and an ERROR log line, and clears automatically after every node's peers agree on the member set. As with a nodeName conflict, PreferLeader is not gated: it already accepts double-runs as the price of never skipping.

Note: the check compares the declared size N. It does not detect a same-N but different-membership divergence (for example, swapping one peer for another while keeping the count). To stay safe, change membership one node at a time: a change of one member keeps the old and new majorities overlapping, so each step keeps the quorum gate's best-effort at-most-once behavior. Let each change converge (the dashboard shows agreed on every node) before the next.

A Leader job fires successfully only while a quorum is up and mutually reachable, so pick an odd size. Each odd size adds one failure of headroom (3 tolerates 1, 5 tolerates 2, 7 tolerates 3), while an even size needs the same quorum as the odd size below it yet has an extra node that can fail, so cronstable warns on even sizes (for N > 2). A 2-node cluster is strictly worse than a single replica (both must be up, with no failover upside), so cronstable refuses to start with electLeader and 2 nodes (a ConfigError). A 2-node cluster is fine for attestation-only.

Framed as expected skipped firings for an hourly Leader job (8760 firings/year), which is often the more intuitive view:

The binomial derivation and full availability tables are in the appendix on cluster sizing math at the bottom of this page.

If electLeader is configured but the cluster listener fails to start (bad cert files, a bad listen address, a port already in use), the node logs the error and keeps running, and each policy honors its own contract. Leader jobs fail closed and stay idle rather than falling back to running everything on every replica. PreferLeader jobs are never-skip and run anyway: a node with no manager is exactly the store-or-quorum-unreachable outage PreferLeader accepts a double-run to survive, and skipping would drop the job to zero runs on a fleet-wide start failure.

The operational consequence: a listener broken on every replica means every replica runs every PreferLeader firing, which is why PreferLeader is reserved for idempotent jobs. (EveryNode jobs are unaffected; see the next section.) The node logs a leadership transition each time it acquires or loses scheduled-job leadership.

The cluster-wide electLeader switch sets the default behavior. Each job can override it with clusterPolicy to choose how it handles unavailable leadership. No policy guarantees exactly-once execution: Leader may skip a run, PreferLeader may run it more than once, and EveryNode deliberately runs it on every node.

In the guarantee column, "fenced" is the hard, single-holder guarantee, "best-effort" admits the narrow gossip windows, and "may skip" / "may dup" name the side the policy gives up. The fuller per-backend wording is in guarantees and trade-offs.

jobs:
  - name: charge-cards          # must not double-charge; skip-tolerant
    command: ./charge.sh
    schedule: "0 * * * *"
    clusterPolicy: Leader       # the default; can be omitted

  - name: refresh-cache         # idempotent, but must not be skipped
    command: ./refresh.sh
    schedule: "*/5 * * * *"
    clusterPolicy: PreferLeader

  - name: rotate-local-logs     # inherently per-node
    command: ./rotate.sh
    schedule: "@daily"
    clusterPolicy: EveryNode

Notes:

• clusterPolicy is inert unless cluster.electLeader is on. Without election, every job runs on every instance regardless of its policy.

• When election is configured but no manager is running (for example, the listener failed to start), Leader jobs fail closed (skip), while PreferLeader jobs are never-skip and run anyway, on this and every other manager-less replica: the documented double-run cost, preferred to dropping the job to zero runs. EveryNode jobs are independent of cluster health, so they keep firing regardless.

• The active policy for each job (when election is on) is shown in the dashboard's job drawer and included in the GET /jobs payload. To keep the per-poll payload lean for the common single-instance case, clusterPolicy is omitted from GET /jobs when election is not configured.

• clusterPolicy is part of the job-set ID, so replicas that disagree on a job's policy surface as drift.

The decision for one node, one firing, is exactly:

election off  -> run (every node runs everything)
EveryNode     -> run (always, even if the manager failed to start)
no manager    -> Leader skips (fail closed), PreferLeader runs (never-skip; the leadership listener failed to start, which is the very outage PreferLeader accepts a double-run to survive)
conflict      -> skip (fail closed; a duplicate nodeName, a cluster-size disagreement, OR a coordination-policy mismatch is visible)
PreferLeader  -> run only if this node is the lowest reachable agreeing node
Leader        -> run only if this node is the quorum-gated elected leader

(The conflict row applies to Leader only. PreferLeader and EveryNode are gated on none of a duplicate nodeName, a cluster-size disagreement, or a coordination-policy mismatch. A coordination-policy conflict is a reachable peer advertising a different distribution or elect_leader setting, surfaced as policy_conflict: true with the differing descriptors in conflicting_policies. It is the third trigger of the umbrella conflict flag alongside conflict_names and size_conflict. Under distribution: spread, described next, the last two lines become "the per-job owner among the reachable agreeing nodes" and "the quorum-gated per-job owner" respectively.)

One transient exception to PreferLeader's never-skip applies to the gossip backend only. A node whose manager was newly (re)built (a cold boot, or a reload that changed the cluster section) holds its never-skip gates closed while its view is still converging: until every configured peer has been polled once, and (bounded by about two poll intervals) until the current-build agreeing peers re-attest this incarnation.

Without the hold, a fresh manager's blank view would elect itself on every node at once and double-run each due PreferLeader firing on a responsive cluster. The cost is the mirror image: a PreferLeader firing due inside that window is skipped, on every node when the held node is the rightful owner, because its peers still defer to it: transient and self-healing, the same fail-closed trade the quorum gate makes. A pending retry treats the hold as a transient denial (deferred, never abandoned).

@reboot fires once at startup, which is the one instant the cluster has not yet converged: no peer has been polled, so there is no quorum and no elected owner. Running a leader-gated @reboot job immediately would misfire: a Leader job would see no quorum and skip forever (@reboot never re-fires), and a PreferLeader job would see only itself on every node and run everywhere. So under electLeader an @reboot job with Leader or PreferLeader policy is deferred: held until the cluster converges, then run once on the owner that policy resolves to. The deferral covers only the boot-time window in which every node sees only itself. Which owner runs it, and whether it runs at all without a quorum, follows the job's policy exactly as for a scheduled firing:

• Leader runs on the quorum-gated elected owner. If no quorum ever forms, the deferred job does not run (the at-most-once trade: a skip is preferred to a double-run), and it also stands down while a nodeName/size conflict is visible.
• PreferLeader runs on the quorum-free availability owner (the lowest reachable agreeing node), so it always resolves to some node and runs even with no quorum (an isolated or minority node runs it itself), exactly mirroring PreferLeader's never-skip contract for scheduled jobs. The price, as ever for PreferLeader, is a possible double-run across a partition.

For @reboot work that must run on every node at boot (warming a local cache, announcing the node), use clusterPolicy: EveryNode, which is not deferred.

A deferred @reboot one-shot is never silently lost across a reload. Deferral only happens at the boot instant, so a job whose name momentarily disappears from the loaded config before the cluster converges (a templating glitch, or a remove-then-re-add seen mid-reload) is kept pending, not dropped, and runs after the name comes back. The launch is always gated on the name being present and still a Leader/PreferLeader @reboot, so:

• a job you deliberately remove from the config (and leave removed) never runs, because its name stays absent;
• a name reused for a different @reboot job runs the current definition, never the one captured at boot; and if the reused job is no longer a deferrable @reboot (it became EveryNode, or a real schedule), the original one-shot is considered gone and the new job is left to its own scheduling.

On a lease backend the same deferral applies, translated to lease vocabulary:

• A Leader @reboot runs on the lease holder, and skips while the store is unreachable.
• A PreferLeader @reboot runs on this node when the store is unreachable (a possible boot-time double-run).
• EveryNode is not deferred.

The cross-node "already ran" record that gossip advertises peer-to-peer is persisted in the lease store instead (a Kubernetes Lease annotation, an etcd sibling key, append-only records in the filesystem store's cluster/reboot-ran stream), scoped to the current job-set ID, so a failover holder does not re-run the one-shot.

That guarantee is closed from the read side too. A node that has newly gained leadership answers "did this one-shot already run?" conservatively, keeping pending @reboot one-shots deferred, until it has completed a fresh read-back of the persisted record. On Kubernetes the ran-set rides the very Lease read that wins leadership; the filesystem and etcd backends force the re-read before a new holder may answer "not ran". A store that cannot answer therefore delays a one-shot rather than risking a second run of one the previous leader marked moments before failing over.

Because the store outlives the processes, this shifts the semantics: a Leader @reboot runs once per job configuration, not once per boot. Restarting the whole fleet with an unchanged config does not re-fire it, because every node reads the record back and retires the one-shot without running it. It fires again only when the job set changes (a new job-set ID invalidates the stored record), so a warm-up or migration step that must run on every deploy cannot rely on a leader-gated @reboot here unless the deploy also changes the job set.

By default (distribution: single-leader) the single elected leader runs every Leader job, so the other replicas are pure standby for scheduled work. That is the simplest model, but on a busy cluster it makes the leader a hotspot while the rest idle.

Setting distribution: spread keeps the same quorum gate but replaces the one leader with per-job ownership: each leader-gated job is assigned to a single node by rendezvous (highest-random-weight) hashing of the job name against the agreeing members. Different jobs hash to different nodes, so the scheduled workload fans out roughly evenly across the cluster.

cluster:
  listen: "0.0.0.0:8443"
  tls:
    ca: /etc/cronstable/cluster-ca.pem
    cert: /etc/cronstable/this-node.pem
    key: /etc/cronstable/this-node.key
  peers:
    - host: cronstable-b.internal:8443
    - host: cronstable-c.internal:8443
  electLeader: true
  distribution: spread      # default is single-leader

What to know:

• Same best-effort guarantee. Spread keeps the quorum gate, so Leader jobs get single-leader's best-effort at-most-once behavior. In a converged view every quorate node computes the same owner for a job.

  The subtle case is a thin bridge, a quorate pair that share too few witnesses to confirm each other. Because the rendezvous winner is per-job, it can be exactly such an unconfirmable node, so the raw hash would let a peer that cannot see it self-own the job and double-run it, even though single-leader (whose winner is always the one global-lowest node, which everyone can see) would not.

  Spread closes this by folding the unconfirmed peers a quorate neighbor vouches for into each job's rendezvous and deferring to any that outrank it (two strict majorities of one N always overlap, so a co-owner you cannot see normally appears in a shared peer's /peer report, except across a thin bridge or while views converge). The price is the same fail-closed trade single-leader makes: a job whose owner no quorate peer can currently confirm stands down until the view converges.

  This is a load optimization. It does not change the overall best-effort guarantee. Leader jobs still skip without quorum, and PreferLeader still ignores quorum (its owner is computed over the reachable set, so an isolated node owns and runs all of its jobs, and it keeps its never-skip contract unchanged).

• Rendezvous hashing, not modulo. When a node leaves or joins, only its share of jobs is reassigned (to the next-highest-weight node). The rest stay put. A membership change is therefore minimally disruptive, unlike hash % N, which would reshuffle everything.

• Best with many or heavy jobs. Hashing is only roughly even, so with a handful of jobs the split is lumpy (several can land on one node). It helps when a single node cannot comfortably carry all the scheduled work. For light workloads the default single leader is simpler and equally correct.

• Keep it consistent. Every node must agree on distribution (as with the peer list and electLeader). A reachable peer that advertises a different distribution (or electLeader), whatever job set it runs, is treated as a first-class coordination-policy conflict: it surfaces as policy_conflict: true (with the differing descriptors in conflicting_policies) and, as the third trigger of the umbrella conflict flag, stands this node's Leader jobs down (fail closed) until every node reconverges on one policy.

  distribution is not part of the job-set ID (it is cluster config, not a job property), so a mismatch does not show up as drift. Treat it like electLeader and roll it out uniformly. It is inert without electLeader (and cronstable warns if you set it anyway).

The bundled three-node demo names its two scheduled leader-gated jobs tick-leader-only (Leader) and tick-prefer-leader (PreferLeader). With all three nodes up:

Under spread, tick-prefer-leader moves to cronstable-c while tick-leader-only stays on cronstable-a, so the two jobs run on different nodes. The owner is a deterministic function of the job name and the live member set, so it stays put until membership changes (then only the affected jobs move). You can confirm it live:

curl -s http://localhost:8080/jobs | python -m json.tool | grep -A1 clusterOwner
# under distribution: spread, each leader-gated job carries a "clusterOwner"
# naming the node that runs it; single-leader mode emits no clusterOwner (the
# leader from GET /cluster owns every Leader job)

Leadership transitions can page you as well as show up in the logs: the top-level notify: block fires a reporter (webhook, mail, …) on the leader_change event (this node acquired or lost scheduled-job leadership) and the quorum_loss event (this node left quorum, so its Leader jobs stood down). Both are also exported as the cronstable_cluster_leader_transitions_total / cronstable_cluster_quorum_transitions_total Prometheus metrics.

GET /cluster on the web/HTTP interface returns the current view as JSON. When no cluster section is configured it returns {"enabled": false, "peers": []}. Otherwise it returns the node's view:

{
  "enabled": true,
  "backend": "gossip",             // the active cluster.backend
  "node_name": "cronstable-a",
  "job_set_id": "v1:…",
  "cluster_size": 3,
  "quorum": 2,
  "elect_leader": true,
  "distribution": "single-leader", // or "spread"
  "conflict": false,               // umbrella: true if any conflict pauses Leader jobs
  "conflict_names": [],            // the duplicated nodeName(s), if any
  "size_conflict": false,          // true if a reachable peer declares a different N
  "conflicting_sizes": [],         // those divergent cluster sizes, if any
  "policy_conflict": false,        // true if a reachable peer declares a different distribution/elect_leader
  "conflicting_policies": [],      // those differing coordination-policy descriptors, if any
  "candidates_truncated": 0,       // nonzero: the fleet outgrew the candidate advertisement cap (see below)
  "quorate": true,                 // whether this node sees a quorum
  "leader": "cronstable-a",            // null when not quorate, or always in spread mode
  "is_leader": true,               // always false in spread mode (no single leader)
  "peers": [
    {"host": "cronstable-b.internal:8443", "status": "agreed",
     "node_name": "cronstable-b", "job_set_id": "v1:…",
     "last_seen": "2026-06-23T19:00:00+00:00", "last_error": null,
     "mismatch_streak": 0},
    {"host": "cronstable-c.internal:8443", "status": "unreachable",
     "node_name": null, "job_set_id": null,
     "last_seen": null, "last_error": "Cannot connect to host …",
     "mismatch_streak": 0}
  ]
}

In spread mode there is no single leader, so leader is null and is_leader is false. Use quorate to tell whether this node is running its owned jobs. The per-job owners appear as a clusterOwner field on each leader-gated job in GET /jobs.

The preceding JSON is the gossip shape. A lease backend returns a lease-shaped view instead: backend names the backend, peers is [], cluster_size/quorum are 1, and conflict/size_conflict/policy_conflict are always false. An extra lease block carries the holder and expiry: for kubernetes {name, namespace, identity, holder, expiry}, for etcd {electionName, identity, holder, leaseId, expiry}, for filesystem {path, electionName, identity, holder, fence, expiry} (fence is the store's monotonic takeover counter). There quorate means the node has a fresh read of the lease store (see operating the lease backends and the GET /cluster reference).

The web dashboard renders the same view as a cluster panel: a status dot per peer, the agreement tally, and (when election is on) the quorum count and this node's role (leader, follower, "no quorum", or, in spread mode, "spread (per-job owner)").

Beyond agreement and roles, the gossip exchange also carries observability payload: each node piggybacks a compact per-job run summary (running / enabled / next fire / last finished run) on its /peer response, capped so it can never push the response past the gossip byte limit.

Every node therefore holds a fleet-wide picture of what ran where that is at most one interval stale per peer, served as GET /fleet and rendered as the dashboard's fleet view, the consolidated view for spread mode, where each job's runs land on a different node. The summaries are display-only: election, quorum, and every run/skip decision ignore them, and a malformed summary from a peer degrades to "no data for that node". The lease backends exchange nothing node-to-node, so /fleet reports enabled: false there unless cluster.observability provides an overlay mesh.

Gossip as a secondary data plane (cluster.observability). The same gossip mechanism can carry more than election. Opt in and each node also gossips its whole-node CPU/memory (the GET /node numbers), so the fleet view shows every node's live load beside its runs, which matters in spread mode for watching work distribute.

The reading rides each /peer response as a small X-Cronstable-Node-Stats header rather than in the body, on full responses and bodyless 304s alike, so live load values never defeat the exchange's conditional 304 optimization: a sharing cluster's steady-state round still costs headers only, and each absorbed reading is as fresh as the last successful poll.

This is available to any backend: a kubernetes/etcd/filesystem cluster (which has no node-to-node channel of its own) can stand up a second, election-inert gossip mesh purely for this observability data, leaving election with the lease store. See cluster.observability for the config and the two shapes (an opt-in marker under backend: gossip, and a dedicated mesh for the lease backends). Like the run summaries, node stats are best-effort display data: a hostile or malformed peer payload degrades to "no data", never poisoning the view or any decision.

quorate is the field to alert on for every backend. On gossip it means this node sees a majority; on a lease backend, that it has a fresh read of the store. This section covers the gossip signals. The lease equivalents are in monitoring the lease backends.

Every signal you would alert on is exported by cronstable itself at GET /metrics (see metrics with Prometheus) as cronstable_cluster_* series: cronstable_cluster_quorate, cronstable_cluster_is_leader, cronstable_cluster_conflict{kind}, cronstable_cluster_peers{status}, cronstable_cluster_size / cronstable_cluster_quorum, plus the cronstable_cluster_leader_transitions_total and cronstable_cluster_quorum_transitions_total counters. Each mirrors a pre-derived field on GET /cluster (the statsd integration is still per-job). Useful alerts:

A node gossips the set of peers it can confirm are themselves quorate, and spread ownership is computed over that set. The set is capped so one peer feeding a node inflated gossip cannot push that node's own /peer body past the size every peer enforces, which would cost it its place in the quorum. The cap is 512 names, sized so a worst-case set (every name at the 256-byte field limit) still leaves half the response budget for the election fields.

Reaching it takes a fleet of several hundred bridge-discovered nodes. If you do, candidates_truncated carries the full count the node derived and the daemon logs a warning naming it. The dropped names are the highest-sorting ones, and every node drops the same ones, so single-leader election is unaffected: it reads the lowest name, which always survives. Only spread ownership degrades, and only for jobs whose owner sits in the tail.

The example alerts on the Prometheus page include the quorum rule and the split-brain check (sum(cronstable_cluster_is_leader) > 1).

A blackbox / JSON-exporter probe (Prometheus json_exporter, a Nagios check, or similar) scraping GET /cluster on every replica remains a valid alternative, and is the only source for the detail fields the metrics do not carry: per-peer last_seen and last_error, conflict_names, conflicting_sizes, and conflicting_policies. The leader's name surfaces only as the cronstable_cluster_leader_info{leader} label.

The same transitions are also logged (leadership and quorum changes, conflict onset at ERROR (clear at INFO), and per-peer untrusted/unreachable/drift at WARNING), so a log-based alert is a viable second source.

The best-effort guarantee admits narrow windows where two nodes each run the same Leader firing (a thin bridge, a >1-hop gossip gap, or mid-convergence). The conflict flag does not catch this: that flag is only for a duplicate nodeName or a size disagreement, and by construction the two transient leaders cannot see each other, so neither one's GET /cluster shows anything wrong (each reports is_leader: true). There is no single-node signal for it.

To detect it, scrape every replica's GET /cluster and alert when more than one calls itself the leader:

# across all replicas, count how many believe they are the leader
for url in cronstable-a:8080 cronstable-b:8080 cronstable-c:8080; do
  curl -s "http://$url/cluster" \
    | python -c 'import sys,json; print(str(json.load(sys.stdin).get("is_leader")).lower())'
done | grep -c '^true$'     # > 1 means a transient double-leader

Under distribution: spread there is no single leader, so instead compare the clusterOwner each replica reports per job (GET /jobs) and alert if any job has more than one distinct owner across the fleet. A non-idempotent job that must never double-run belongs on a fenced backend (kubernetes / etcd, or filesystem given NTP-bounded clocks) or a single replica, not the gossip backend.

The delivery guarantee each clusterPolicy gives depends on the backend. The per-policy, per-backend summary is the guarantee column of the per-job policy table, and the fuller wording follows.

On the lease backends "fenced" holds while the store is reachable. A store outage is the one window a Leader job skips and a PreferLeader job may double-run (see failure modes).

On filesystem the fence additionally assumes NTP-bounded clock skew: the lease expiry is compared across host wall clocks with two 1 s margins (the holder stops calling itself leader 1 s before its lease really expires; a challenger refuses to take over until the observed expiry is 1 s in the past by its clock), so two leaders need inter-host skew above the sum, ~2 s. Run NTP on every node that mounts the store, the same requirement durable state documents under "Clocks on shared mounts". The kubernetes/etcd takeover is judged on a single clock (the challenger's own, or etcd's server), so those two carry no such budget. EveryNode is never gated on any backend.

The gossip backend keeps no shared state and provides best-effort coordination. Each node acts on a view only as fresh as its last poll (interval), leaving narrow windows where behavior degrades:

• Immediately after a leader dies, a Leader firing may be skipped until the survivors notice (up to one interval) and re-elect.

• A leader partitioned away while still alive keeps electing itself on its last (now-stale) view until its own next poll fails (up to one interval), overlapping the majority's re-election, so a clean partition can briefly double-run a Leader firing, not only skip one. It self-heals after the cut-off node re-polls and stands down.

• Asymmetric or partial reachability. Two nodes that never agree with each other can each stay quorate through shared members that bridge them. The election uses that bridge to resolve the split: each side discovers the other through the shared members' gossip and, after it can confirm the other is itself quorate, the lower nodeName wins on both sides, so a bridge of at least quorum - 1 shared members collapses two would-be leaders back to one. A node only ever elects a leader it can confirm is itself quorate, so in a converged view a responsive majority is not silently stood down (it elects a node that does run). Two deliberate trades come with that liveness:

  • Two quorate nodes whose bridge is thinner than quorum - 1 shared members, are more than one gossip hop apart, or are still converging may each elect themselves and double-run a Leader job.
  • Symmetrically, because bridge confirmation is only as fresh as the witnesses' last gossip, a confirmed candidate that has since become isolated can briefly draw the majority into deferring to it, a transient skip until the stale gossip ages out (~1–2 intervals).

  spread behaves the same per job. To fail closed on the double-run (skip rather than double-run) instead would require a lease/consensus store; see the end of this section.

• While a resize is rolling out, nodes briefly disagree on the cluster size N; Leader jobs across the whole cluster stand down (fail closed) until every node's peers agree again, the at-most-once-preserving trade-off (see consistent cluster size).

• A PreferLeader job may double-run across a partition (that is the point of the policy: it never skips).

For fenced leadership, use a shared store: etcd, a Kubernetes Lease, or a shared mount through the filesystem backend with NTP-bounded clocks. The job's clusterPolicy still determines whether unavailable leadership causes a skipped run or permits a possible duplicate. A single replica avoids cross-replica duplicates but can still miss scheduled runs during downtime. Tuning the gossip interval shorter narrows the degraded windows at the cost of more polling traffic.

Rotation is automatic on the gossip backend: cronstable reloads its mTLS contexts when the mounted CA/cert/key change in place. On each config-reload pass (every minute) it compares the on-disk CA, cert, and key against what it loaded at startup and, if any changed, restarts the cluster manager to rebuild the TLS contexts with the new material. An in-place renewal (the cert-manager, Vault, or Kubernetes mounted-secret pattern of same paths, new bytes) therefore needs no manual restart. A detected change is dry-run loaded first, so a half-written cert observed mid-rotation is retried rather than tearing the cluster down. The etcd and Kubernetes backends run the same in-place rotation check on their client TLS files (cluster.etcd.tls.*, the kubeconfig's CA/cert/key) and rebuild the backend once the new files load. The filesystem backend uses no TLS.

For the operational runbooks (leaf-cert rotation, rolling the cluster CA with trust overlap, and recovering from an untrusted cascade), see cluster certificate operations.

The kubernetes, etcd, and filesystem backends replace the gossip protocol with a shared store that controls a single lease holder while the store is reachable. This provides fenced leadership; the filesystem backend also requires NTP-bounded clock skew. They share a base implementation, cronstable.leadership.LeaseBackend, and differ in how they access their stores. This section covers how they elect, how to deploy each, their failure modes, and how to monitor them. The config keys are in the configuration reference.

The built-in kubernetes and etcd transports use HTTP APIs through the core aiohttp dependency, with HTTPS supported: the Kubernetes apiserver's REST API and etcd's v3 gRPC-gateway JSON API. The filesystem backend accesses a directory through the standard library. These transports add no new dependencies, including no grpc/protobuf wheels, preserving cronstable's architecture support.

The Kubernetes backend can optionally use the official kubernetes client when it is installed (pip install cronstable[kubernetes]): cluster.kubernetes.clientLibrary: auto (the default) prefers it when importable and otherwise falls back to the hand-rolled REST transport, so the choice is automatic per architecture (library requires the client, http forces the hand-rolled path). etcd always uses its own v3 JSON gateway, so it has no optional client.

• No peer list, no peer mTLS, no quorum math. The store decides leadership, so the gossip-only keys listen, tls, peers, interval, and driftAfter are ignored (the daemon logs one startup advisory naming the ignored keys). A lease backend always elects, so electLeader is implied and electLeader: false is likewise ignored with an advisory (configuring a lease backend is opting into leadership). The cluster is logically a single holder (cluster_size / quorum report 1), and GET /cluster returns lease status fields. Its full field list is under observing the cluster.

• The lease is the fence, not a name. Leadership is decided by the lease, so a duplicate node identity cannot make two nodes both lead the way it can on a naive lease holder. On etcd the fence is the bound lease id (only the node whose own lease backs the election key leads), and Kubernetes and filesystem write a per-process token into the holder they record (<identity>#<token>) so two nodes sharing a nodeName still write distinct holders. You should still give each node a stable, unique name for clear observability (see node identity).

• Local-expiry safety. A holder only calls itself leader until a locally-computed lease deadline (renew time + duration, minus a small clock-skew margin), so a node whose renew loop stalls self-demotes without a network round-trip. On Kubernetes and etcd, two holders therefore do not act at once. On filesystem this holds only while inter-host clock skew stays within the ~2 s budget.

• PreferLeader keeps never-skip semantics. A node that currently cannot reach the coordination store runs a PreferLeader job anyway (it may double-run). A responsive follower that can see the holder defers. Leader stays fail-closed: it skips while the store is unreachable. This is the deliberate, documented trade: a PreferLeader job never skips, at the cost of a possible double-run during a store outage.

• distribution: spread (an opt-in, gossip-only mode that fans jobs across nodes instead of one leader; see distribution) is rejected at config load on a lease backend (a hard ConfigError, not a silent fallback). A single lease holder cannot also be a per-job owner, so use the gossip backend if you need per-job spread.

All three reduce leadership to "hold a short-lived lease on a shared object and keep renewing it":

• Kubernetes drives a single coordination.k8s.io/v1 Lease. The holder writes its identity into spec.holderIdentity and refreshes spec.renewTime every retryPeriodSeconds; if it stops, another node observes the lease go stale and takes it over, the standard client-go leader-election algorithm. The takeover deadline is anchored to the challenger's own clock from the moment it first saw the record (client-go's observedTime), so it is immune to clock skew between holder and challenger: a fast clock cannot steal a freshly-renewed lease.

• etcd creates a single key (electionName) with a create-if-absent transaction (compare CREATE revision == 0), bound to a short-TTL lease it keeps alive. At most one node's transaction wins; if the holder dies the lease expires, etcd deletes the key, and another node's transaction wins. etcd's server enforces the TTL.

• Filesystem takes the same flock-guarded, fence-counted TTL lease the durable state store provides, on a lease file in the shared directory. The holder renews it under the lock every max(1s, ttl / 3); if it stops, the written expiry passes and a challenger takes the lease over, bumping the fence counter. The takeover compares the challenger's wall clock against the expiry the holder wrote, a cross-host clock comparison, so unlike the two preceding backends it carries a ~2 s clock-skew budget (see filesystem later).

All three gate is_leader() on a locally-computed lease deadline (renew/keepalive time + duration − a 1 s clock-skew margin), so a node whose renew loop stalls self-demotes with no network call: that local expiry, not the store, is what guarantees two holders never act at once (on filesystem, together with the challenger-side margin and NTP-bounded clocks).

Separately, is_quorate() reflects whether the node has a fresh successful read of the store (within one lease duration / TTL). When it goes stale, Leader jobs fail closed and the never-skip PreferLeader default runs the job anyway.

Precision note. The clock-skew margin applies to is_leader's lease deadline, not to the is_quorate freshness window (the full duration / TTL, no margin). So a follower's view of who leads can briefly lag a dead holder by up to one freshness window, while the would-be leader has already self-demoted: bounded and self-healing, and PreferLeader never skips during it.

Leadership is fenced on the lease, but each node still carries an identity:

• etcd: the value written at the election key is cluster.nodeName (there is no separate etcd.identity key). Leadership is decided on the bound lease id, not this string, so even two nodes sharing a nodeName cannot both lead (only the node whose lease backs the key is leader); a shared name only makes the displayed holder ambiguous.

• Kubernetes: cluster.kubernetes.identity (defaulting to nodeName) is the human-readable holder, and cronstable appends a per-process token to the holderIdentity it writes (<identity>#<token>), so two nodes sharing an identity still write distinct holders and cannot both renew. The dashboard and GET /cluster strip the token back to the readable name (so kubectl get lease … -o jsonpath='{.spec.holderIdentity}' shows the suffixed form, while the dashboard shows the clean name).

• filesystem: the holder written into the lease file is always <nodeName>#<12-hex per-process token>, so two nodes sharing a nodeName (or a restarted daemon) can never adopt or renew each other's lease; as on Kubernetes, the dashboard and GET /cluster strip the token back to the readable name, and no run/skip decision ever string-compares it.

A duplicate identity therefore does not break the fence, but give each node a stable, unique name anyway so the holder shown in the dashboard, kubectl get lease, or etcdctl get unambiguously names one node. In Kubernetes both a Deployment and a StatefulSet give each pod a unique hostname; a StatefulSet's ordinals make the holder name predictable across restarts.

No mTLS, no peer list, no odd-replica rule: the apiserver is the authority, not a quorum, so a plain Deployment with any replica count works:

cluster:
  backend: kubernetes
  kubernetes:
    leaseName: cronstable-leader      # the Lease object the replicas contend for
    # leaseNamespace: null         # default: the pod's own namespace
    leaseDurationSeconds: 15        # failover happens within ~this long
    renewDeadlineSeconds: 10        # must be < leaseDurationSeconds
    retryPeriodSeconds: 2           # renew/observe cadence; must be < renewDeadlineSeconds
    # clientLibrary: auto          # auto | http | library (see below)

The three timings are cross-checked at config load (a ConfigError otherwise): renewDeadlineSeconds > 0, leaseDurationSeconds > renewDeadlineSeconds, 0 < retryPeriodSeconds < renewDeadlineSeconds, and renewDeadlineSeconds + retryPeriodSeconds < leaseDurationSeconds (so a renew that narrowly misses its deadline still leaves a retry inside the lease window). Run cronstable --validate-config to check them before deploying. See the command-line reference.

• RBAC (required). The backend needs get (observe) and update (renew / take over / release) on the one named Lease, plus create on leases to first acquire it. create cannot be scoped by resourceNames, so it is a second, unscoped rule:

  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  rules:
    - apiGroups: ["coordination.k8s.io"]
      resources: ["leases"]
      resourceNames: ["cronstable-leader"]   # keep in sync with cluster.kubernetes.leaseName
      verbs: ["get", "update"]
    - apiGroups: ["coordination.k8s.io"]
      resources: ["leases"]
      verbs: ["create"]                    # create cannot be scoped by resourceNames

  A ready-to-apply ServiceAccount + Role + RoleBinding + 3-replica Deployment is in example/kubernetes/deployment.yaml. Run cronstable --validate-config on the config before applying the manifests. See the command-line reference.

• Credentials. In-cluster, the backend uses the pod's service-account token, CA, and namespace file automatically (leaseNamespace defaults to the pod's own namespace). For out-of-cluster / local testing set cluster.kubernetes.kubeconfig (and optionally apiServer to override the server URL).

• Transport (clientLibrary). auto (default) uses the official kubernetes client when it is importable (pip install cronstable[kubernetes]) and otherwise falls back to a hand-rolled apiserver REST transport over the core aiohttp dependency, so on an architecture without the client it still works. http forces the REST transport, and library requires the official client and fails the backend start (the node's Leader jobs then fail closed) if it is not importable. Both transports drive the same Lease, so the choice is purely about which client code runs.

• Failover timing. A holder that dies is replaced within ~leaseDurationSeconds. On a graceful shutdown the holder clears holderIdentity so a survivor takes over immediately. Shorter durations fail over faster at the cost of more apiserver traffic. All three timings are ordered and enforced at config load: leaseDurationSeconds > renewDeadlineSeconds, retryPeriodSeconds < renewDeadlineSeconds, and renewDeadlineSeconds + retryPeriodSeconds < leaseDurationSeconds (so a renew that narrowly misses its deadline still leaves a retry inside the window).

Point the backend at one or more etcd endpoints (tried in order for failover):

cluster:
  backend: etcd
  etcd:
    endpoints:
      - http://etcd-0:2379
      - http://etcd-1:2379
    electionName: cronstable/leader   # the key; its value is the holder's nodeName
    ttl: 15                         # lease TTL, seconds (>= 3; keepalive every ~ttl/3)
    # For an auth-enabled cluster, use https:// endpoints and add:
    # username: root
    # password:
    #   fromEnvVar: ETCD_PASSWORD
    # tls:
    #   ca: /etc/etcd/ca.pem
    #   cert: /etc/etcd/client.pem
    #   key: /etc/etcd/client.key

• Transport. Speaks etcd's v3 gRPC-gateway JSON/HTTP API directly over aiohttp: no etcd3/grpc client, so no extra dependency and full architecture coverage. There is no optional official-client extra for etcd.

• Authentication. For an auth-enabled cluster set username and resolve the password from exactly one of value / fromFile / fromEnvVar (like web.authToken); a configured-but-empty source fails closed at load. The auth token is obtained at startup and re-fetched automatically when it expires (cronstable re-authenticates on an etcd 401), so a token TTL does not wedge the backend. Always pair a username with a resolvable password. Authentication requires every endpoint to be https://: a username or password with any http:// endpoint is a configuration error at load.

• TLS. For https:// endpoints set tls.ca (and tls.cert / tls.key for client-cert auth). http:// and https:// endpoints are detected per-URL.

• Failover timing. A dead holder is replaced within ~ttl. On a graceful shutdown the holder revokes its lease, deleting the key at once for immediate failover. The value at electionName is the holder's nodeName, so etcdctl get cronstable/leader shows who leads. ttl must be >= 3 (a smaller value is rejected at config load). etcd may grant a smaller TTL than requested (its --min-lease-ttl setting, or server load), which the backend honors: a smaller server-granted TTL narrows the effective leader window accordingly.

• Small TTLs and etcd latency. The renew round budgets its per-request timeout from the TTL (roughly (ttl - ttl/3 keepalive - 1s skew margin) / 5 per POST), so a ttl below 9 leaves each renew POST less than about one second. Config load then logs a one-time, non-fatal advisory: if a single round-trip to etcd is slower than that budget (for example, a cross-AZ/region endpoint), every renew round times out and the node treats a reachable etcd as unreachable, so its Leader jobs fail closed. Raise cluster.etcd.ttl unless etcd is local and low-latency.

Point the backend at a directory on a mount every node shares (an Amazon S3 Files / EFS / NFSv4 mount). The mount is the store, so there is no service to deploy and nothing beyond the standard library at work:

cluster:
  backend: filesystem
  filesystem:
    path: /mnt/shared/cronstable       # the directory the election lease lives in (required)
    # electionName: cluster/leader  # the lease's name inside the store
    # ttl: 15                        # lease time-to-live, seconds (>= 3; renewed every ~ttl/3)
    # deploymentId: null             # namespace inside the store; null -> "default"
    # topology: auto                 # auto | single-node | shared (assert shared on Windows/macOS)
  nodeName: cronstable-a

The full key table is in the configuration reference. deploymentId and topology carry the same semantics as their state: counterparts. ttl must be >= 3 (a smaller value is rejected at config load: the holder keeps the lease only until ttl minus the clock-skew margin and renews every max(1s, ttl / 3), so a smaller ttl would make a node treat its own freshly-won lease as expired).

• How it elects. Leadership is the durable state store's flock-guarded, fence-counted TTL lease: one small file under the store's leases/ directory, taken and renewed under an advisory lock, with a monotonic fence counter that bumps on every takeover. The holder string written into the lease is <nodeName>#<12-hex per-process token> (see node identity).

  The holder renews every max(1s, ttl / 3) and calls itself leader only until a local time.monotonic deadline anchored before the renewing write, minus a 1 s clock-skew margin, so a stalled renew loop self-demotes with no I/O. A challenger refuses to take over until the observed expiry is a further 1 s in the past by its own clock.

• Clocks: run NTP on every node. Those two 1 s margins are the whole skew budget: the takeover compares wall clocks across hosts, so two simultaneous leaders need inter-host skew above their sum, ~2 s. NTP keeps real fleets orders of magnitude below that. It is the same requirement durable state documents under "Clocks on shared mounts". This backend does not inherit the kubernetes/etcd skew immunity (those judge takeover on a single clock).

• The lock-fidelity probe (a hard refusal). start() refuses a store whose locks are demonstrably fiction, raising a ConfigError of the form cluster.backend filesystem: refusing to elect over <path>: … instead of silently electing two leaders. Two same-host checks:

  • A functional probe: two descriptors of one file must contend on a non-blocking exclusive lock. A mount that grants both has no-op locks, for example … grants two exclusive locks on one file (its locks are no-ops).
  • On Linux, a mount-option sniff: an NFS mount carrying nolock or local_lock=flock/local_lock=all satisfies flock host-locally, so its locks are host-local and cannot fence other nodes.

  A refused start leaves the manager unbuilt, so Leader jobs fail closed, the safe direction (see failure handling).

• The Windows/macOS residual. Both probe checks run on one host, so a mount whose locks are real locally but not propagated across hosts is not detectable on platforms without /proc/mounts (Windows, macOS). There topology: auto cannot probe and resolves single-node: the store then warns that "its locks only exclude processes on THIS host" and asks you to set cluster.filesystem.topology: shared if the directory really is a shared mount.

  On a Windows shared mount, start() logs a loud advisory: "cross-host lock fidelity cannot be verified on this platform (no /proc/mounts); the election is safe only if the mount honours byte-range locks across hosts". The residual safety rests on that operator assertion.

• What quorate means here. A fresh positive observation of the lease store within one ttl: only an operation that returned an actual lease (a renew, an acquire, or a read that parsed a live holder) counts. A None from the lease API is deliberately not contact (the store fails closed, conflating "denied" with "unreadable"), so a malfunctioning store lapses quorum: Leader jobs fail closed and never-skip PreferLeader jobs run anyway (they may double-run across the outage), the same posture as the other lease backends' store outage. Each failed round logs cluster: filesystem election round failed: … at WARNING.

• Sharing a directory with state:. Legal, and recommended when both are used (same path and deploymentId): the election's embedded store runs none of the scheduler's durable-state chores (no manifests, no GC, no counters), the stream namespaces are disjoint, and election lease files are never deleted at all. The scheduler's GC reclaims only the per-run DAG advance lease class, so a lease whose fences other nodes may still compare against is left alone at any age.

• @reboot one-shots. Same semantics as the other lease backends: once per job configuration, not once per boot. The "already ran" set is persisted as append-only records (one per newly-ran job, tagged with the job-set ID) in the store's cluster/reboot-ran stream. Readers union the records matching the live job-set ID, and the stream is pruned to the newest 512 records (a documented bound). The set is re-read every 60 s and immediately on gaining leadership, so a failover leader never re-runs a one-shot the old leader marked moments before.

• Failover timing. A dead holder is replaced within ~ttl (plus the 1 s challenger margin). On a graceful stop the holder releases the lease best-effort so a survivor takes over at once. TTL expiry is the fallback.

• Monitoring. GET /cluster returns the lease-shaped view with backend: "filesystem" and a lease block {path, electionName, identity, holder, fence, expiry}. fence is the store's monotonic takeover counter, so a bump marks each real leadership change. No new Prometheus families; see monitoring the lease backends. A successful start logs cluster: filesystem election ready at <path> (election=…, identity=…, ttl=…).

• Store unreachable (apiserver / etcd down, an unresponsive or unmounted shared mount, or partitioned from this node). Within one duration / TTL the node's is_quorate() goes stale, so its Leader jobs fail closed (skip) and its PreferLeader jobs run anyway (the never-skip rule: they may double-run across the outage). When the store returns, leadership re-establishes within ~one duration. This is the lease backends' one double-run exposure, and it is the documented PreferLeader trade, not a fence break.

• A fenced backend never shows two simultaneous leaders. Unlike gossip there is no thin-bridge / convergence double-run window for Leader jobs, so scraping every replica for is_leader: true (the gossip double-run check) never finds two while the store is functioning normally. On filesystem this holds while inter-host clock skew also stays inside the ~2 s budget (see filesystem). A store with fake locks never gets this far: it is refused at start with a ConfigError ("refusing to elect over …") and Leader jobs fail closed.

• Kubernetes optimistic concurrency. Writes carry the observed resourceVersion; a node that loses the race gets an HTTP 409, stands down for that round, and retries. The graceful release is best-effort: if it races a concurrent write the handover may instead wait out leaseDurationSeconds.

• etcd lease loss. If a keepalive reports the lease gone (TTL ≤ 0) the holder re-grants a fresh lease and re-campaigns, becoming leader again only if it re-wins the key.

• etcd TTL below the floor. If the server-honored TTL ever drops below the usable floor (3 s), the fence collapses to ~zero and this node's Leader jobs fail closed (safe, but it can no longer lead). The daemon logs a one-time warning on that transition, and a recovery after the granted TTL returns above the floor. Check etcd's --min-lease-ttl and load if you see it.

quorate is the field to alert on for every backend. On a lease backend it means this node has a fresh read of the store (on filesystem, a fresh positive read: an operation that returned a lease), not that it sees a majority.

The gossip alerts on the monitoring table (per-peer status, agreed-peers-vs-quorum, untrusted certs, the multi-leader scrape) do not apply: a lease view has peers: [], quorum: 1, and conflict is always false, and a fenced backend never reports two leaders (on filesystem, within its ~2 s skew budget). So the signal that matters is quorate:

Probe GET /cluster on each replica, or watch the store directly (kubectl get lease <name> -o jsonpath='{.spec.holderIdentity}', etcdctl get <electionName>; the filesystem election lease is a small JSON file under the store's leases/ directory on the mount). The same leadership transitions are logged.

The lease-backend @reboot behavior is covered inline in the main @reboot section.

For running multiple replicas on Kubernetes (both backend: kubernetes and backend: gossip on a StatefulSet), see production and container deployment.

The repository ships a ready-to-run three-node cluster in example/cluster/docker-compose.yml. It generates a throwaway cluster CA and per-node certificates, brings up three mutually-attesting nodes with electLeader: true and one job of each clusterPolicy, and publishes each node's dashboard on a separate port (8080/8081/8082) so you can watch leadership move when you stop the leader:

docker compose -f example/cluster/docker-compose.yml up --build
# then open http://localhost:8080/ , :8081 , :8082 and watch the cluster panel
docker compose -f example/cluster/docker-compose.yml stop cronstable-a   # watch leadership move to cronstable-b

The compose file's header comments document the full set of things to try (losing quorum, drift, and the per-policy job behavior).

A full showcase. For the fullest end-to-end demo (distribution: spread, all three clusterPolicy values, and mTLS together), the repository ships example/grand-tour/docker-compose.yml; its walkthrough is in example/grand-tour/README.md.

To watch distribution: spread fan real load across the cluster, the repository also ships example/cluster-large/docker-compose.yml: ten nodes (dashboards on ports 8080–8089) running a larger job set with several CPU-heavy jobs, defaulting to spread. Each node's config is generated from environment variables by a small entrypoint, so there are no per-node files to maintain.

docker compose -f example/cluster-large/docker-compose.yml up --build
docker stats     # watch CPU spread across the nodes (a few cores busy on several nodes)

# contrast: pin everything to one leader and watch a single node light up
DISTRIBUTION=single-leader docker compose -f example/cluster-large/docker-compose.yml up -d

(Ten is an even size, so the nodes log the even-size warning; that is expected and called out in the file. Quorum is 6, so it tolerates four failures.) Its header comments list how to inspect per-job owners and fail nodes.

To try a lease backend instead of gossip, example/etcd/ ships a docker-compose.yml with a single etcd plus two cronstable instances (backend: etcd) and one job of each clusterPolicy:

docker compose -f example/etcd/docker-compose.yml up --build
# exactly one instance leads; fail it over and watch the other take over within ~ttl:
docker compose -f example/etcd/docker-compose.yml stop cronstable-a
docker compose -f example/etcd/docker-compose.yml exec etcd etcdctl get cronstable/leader

For the Kubernetes Lease backend, example/kubernetes/ has the RBAC + Deployment to apply against any cluster (k3d/kind for local).

• job-set ID: the fingerprint the cluster compares (exact field coverage, no-secrets guarantees, every surface it appears on).
• Configuration Reference: the cluster section and clusterPolicy option schema.
• HTTP Control API: the GET /cluster endpoint.
• Web Dashboard: the cluster panel and per-job policy display.
• Production and Container Deployment: running multiple replicas under Kubernetes.
• Architecture and Internals: where cluster.py fits in the daemon.

This expands the practical rule in sizing the cluster with the underlying probability. A Leader job fires successfully only while a quorum is up and mutually reachable. If each node is independently up with probability p, and the quorum is q = ⌊N/2⌋ + 1, then the chance a given firing runs is the probability that at least q of N nodes are up, a binomial tail:

P(runs) = Σ (from k=q to N)  C(N, k) · p^k · (1 − p)^(N − k)

The table evaluates that for a few realistic per-node availabilities, as a fraction and as "nines" (−log₁₀(1 − P)). "Tol." is how many simultaneous node failures the size survives (N − q).

How to read it:

• Prefer odd sizes. Each odd size adds one failure of headroom over the previous odd size: 3 tolerates 1, 5 tolerates 2, 7 tolerates 3.

• Even sizes are equal-or-worse, never better. N=4 still needs a quorum of 3, so it tolerates the same single failure as N=3, but it has an extra node that can fail, so its P(runs) is actually slightly lower (0.99941 vs 0.99970 at p=0.99). The daemon warns on even sizes for exactly this reason.

• 2 is worse than 1. A 2-node quorum is 2, so both must be up: P = p² (0.9801 at p=0.99), below a single node's p (0.99), with no failover upside. The daemon refuses to start with electLeader and a 2-node cluster, raising a ConfigError ("...strictly worse than a single replica..."). The same 2-node cluster is fine for attestation-only (without electLeader).

Caveats on the math:

• It assumes independent failures. Correlated failures (a bad config push, a shared host, zone, or power domain) break that assumption, and then more nodes can even hurt. Spread the nodes across independent failure domains; p should be realistic uptime including deploys and restarts, not raw hardware MTBF.
• It only models "is a quorum up". It does not capture the narrow membership-change windows in guarantees and trade-offs (a firing may still slip through them), nor PreferLeader duplication, which is about partitions rather than node-up probability.