Production Deployment - ptweezy/cronstable GitHub Wiki

Production and container deployment

This page covers running cronstable in hardened Linux container and Kubernetes environments: the published image, the security context it is built to satisfy, the Kubernetes Deployment manifest, building a custom image from the published image, and the few cases that require a writable path. The published Docker image is Linux-only. See installation for the package/binary install methods and the HTTP control API for the optional web interface. For native Windows deployment (the MSI, winget, the one-directory zip, or the standalone .exe), see running on Windows.

Why cronstable fits a locked-down pod

At runtime the daemon only reads its configuration and secrets and writes its output to stdout/stderr. It needs no writable working directory, no temp files, and no log files. It runs in the foreground, makes no exotic syscalls, and requires no special privileges. The published image runs under a restricted Kubernetes/container security context with no writable paths or elevated privileges required.

The single exception is the optional per-job user/group switching feature, which calls os.initgroups/os.setgid/os.setuid (cronstable/job.py) and so requires the daemon to run as root. It is unavailable in the non-root published image. If you do not use that feature, drop root entirely. See commands and environment for the feature itself.

Per-run resource limits (resourceLimits) also need more than the published image grants: root and a cgroup that cronstable can write. See resource limits in a container.

The published image

Prebuilt, multi-architecture (linux/amd64, linux/arm64, linux/386, linux/arm/v7, linux/ppc64le, linux/s390x, and linux/riscv64) images are published to the GitHub Container Registry on every release.

For compatible x86-64-v3 hosts, the explicit -amd64v3 tags are recommended (for example, latest-amd64v3 and latest-alpine-amd64v3). Existing tags retain their current platform coverage and CPU requirements. The deployment examples below use those existing tags; select the v3 tag for a compatible host, including the CPU features exposed to a VM. See CPU requirements and variant selection.

Property Value
Registry/image ghcr.io/ptweezy/cronstable
Tags the release version (for example, 1.2.60), plus latest
Base python:3.14-slim (multi-stage; runtime stage copies a self-contained venv)
User 65534:65534 (nobody), set with USER in the Dockerfile
Entrypoint cronstable
Default command -c /etc/cronstable.d
Config path /etc/cronstable.d (a file or a directory of *.yaml/*.yml files)
Env PYTHONUNBUFFERED=1, PYTHONDONTWRITEBYTECODE=1, PATH=/opt/venv/bin:$PATH

PYTHONUNBUFFERED flushes stdout/stderr immediately (where cronstable logs) and PYTHONDONTWRITEBYTECODE prevents .pyc writes. Both matter under a read-only root filesystem.

A second python:3.14-slim-based Dockerfile under example/docker/ exists for demonstration. It pip installs cronstable and is not non-root. For production, use the published GHCR image (or base your own image on it, described later), not the example.

Quick start

Mount a crontab read-only and run:

docker run --rm \
  -v "$PWD/cronstable.yaml:/etc/cronstable.d/cronstable.yaml:ro" \
  ghcr.io/ptweezy/cronstable:latest

For production, pin a specific version instead of latest (for example, ghcr.io/ptweezy/cronstable:1.2.60).

Baking config into your own image

If you would rather bake the configuration into an image, base it on the published image. Your image inherits the non-root user, entrypoint, and config path:

FROM ghcr.io/ptweezy/cronstable:latest

# The base image already runs as the non-root user 65534.
COPY cronstable.yaml /etc/cronstable.d/cronstable.yaml

Hardened security context

The published image satisfies a fully restricted pod security context:

  • runAsNonRoot / non-root UID: the daemon needs no privileges, so it runs as an unprivileged UID (65534). Only per-job user/group switching and per-run resource limits need root, and neither is available here.
  • RuntimeDefault seccomp: cronstable makes no exotic syscalls, so the default seccomp profile (or an equivalently strict custom one) works.
  • readOnlyRootFilesystem: neither the image nor a pip/pipx install requires runtime writes. Mount the crontab read-only. See writable-path exceptions for the two features that need a small writable volume: a Unix-socket web listener (with the image) and the standalone binary's self-extraction temp directory (only if you deploy the binary).
  • fsGroup-mounted config/secrets: mount config and secret volumes with an fsGroup (for example, 65534) so the non-root process can read them.
  • drop: [ALL] capabilities and allowPrivilegeEscalation: false: cronstable needs no Linux capabilities and never escalates privileges.

Kubernetes Deployment

A Deployment mounting the crontab from a ConfigMap, read-only, under a fully restricted security context:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cronstable
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cronstable
  template:
    metadata:
      labels:
        app: cronstable
    spec:
      securityContext:           # pod-level
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        fsGroup: 65534           # lets the non-root process read mounted volumes
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: cronstable
          image: ghcr.io/ptweezy/cronstable:latest
          args: ["-c", "/etc/cronstable.d"]
          securityContext:       # container-level
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          resources:
            limits:
              cpu: 200m
              memory: 128Mi
            requests:
              cpu: 10m
              memory: 64Mi
          volumeMounts:
            - name: crontab
              mountPath: /etc/cronstable.d
              readOnly: true
      volumes:
        - name: crontab
          configMap:
            name: cronstable

replicas: 1 is the safe default: cronstable holds the schedule in-process, so two replicas with no coordination each run every job independently.

To coordinate scheduled jobs across replicas, enable leader election with a cluster section. Execution guarantees depend on the backend and each job's clusterPolicy. Choose a backend based on the coordination services you already run:

  • backend: kubernetes (recommended on Kubernetes). A coordination.k8s.io/v1 Lease provides fenced leadership with a single lease holder while the apiserver is reachable. No peer mTLS, no peer list, no odd-replica rule: a plain Deployment with any replica count works. You only grant the Lease RBAC (get/create/update). Likewise backend: etcd if you run etcd. See operating the lease backends and example/kubernetes/.
  • backend: gossip (the default, no coordination store). A quorum-gated, mutual-TLS election that keeps no shared state, so:
    • Use an odd replica count. 3 replicas tolerate one failure, 5 tolerate two. Even counts buy no extra fault tolerance, and replicas: 2 is worse than 1: a majority of 2 is 2, so both must be up to run anything. The daemon refuses to start with electLeader and a 2-node cluster, and warns on even sizes. Spread replicas across nodes/zones with topologySpreadConstraints. Correlated failures defeat quorum regardless of count.
    • A minority partition skips Leader jobs once it detects lost quorum. Each node's view is only as fresh as its last poll, so a leadership change can briefly cause skipped or duplicate runs. Use a lease backend for fenced leadership. No backend guarantees that every scheduled job runs exactly once; see the per-job policies.
    • Provision the per-pod certificates from your own PKI (for example, cert-manager) and give each pod a stable nodeName. A StatefulSet's ordinal hostnames make both the cert SANs and the peer list straightforward.

See clustering and leader election for the full trust model, quorum math, per-job clusterPolicy, the lease backends, and a runnable local example.

mTLS listener and networking (gossip backend)

The gossip backend adds a second listener, the cluster.listen mTLS port (0.0.0.0:8443 in the examples), that peers poll over mutual TLS. It is separate from the optional web API port, so plan the pod network for it:

  • Open the port for pod-to-pod traffic. Every node both serves /peer on cluster.listen and connects out to each peer's cluster.listen, so the port must be reachable between the replicas. Expose it on the container (containerPort: 8443).
  • Add a NetworkPolicy in a default-deny cluster. If you run default-deny ingress, allow the cluster.listen port from the pods carrying the same app label (the peers), so the mTLS gossip is not silently dropped. A blocked port shows up as unreachable peers and a lost quorum, not a clear error.
  • Use a headless Service for peer DNS. Pair a gossip StatefulSet with a headless Service (clusterIP: None) so each pod is addressable at a stable per-ordinal DNS name (cronstable-0.<svc>.<ns>.svc, cronstable-1.<svc>…). You list those names under cluster.peers, and the certificate SANs must match them. A lease backend needs none of this: it has no peer listener.

Health checks

On the web API, cronstable serves a GET /version that returns 200 after the daemon is up, which makes a cheap liveness probe. Enable an http:// web listener (for example, a web.listen entry of http://0.0.0.0:8080) and point the probe at it:

          ports:
            - name: web
              containerPort: 8080
          livenessProbe:
            httpGet:
              path: /version
              port: web
            initialDelaySeconds: 5
            periodSeconds: 10

If you protect the web API with a web.authToken, only the dashboard page (/) stays unauthenticated, so /version returns 401 without the bearer token. Either add the Authorization header to the probe (httpGet.httpHeaders) or point it at / (when the dashboard is enabled; with web.ui: false there is no / route).

For clustered deployments, do not turn quorum loss into a pod restart: a livenessProbe that fails on lost quorum would restart exactly the nodes you need to reform it. Keep the probe scoped to daemon liveness (/version) and alert on cluster health separately. quorate on GET /cluster is the field to alert on: a node that cannot see a majority, or that on a lease backend cannot reach the store, stands its Leader jobs down. For the full signal set, see the monitoring and alerting table.

Cluster certificate operations

Two different tls: blocks. This section is about cluster.tls, the peer certificates the gossip mesh presents to other cronstable nodes on the cluster.listen port. They are unrelated to web.tls, the listener certificates the HTTP control API serves to operators, browsers, and clients on the web.listen addresses, and to state.jobApi.tls, which serves the job-facing state endpoint. Keep their PKI separate: the cluster CA is a membership allowlist (explained later), so a certificate issued for a dashboard user must not chain to it. Provision each from its own issuer. For the listener side, see listener TLS.

This applies to the gossip backend only (the lease backends use no per-node mTLS certs). Day-to-day operation needs little intervention: cronstable reloads its mTLS contexts in place. On each config-reload pass it compares the on-disk ca/cert/key against what it loaded, and if any changed it rebuilds the TLS contexts within ~1 minute, with no restart. It dry-runs the new material first, so a half-written file is retried, not fatal. It therefore picks up in-place renewals from cert-manager, Vault, and mounted-secret refreshes (same paths, new bytes) automatically. The Clustering page covers the full mechanism under certificate rotation; the operational cases:

  • The cluster ca is the membership allowlist. The daemon trusts any cert the configured ca signs to assert a node identity and gossip state, so the ca must be a dedicated, single-purpose CA issued only to cronstable nodes, not a shared service-mesh or organization-wide CA. Any cert the CA admits could otherwise fabricate the /peer payload (fake agreement, trip the conflict gate, or suppress an @reboot job). Provision leaf certs from that dedicated PKI (a private cert-manager issuer, an internal CA). The daemon only consumes them.

  • Rotating a single node's cert. If a leaf your PKI renews still chains to the same ca, rotating it needs no coordination: write the new cert/key in place, the node reloads within ~1 minute, and its peers keep trusting it. Issue the replacement with a comfortable overlap (well before the old cert expires) so a slow refresh never leaves the node serving an expired cert.

  • Rolling the cluster CA. Changing the CA itself needs care, because a node only trusts peers whose certs chain to the CA bundle it currently holds. Roll it so trust always overlaps:

    1. Distribute a bundle file containing both the old and new CA as the tls.ca on every node first (an additional trust anchor). Each node reloads within ~1 minute and now trusts either CA.
    2. Confirm every node still shows its peers agreed on GET /cluster (no untrusted).
    3. Re-issue each node's leaf cert from the new CA, one node at a time, watching GET /cluster after each until the rotated node and its peers return to agreed.
    4. After every node presents a new-CA cert, narrow the bundle back to the new CA alone and distribute it everywhere.

    Never cut the CA over in a single step: if some nodes trust only the new CA while others still present old-CA certs, they reject each other as untrusted and the cluster loses quorum until trust overlaps again.

  • Recovering from an untrusted cascade. If peers start showing untrusted on GET /cluster (or the peer … is untrusted WARNING in the logs) after a rotation, certs and CA trust have diverged. The usual causes are a CA roll that skipped the overlap step, or a node whose refresh lagged. Recovery needs no restart: restore the trust overlap (push a CA bundle covering whichever CA the still-untrusted peers were issued from), or finish rolling the lagging nodes onto the new CA. Each node reloads within ~1 minute and peers return to agreed automatically. Leader jobs stand down (fail closed) while quorum is lost, so the cascade skips firings rather than double-running them: no split-brain during recovery, only missed firings until trust reconverges.

Production clustering checklist

Before scaling past replicas: 1, work through this checklist for your backend. Each row links to the authoritative section on the Clustering page.

Concern gossip (default) Lease (kubernetes / etcd)
Trust / auth (gossip mTLS / lease store) per-node certs from a dedicated cluster CA; open the cluster.listen mTLS port apiserver Lease RBAC (get/create/update) or reachable etcd endpoints; no mTLS
Node identity (gossip / lease) stable, unique nodeName (distinct cert SANs / stable hostnames) stable name too, but the lease is the fence, so a duplicate name cannot double-lead
Listener / port (attestation) cluster.listen reachable pod-to-pod; headless Service for peer DNS none (no peer listener)
Replica count (sizing) odd (3/5/7); cronstable rejects an electLeader 2-node cluster and warns on even sizes any count; the store fences, not a quorum
Rotation (cert rotation) in-place cert/CA reload (earlier); roll the CA with an overlap n/a (no per-node mTLS certs)
Preflight cronstable -c <path> --validate-config (catches lease ordering, CA/cert paths, an electLeader 2-node cluster) cronstable -c <path> --validate-config (same)

cronstable -c <path> --validate-config parses the config and exits nonzero on any error, so it belongs in CI before a cluster deploy. It enforces the lease timing ordering (leaseDurationSeconds > renewDeadlineSeconds, retryPeriodSeconds < renewDeadlineSeconds, and other lease timing rules), the RFC1123 leaseName/leaseNamespace shapes, the etcd endpoint/TLS rules, and the electLeader 2-node rejection. See command-line reference.

Resource limits in a container

In the published image, resourceLimits caps do not apply. The image runs as uid 65534, and container runtimes mount /sys/fs/cgroup read-only by default, so cronstable cannot give a run a cgroup of its own. A capped job runs uncapped, the daemon logs one warning naming the capped jobs, and each run's record says its caps were not enforced. A job with resourceLimits.required: true fails each run instead. The container's own limits, such as a pod's resources.limits or docker run --memory and --cpus, still bound cronstable and all of its jobs together.

For per-run caps, the container needs:

  • cronstable running as root inside it, from an image of your own;
  • a writable cgroupfs, in the container's own cgroup namespace or in a container cgroup that the runtime delegated. On a cgroup v2 host, Docker and Podman give each container its own cgroup namespace by default, and --privileged or, on Docker Engine 28 and later, --security-opt writable-cgroups=true makes the cgroupfs writable.

Kubernetes has no pod setting for this. In a privileged pod, which shares the host's cgroup namespace, whether caps apply depends on whether the runtime delegates the container's cgroup; check limits on GET /node. An unprivileged pod gets a writable cgroupfs only from a node-level runtime setting, such as containerd's cgroup_writable option.

When the caps can apply, cronstable moves every process in the container's cgroup, its own included, into a daemon/ child cgroup, and gives each capped run a cgroup under runs/. A docker exec shell joins the daemon/ cgroup, so a second cronstable started that way shares the first daemon's daemon/ and runs/, and its cpu controller too: either daemon's config apply can turn cpu control on or off for both, and neither turns it off while a run of either one holds a cpu cap. For independent cpu control, run a second capped daemon in a cgroup of its own (see the cgroup layout). A job that runs as root, as jobs do in such a container unless they set user, can write its own cgroup's files, so the caps there do not sandbox it (see security).

Writable-path exceptions

A read-only root filesystem is sufficient for the published image in the normal case. Two features need a small writable mount, described in the following sections. The durable state store (state.path) and the filesystem election backend also need a writable volume, as does an out-of-cluster kubeconfig that embeds client-certificate-data/client-key-data (set TMPDIR to a writable mount).

1. Unix-socket web interface

If you enable the optional HTTP control API on a Unix socket (a web.listen entry such as unix:///path/cronstable.sock), cronstable binds a UnixSite and creates the socket file at that path, a write. Point the socket at a small writable volume (a Kubernetes emptyDir) rather than the root filesystem:

          volumeMounts:
            - name: crontab
              mountPath: /etc/cronstable.d
              readOnly: true
            - name: run
              mountPath: /run/cronstable
      volumes:
        - name: crontab
          configMap:
            name: cronstable
        - name: run
          emptyDir: {}

with unix:///run/cronstable/cronstable.sock as a web.listen entry in your configuration. (TCP listeners such as http://0.0.0.0:8080 need no writable path.) The optional web.socketMode config key, if set, chmods the socket after bind.

Windows does not support unix:// web listeners (the Proactor event loop lacks create_unix_server). The daemon skips such a listen URL with the warning Ignoring web listen url <url>: unix-socket listeners are not supported on this platform. Use an http:// listener instead. web.socketMode only applies to unix sockets, so it is irrelevant on Windows. See running on Windows.

2. The standalone binary under a read-only rootfs

This applies only if you deploy the standalone binary (from the GitHub releases) instead of the published image. The binary self-extracts on each start, so it needs a temp directory that is both writable and executable, and a read-only root filesystem makes /tmp fail that test. For the error messages and the Docker --tmpfs exec recipe, Kubernetes emptyDir mount, and TMPDIR override, see the standalone binary temp-directory requirement on the Installation page.

Operational notes

  • Logging: cronstable logs to stdout/stderr. Collect logs at the platform level (kubectl logs, the container runtime's log driver). Adjust verbosity with -l/--log-level (default INFO) or a logging: config section.
  • Shutdown: on POSIX, cronstable installs handlers for SIGINT and SIGTERM and shuts down gracefully, so the default pod termination path needs no extra configuration. On Windows (where the Proactor loop has no add_signal_handler) it instead handles SIGINT/SIGBREAK with signal.signal plus a heartbeat timer, and console close or OS shutdown with a native console-control handler. Logoff is excluded deliberately: a service receives it for every user's sign-out. Either way, the daemon finishes the currently-running jobs first. Supervised deployments can also stop it over HTTP with the authenticated POST /shutdown route. See running on Windows and the HTTP control API.
  • Validate before deploy: cronstable -c <path> --validate-config parses the config and exits, useful as a CI/pre-deploy gate. See command-line reference.
  • Config not found: the default config path is platform-specific: /etc/cronstable.d on POSIX. On Windows it is %ProgramData%\cronstable when that directory holds configuration, else %APPDATA%\cronstable, falling back to the user profile ~ if APPDATA is unset. With no arguments and no default configuration, cronstable prints setup guidance and exits 0. Pass -c explicitly in a supervised command so a missing configuration exits 1. In the container, mount the config volume at /etc/cronstable.d.
⚠️ **GitHub.com Fallback** ⚠️