Push Notifications - ptweezy/cronstable GitHub Wiki

Push notifications

The push reporter sends end-to-end encrypted job and daemon alerts to paired devices through a hosted relay. Only the paired devices can decrypt alert contents.

The encryption model: the daemon seals each alert to every paired device's public key under the device's sealing suite (a libsodium sealed box for x25519, X-Wing HPKE for xwing), so only that device's private key can open the payload. That key is generated on the phone and never leaves it. The relay forwards alerts to the platform push service (APNs), and sees only a device token, a ciphertext, an opaque coalescing hash (keyed with a per-installation salt the relay never sees), a priority, and an event flag, never job names, hostnames, or log lines.

Enabling push

Push is an optional extra. Install it alongside cronstable:

pip install "cronstable[push]"

The extra carries two libraries. PyNaCl, which bundles libsodium, seals the x25519 suite on every platform. cryptography seals the post-quantum xwing suite (see pairing devices) and comes along on every platform it publishes a wheel for. The release binaries and the container images bundle both. A build that cannot carry PyNaCl ships without push, and the daemon then refuses to start with a config that asks for it and reports an error that says so. A build that cannot carry cryptography seals x25519 only.

cryptography publishes wheels for fewer platforms than PyNaCl, so the pip route reaches post-quantum sealing on fewer platforms than the binaries and images do:

How you run cronstable Post-quantum sealing
pip install "cronstable[push]" Linux x86_64 and aarch64, macOS Apple Silicon, and Windows x64
Release binary Every Linux build except linux-mips64le and linux-armel; every macOS, Windows, FreeBSD, OpenBSD, NetBSD, and illumos build, subject to successful source builds where wheels are unavailable
Docker image Every platform of every tag

The binaries and images reach past pip because CI builds cryptography from source wherever PyPI has no wheel, against an OpenSSL 3.5 and a Rust toolchain the lane installs or builds. Each source build seals a real X-Wing probe before it is frozen in, and a build that fails the probe ships without the suite and says so in the build log. The two Linux builds left out run on Debian bookworm, whose Rust and OpenSSL are both too old, and have no newer Rust toolchain for their architectures.

Everywhere else, the extra installs PyNaCl alone and the daemon seals x25519 only, which works on every platform. The daemon logs the suites it can seal at startup. If you expected post-quantum sealing and did not get it, that line says so. To ask a binary directly, run cronstable --sealable-suites, which prints one suite per line. On a platform without a wheel, you can attempt post-quantum sealing by building cryptography 50.0.2 or newer from source (pip install "cryptography>=50.0.2,<51", which needs a Rust toolchain and the OpenSSL 3.5 headers on that host).

Cryptography 49.0.0 dropped support for Intel macOS and 32-bit Windows before the latest security fixes. The push extra installs PyNaCl alone there, while release binaries attempt to compile patched cryptography with Rust and a static OpenSSL. On Windows the build targets 32-bit Rust and OpenSSL explicitly, even when the build host is 64-bit. These source builds are best effort: only a build that passes the X-Wing sealing probe is bundled, and there is no fallback to the vulnerable 48.x release line.

If a binary does not list xwing in --sealable-suites, existing xwing pairings need to be paired again under x25519; open the companion app and connect to the daemon so it can update the pairing. Until then, the daemon reports those devices as unsealable and cannot deliver their alerts. A 32-bit Python on 64-bit Windows reports the 64-bit machine, so its dependency marker still requests cryptography; use 64-bit Python, or install cronstable plus pynacl by hand if a source build is not possible.

The pip route stops at two Linux architectures because a dependency marker cannot see which libc you are on, and cryptography publishes ppc64le and armv7l wheels for glibc but not for musl. Each release binary is built against a known libc, so the armv7 build reaches one architecture past pip on the same hardware. The Docker images resolve the dependency from inside the target image, where the libc is known, so the glibc images reach both architectures. Every image build proves the result the same way the binaries do, by running cronstable --sealable-suites and requiring xwing wherever cryptography was installed.

Install cryptography on every node that shares a device registry. The pairing is stored once and read by all of them, while the library is per node, so a node without cryptography cannot page a device that paired under xwing.

Then configure the daemon-global push: section, which says where alerts go and where device pairings are stored:

push:
  relay:
    url: https://relay.cronstable.com/
    timeout: 10
  devicesFile: /var/lib/cronstable/devices.json

https://relay.cronstable.com/ is the hosted relay. Its full source is published at ptweezy/cronstable-relay (MIT), including its delivery policy (per-device coalescing, flap suppression, rate limits) and self-hosting instructions. Because alerts are sealed before they leave the daemon, pointing at the hosted relay trusts it with routing metadata only, never content.

devicesFile is needed only on a stateless install. With a state: section the registry rides the durable store instead (see where pairings are stored).

The section alone sends nothing. Each job hook (or the daemon-level notify: block) opts in through the report schema, exactly like the other reporters. A typical setup pushes on every failure through the file's defaults::

defaults:
  onFailure:
    report:
      push:
        enabled: true
        priority: time-sensitive
        includeLogTail: true

The same block under notify.report pushes daemon and orchestration events too: DAG failures, approval gates awaiting a decision, and leadership and quorum changes:

notify:
  report:
    push:
      enabled: true

report.push options

Available on onFailure, onPermanentFailure, onSuccess, and onLate, and under notify.report:

Option Type Default Description
enabled bool (Opt) false Opt this hook into the push channel. Enabling it anywhere requires the daemon-global push: section (a ConfigError otherwise).
priority time-sensitive or passive (Opt) time-sensitive Relayed to APNs as the interruption level. time-sensitive breaks through scheduled summaries; passive does not.
includeLogTail bool (Opt) true Carry the last captured output lines (stderr when it has output, otherwise stdout, up to 40 lines) inside the sealed payload, trimmed oldest-first to fit the size cap.

The push: section

Key Type Default Description
relay.url str (required) none The relay endpoint the daemon POSTs alerts to. Must be an http(s) URL. The daemon never posts ciphertext anywhere the config did not spell out.
relay.timeout float (Opt) 10 Total timeout in seconds for each relay request. Must be greater than 0. Device POSTs go out together, so this is also the worst case a whole fan-out adds to a report, whatever the fleet size.
devicesFile str (Opt) unset Path of a local JSON file holding the paired-device registry. Required when no state: section is configured. When set, the daemon uses it even with a state: section.
allowUnauthenticated bool (Opt) false Serve the /push/devices pairing endpoints on a routable (non-loopback, non-socket) listener even with no web.authToken. The default fails closed: with no token the web app has no auth middleware at all, so a push: section alongside a routable listener raises a ConfigError at load. An https:// listener with web.tls.clientCa set is exempt without this flag (mutual TLS authenticates the caller). Plain https:// is not exempt. Set true only when the endpoints are protected by other means (an mTLS-terminating proxy, a network policy).

Delivery quota on the hosted relay

The daemon's push reporter requires no subscription or license key. Delivery quotas apply to the relay service you choose.

The hosted relay forwards up to 500 alerts per device per UTC calendar month without a subscription, and without bound for devices with a Cronstable Pro subscription for iOS. Only alerts that reach APNs count; coalesced, suppressed, and rate-limited envelopes do not. Past the bound the relay answers the daemon with {"outcome": "digested"} (a 2xx, so the daemon logs no failure) and sends the device at most one fixed "alerts are waiting" digest per hour; the app keeps polling its servers directly. The device proves its plan to the relay itself, with the App Store's signed transaction; the daemon takes no part beyond naming the relay in pairLinkBase. A self-hosted relay applies whatever policy it runs, and the reference relay's bound is the RELAY_FREE_MONTHLY_FORWARDS knob. The wire details are in the relay protocol.

Pairing devices

Pairing registers a device's public key and platform push token with the daemon. The dashboard has a Pair a device panel, in the command palette and in settings. Its QR encodes a pairing link: the {v: 1, name, url, token} payload, base64url-encoded into the fragment of the relay origin's /pair route, so a phone-camera scan deep-links into the companion app, or lands on install pointers when the app is missing.

The panel shows the same payload as a copyable JSON string, and the app's in-app scanner accepts both forms. Either way the app completes the pairing by calling POST /push/devices with the scanned token. The relay protocol specifies the link format in its "Pairing links" section. The panel warns when the token it would embed is the all-scopes one. Give a phone a scoped web.authTokens entry instead.

To pair without the dashboard page, see pairing from the terminal: cronstable pair and the terminal dashboard draw the same code.

The Pair a device panel: a QR code deep-linking the connection payload into the companion app, the payload as a copyable JSON string, and the all-scopes token warning

Pairing is also one API call (control scope):

$ curl -X POST http://127.0.0.1:8080/push/devices \
    -H "Authorization: Bearer s3cr3t" \
    -H "Content-Type: application/json" \
    -d '{"name": "parker-iphone", "platform": "ios",
         "publicKey": "jSNlDu28No2itHnvrs6ajHHuNAxvqgOjmGxHJrMo8yg=",
         "pushToken": "8f3a1bc2…d94af1c9"}'
{
    "device": {
        "id": "f1e2d3c4b5a69788",
        "name": "parker-iphone",
        "platform": "ios",
        "publicKey": "jSNlDu28No2itHnvrs6ajHHuNAxvqgOjmGxHJrMo8yg=",
        "fingerprint": "8c2f-91ab-04de",
        "pushToken": "…4af1c9",
        "createdAt": "2026-07-23T14:00:00+00:00",
        "createdBy": "parker-iphone"
    },
    "created": true
}

publicKey must be base64 decoding to the length its suite requires (32 bytes for the default x25519, 1216 bytes for xwing) and be a usable key for that suite. At pairing, rather than on the first alert, the daemon rejects a key its library refuses to seal to: an all-zero or low-order X25519 key, or an unusable X-Wing key. suite is optional and defaults to x25519. The daemon refuses a pairing that names a suite it cannot seal to, rather than storing a record whose every alert would fail. name, platform, and pushToken are bounded strings. Validation failures are a 400 naming the field.

xwing is the post-quantum hybrid suite: X-Wing combines ML-KEM-768 with X25519, so recovering the plaintext requires breaking both. Sealing under it requires the cryptography library, which the push extra carries on every platform with a wheel; a daemon without it refuses xwing pairings and seals x25519 with PyNaCl alone. The app picks the suite for you: GET /whoami lists the daemon's sealable suites as sealableSuites, and the app pairs under xwing whenever that list advertises it. The daemon proves xwing with one real probe seal through its own library before advertising it, so an install whose cryptography is present but broken advertises x25519 only.

A device already paired under x25519 keeps that suite across re-pairs, because the public key is the pairing identity and a suite change means a new record. When a daemon starts advertising xwing after the pairing was made (an upgrade to a release that carries cryptography, or a node that installed it), the app moves the pairing up on its own: once sealableSuites has listed xwing on two checks in a row, it registers the device's X-Wing key as a new record, then revokes the superseded one. The pairing screen shows the same move as a button for anyone who would rather not wait for the next check. The wire construction is normative in the relay protocol.

The app also moves a pairing the other way. An xwing pairing outlives the daemon's ability to seal it whenever the daemon loses cryptography: a reinstall without it, a move onto an image or a binary with no wheel for that platform, or a cluster node that never had it. Every alert to that device is then dropped at sealing, and the re-pair the app sends on each push-token rotation answers 400 for as long as that lasts. When sealableSuites omits the pairing's suite on two checks in a row, the app re-pairs under x25519, which every daemon seals, and revokes the stranded record. The pairing screen shows the same state with a manual switch. After a move down, the automatic move back up stands down until the app next launches, and the pairing screen's upgrade button remains, so a cluster where only some nodes carry cryptography does not trade the pairing back and forth.

The heal trades sealing strength for delivery on purpose. The daemon's advertisement is authenticated only by the transport, so a party who can rewrite two GET /whoami answers can move a pairing from xwing down to x25519. On a transport that allows that, the same party can already read the bearer token and substitute the key at pairing (see pair over a trusted transport). On a cluster where only some nodes carry cryptography, the daemon's answers alternate as requests land on different nodes, which is one more reason to install cryptography on every node that shares a registry.

Re-pairing the same public key (push tokens rotate; phones get renamed) answers 200 with created: false and updates name/platform/pushToken in place, keeping the record's id and createdAt so revocation references stay stable. createdBy records the label of the bearer token that performed the pairing.

List pairings (view scope; push tokens are redacted to their trailing six characters, and public keys are returned whole):

$ curl -H "Authorization: Bearer s3cr3t" http://127.0.0.1:8080/push/devices

Send a test alert through the relay to one device (control scope; 200 with the relay outcome, 502 when sealing or the relay failed, so a silent phone is debuggable from the dashboard):

$ curl -X POST -H "Authorization: Bearer s3cr3t" \
    http://127.0.0.1:8080/push/devices/f1e2d3c4b5a69788/test
{"device": "f1e2d3c4b5a69788", "status": 200, "error": null}

Revoke a device (control scope):

$ curl -X DELETE -H "Authorization: Bearer s3cr3t" \
    http://127.0.0.1:8080/push/devices/f1e2d3c4b5a69788
{"revoked": "f1e2d3c4b5a69788"}

Revoking the pairing stops future alerts to that device. To also revoke its API access, drop its web.authTokens entry and reload. Endpoint details are in the HTTP control API.

Pair over a trusted transport, then compare fingerprints

Pairing endpoints sit behind the web API's bearer token whenever one is configured, and the daemon refuses to start a push: section on a routable listener that has none (see failure behavior). An ungated pairing endpoint can therefore exist only on loopback or a unix socket.

Three things still travel during pairing that are worth protecting:

  • The QR payload embeds a bearer token.
  • An app-absent camera scan loads the relay host's landing page with that payload in the URL fragment, readable by the page's script. The in-app scanner and the copyable JSON involve no third party.
  • The pairing POST carries the device public key that every future alert is sealed to.

On a transport an attacker can read or rewrite (plaintext HTTP across a shared network), the token can be stolen and, worse, the key can be substituted: alerts would then seal to the attacker's key. So pair over HTTPS (web.tls or a TLS-terminating proxy), over loopback (an SSH tunnel to the daemon host), or on a network you trust.

With web.bonjour on, the daemon also advertises its URL to the local network. The advert carries no secrets, but it does make the daemon easier to find, which is one more reason pairing belongs on a trusted network.

Key substitution is detectable after the fact, and the check takes 10 seconds. The pairing response and the device listing carry a fingerprint (the first 12 hex characters of SHA-256 over the raw key bytes, grouped for reading aloud), and the companion app displays the same fingerprint for the key it generated. Compare them. If they differ, the daemon stored a key that is not your phone's: revoke the device and pair again over a trusted transport.

Where pairings are stored

The registry has two homes, and exactly one is in effect:

  • The durable state store (when a state: section is configured and devicesFile is not set): one document per device, so pairing and revocation are per-key atomic operations, and every node sharing the store sees the same registry. Whichever node fires a report pushes to the same device set. State garbage collection never sweeps the documents: a pairing lives until it is explicitly revoked.
  • push.devicesFile (required on stateless installs): a single local JSON file, written atomically (a uniquely named temp file plus a rename) and created with owner-only permissions where the platform honors them.
    • If the file fails to parse, the daemon refuses writes rather than overwriting possibly recoverable pairings, and a later write sweeps a temp file that an unfinished write left beside it.
    • Reads and writes run on a private worker thread, are bounded at 10 seconds, and are serialized process-wide per file path. A registry on an unresponsive mount therefore degrades pairing (the endpoints answer 503) without stalling the scheduler or process shutdown.

Both homes also hold the installation's collapse salt: the collapseSalt key in devicesFile, or a pushmeta document in the state store. It is the secret that keys the coalescing hash sent to the relay, created on first use. Every node sharing a registry derives the same hash for the same alert, and the relay never sees the salt itself.

The reporting path reads an in-memory mirror of the registry, refreshed at most every 60 seconds, so a slow or briefly unavailable store can never stall a report fan-out. A pairing made on another node becomes effective within that window. The daemon remembers a failed read for about 5 seconds, so a burst of alerts against an unreachable store costs one attempt between them rather than one each. The pairing endpoints always read and write the store directly.

Size limits and what an alert contains

The sealed plaintext is a compact JSON document: v, kind (success, failure, sla, event, or test), name, success, host, and ts. It also carries the run context that is known (run_id, schedule, started_at, exit_code, fail_reason), the log tail on job alerts (when includeLogTail is on and output was captured), the event / subject / message fields on daemon events, and the breach fields on SLA alerts. The full field-by-field schema is in the relay protocol document.

APNs rejects notifications over 4096 bytes, so the daemon caps the sealed, base64-encoded ciphertext at 3800 characters: 4096 minus the 189 bytes of the relay's own APNs envelope, minus a 107-byte reserve for future protocol fields. The daemon fits each device's payload to that device's own suite budget (2802 bytes of plaintext under x25519, 1714 under xwing), so a device paired under a wider-ciphertext suite is trimmed harder without costing the devices beside it any log lines. It trims an oversized payload in order: log-tail lines oldest-first (the newest lines carry the failure), then long free-text fields halved (never below 64 characters), then the optional context fields dropped. The alert's identity (name, kind, host) is never trimmed. A relay that enforces the 3000-character floor (the smallest cap the relay protocol allows) answers a larger ciphertext with a 400. The daemon then re-fits that alert to the floor, posts it again, and logs once that the relay is behind, so the page lands with fewer log lines instead of bouncing. There is no per-job template: the companion app renders the decrypted fields itself.

Failure behavior

Push is an alerting channel, so its configuration fails closed: a channel that silently self-disables is a missed page. All of the following are ConfigErrors at parse time (so --validate-config catches them), never runtime degradations:

  • a push: section on an install without PyNaCl (install the push extra or remove the section);
  • report.push.enabled: true anywhere (a job, a DAG task, notify:) without a push: section; the error names the offenders;
  • a push: section with neither a state: section nor devicesFile (the registry needs somewhere durable to live);
  • a push.relay.url that is not http(s), or a non-positive push.relay.timeout;
  • a push: section on a daemon whose web API listens on a routable address with no web.authToken/web.authTokens. Pairing is a mutating endpoint, and a token is what installs the auth middleware at all, so without one anything that can reach the listener can pair its own key and keep receiving alerts. Give the web API a token, keep web.listen on loopback or a unix socket, set web.tls.clientCa, or set push.allowUnauthenticated: true.

At runtime, delivery is deliberately non-fatal. The daemon logs a sealing failure or relay outage per device and never raises it into the reporting path, so a relay outage can never look like a reporting crash or affect the other reporters.

The daemon POSTs to the devices concurrently, so an unreachable relay delays a report by one relay.timeout rather than one per device, and therefore does not hold up retry arming or a graceful shutdown by the sum. An alert fired with no device paired is dropped with a warning naming the pairing endpoint.

When the registry store is unavailable, whether unreachable or holding a document this build cannot read, report fan-outs fall back to the last-known device set and say so, and the pairing endpoints answer 503 per request.

Relay protocol

The daemon-to-relay wire contract is documented in docs/relay-protocol.md. The relay implementation lives in the separate ptweezy/cronstable-relay repository; that file is the contract any implementation must satisfy.

The trust model: the relay is not a trusted party. It receives ciphertext and routing metadata only (device token, coalescing hash, priority, event flag), and owns deduplication, rate limiting, and flap suppression, coalescing on the hash without learning what it hashes. The per-installation salt keys that hash, so the hash stays opaque even for guessable job names. It forwards to APNs with mutable-content set so the app decrypts and renders the notification on the device.

Related pages