Listener TLS - ptweezy/cronstable GitHub Wiki

Listener TLS

cronstable's HTTP listeners support TLS without a reverse proxy. Configure web.tls to serve the dashboard, REST API, metrics, and MCP endpoint through the https:// entries in web.listen. Configure state.jobApi.tls to encrypt connections to the job state API. The daemon then gives each job the CA certificate it needs to verify that API's certificate.

Both configurations require a certificate and its private key. web.tls also accepts a client CA for mutual TLS, which requires clients to present certificates signed by that CA. Both use the shared helpers in cronstable/tlsutil.py. The cluster has a separate configuration: cluster.tls is its own mutual TLS configuration for peer connections.

On this page: When you want it · Quickstart · The web.tls block · Mutual TLS · Mixed listeners · The job state API over TLS · Client configuration · Certificate rotation · Troubleshooting

When you want it, and when you do not

Use web.tls when the dashboard or the API is reachable from anywhere that is not this host: another machine on the network, a colleague's laptop, a CI runner, a desktop MCP client. The API carries the bearer token (web.authToken) in an Authorization header. On a routable interface, a plaintext listener therefore puts that token, and every job name, command, and captured output the API returns, on the wire in the clear.

You might not need listener TLS in these deployments:

A loopback-only dashboard on a single-user host. Traffic to http://127.0.0.1:8080 stays on the host. On a host with other local accounts, one of them can bind the port while the daemon is stopped, and a plaintext client then sends it the token. See authentication.
A unix:// listener. The socket lives in the host filesystem and web.socketMode is its access control. unix:// entries stay plaintext even when web.tls is set.
TLS already terminated in front. If an ingress, a service mesh sidecar, or an nginx block already terminates TLS and forwards to a loopback http:// listener, you can keep that setup. Use web.tls to terminate TLS in cronstable instead.

HTTPS with a server certificate authenticates the server to the client. To authenticate callers, use web.authToken or mutual TLS.

Quickstart: an HTTPS dashboard

1. Create a certificate with the required SANs

The certificate's subject alternative names (SANs) must include the names or IP addresses that clients use. A DNS SAN for localhost doesn't match https://127.0.0.1:8443; that URL requires an IP SAN. Include both forms and any network hostname that clients use:

openssl req -x509 -newkey rsa:2048 -sha256 -days 365 -nodes \
  -keyout /etc/cronstable/tls/web.key \
  -out    /etc/cronstable/tls/web.pem \
  -subj   "/CN=cronstable" \
  -addext "subjectAltName=DNS:localhost,DNS:cronstable.internal,IP:127.0.0.1,IP:::1" \
  -addext "extendedKeyUsage=serverAuth"
chmod 600 /etc/cronstable/tls/web.key

IP:::1 is the IPv6 loopback (the IP: prefix followed by ::1). If you only ever dial the host by address, drop the DNS:cronstable.internal entry. Add more DNS:/IP: entries for every other name that will appear in a client's URL. In production, issue the same shape of certificate from your own internal CA or cert-manager instead of self-signing; cronstable only consumes the files.

2. Point `web.tls` at it

web:
  listen:
    - https://0.0.0.0:8443
  tls:
    cert: /etc/cronstable/tls/web.pem
    key:  /etc/cronstable/tls/web.key
  authToken:
    fromEnvVar: CRONSTABLE_WEB_TOKEN

Before deploying, run cronstable --validate-config. It reports a missing key for a configured cert, or an https:// listener without a certificate, before the daemon starts.

3. Point a client at it

For a self-signed certificate, pass the certificate file to clients as the trust anchor. For a certificate issued by an internal CA, pass the issuing CA's PEM file:

cronstable tui --url https://cronstable.internal:8443 \
               --cacert /etc/cronstable/tls/web.pem

# same four flags on the MCP bridge
cronstable mcp --url https://cronstable.internal:8443 \
               --cacert /etc/cronstable/tls/web.pem \
               --token-env CRONSTABLE_WEB_TOKEN --check

curl needs the same anchor: curl --cacert /etc/cronstable/tls/web.pem https://cronstable.internal:8443/status. A browser shows a trust warning until the CA is added to the operating system or browser trust store.

The `web.tls` block

web.listen accepts three schemes. https:// entries are served by the web.tls material:

Scheme	Transport	Notes
`http://host:port`	plaintext	Host and port both required.
`https://host:port`	TLS from `web.tls`	Host and port both required. An omitted port is not defaulted; it is skipped with a warning.
`unix:///path`	plaintext	Filesystem socket; `socketMode` is the access control. POSIX only.

The block itself has three keys:

Key	Default	Meaning
`cert`	(none)	PEM certificate (with any intermediates) served on every `https://` listen address. Required together with `key`.
`key`	(none)	PEM private key for `cert`. Required together with `cert`.
`clientCa`	(none)	Trust anchor for client certificates. When set, the listener requires one, signed by this CA, on every `https://` connection. See mutual TLS.

The daemon builds one context per app start and shares it across every https:// site, so all of them serve the same certificate.

What is validated, and when

These combinations are ConfigErrors at parse time, which means cronstable --validate-config catches them:

Configuration	Why it is refused
`cert` without `key`, or `key` without `cert`	A server certificate cannot be served without its private key.
`clientCa` without `cert`/`key`	A listener cannot demand client certificates without serving one of its own.
`web.tls` set, but no `https://` in `listen`	The material would be silently ignored and the API served in cleartext.
`https://` in `listen`, but no `cert`/`key`	Those listeners have no certificate to serve and would be skipped at startup.

Whether the files exist is deliberately not checked at parse time. Configuration parsing touches no filesystem, --validate-config is often run somewhere that is not the deployment target, and a Kubernetes-mounted secret need not exist yet at first boot. The files are read when the listener starts. If they do not load, the web API does not start; it is never downgraded to plaintext. The daemon logs the error and retries on the next config reload, which is at most a minute away.

Mutual TLS: `clientCa`

Setting clientCa promotes the listener from "encrypted" to "encrypted and caller-authenticated":

web:
  listen:
    - https://0.0.0.0:8443
  tls:
    cert:     /etc/cronstable/tls/web.pem
    key:      /etc/cronstable/tls/web.key
    clientCa: /etc/cronstable/tls/client-ca.pem

On every https:// connection, the client must now present a certificate that chains to client-ca.pem. A client that presents nothing is refused during the handshake, and so is a certificate from any other CA. There is no middle setting: "optional client CA" means the key is optional, never that a presented certificate is. A mode that completed the handshake for a client presenting nothing would turn clientCa into a no-op for the operator who set it precisely to authenticate callers.

The CA file is the caller allowlist. A server cannot do hostname verification against a client, so the listener accepts any certificate that CA has ever signed, with no further check on which one it is. Point clientCa at a dedicated CA that issues to cronstable clients only, never at a shared service-mesh or organization-wide CA. With an organizational CA, every workload that CA admits can call the control API and start, pause, or reconfigure jobs.

Clients answer the requirement with --client-cert and --client-key:

cronstable tui --url https://cronstable.internal:8443 \
               --cacert      /etc/cronstable/tls/web.pem \
               --client-cert /etc/cronstable/tls/ops.pem \
               --client-key  /etc/cronstable/tls/ops.key

mTLS satisfies the MCP token gate

mcp.enabled with no web.authToken on a routable listener is normally a ConfigError. With no token, the web app installs no auth middleware at all, so /mcp would be served unauthenticated. Plain https:// does not change that verdict, because encryption is not caller authentication.

web.tls.clientCa does change it. An mTLS listener authenticates its callers at the transport, which is the same guarantee the gate already accepts from an mTLS-terminating proxy. Therefore mcp.enabled on an https:// listener with clientCa set is allowed with no token. See MCP for the endpoint itself.

Mixed listeners: one daemon, two transports

listen is a list, and each entry becomes its own site on one runner, with its own transport. So a single daemon can serve the same app plaintext on loopback and over TLS on the routable interface:

web:
  listen:
    - http://127.0.0.1:8080          # local curl, local tui, node-local scrapes
    - https://10.0.0.5:8443          # everything off-host
    - unix:///run/cronstable/web.sock   # plaintext, guarded by socketMode
  tls:
    cert: /etc/cronstable/tls/web.pem
    key:  /etc/cronstable/tls/web.key
  socketMode: "0660"

The https:// entry is TLS, the http:// and unix:// entries are not, and all three serve the same routes, dashboard, and metrics. A plaintext request to the TLS port fails, as it should.

With mcp.enabled set, the MCP token gate reads the routable addresses. A mixed list like this needs web.authToken (or clientCa) because of the 10.0.0.5 entry, even though the loopback entry alone would not. Without mcp.enabled, no listener requires a token.

The job state API over TLS

state.jobApi is the endpoint jobs reach for durable state: KV, cursors, artifacts, mutexes, and run-scoped secrets. It defaults to an ephemeral loopback port, which needs no TLS at all. When jobs run off this host and dial it across the network, it needs TLS, because it serves per-run bearer tokens and staged job secrets.

state:
  path: /var/lib/cronstable
  jobApi:
    listen: https://10.0.0.5:9000
    allowNonLoopbackBind: true
    tls:
      cert: /etc/cronstable/tls/state.pem
      key:  /etc/cronstable/tls/state.key
      ca:   /etc/cronstable/tls/internal-ca.pem

Key	Default	Meaning
`cert`	(none)	PEM certificate served on an `https://` listen address. Required together with `key`.
`key`	(none)	PEM private key for `cert`. Required together with `cert`.
`ca`	(none)	The client-side trust anchor handed to every job as `CRONSTABLE_STATE_CACERT`, so the job CLI can verify a certificate no public root signed. Counts as TLS material. Setting it without an `https://` listen is refused, because it would be injected into every job and used for nothing.

listen accepts https://host:port, http://host:port, or a bare host:port (a unix:// path is refused; the job CLI speaks TCP only). The same pairing rules as web.tls apply: cert and key together, TLS material with no https:// listen refused, an https:// listen with no certificate refused.

Two rules are specific to this endpoint:

No wildcard host over https://. https://0.0.0.0:9000, https://[::]:9000, and every other form that the host's socket layer reads as the unspecified address are ConfigErrors. Jobs dial the address they are handed, and no certificate carries a SAN for "every interface", so every job would fail hostname verification. Name the interface explicitly (https://10.0.0.5:9000).
The advertised URL is the configured host, not the bound one. A job receives CRONSTABLE_STATE_URL=https://10.0.0.5:9000, the address it can actually dial and the one the certificate covers. (The daemon uses the bound address for the ephemeral loopback default, where it is the right answer.)

allowNonLoopbackBind is required for any non-loopback listen. If you pair it with a plaintext address, the daemon logs a warning at every load, naming what crosses the network in cleartext. It is a warning and not an error. Terminating TLS in front of the endpoint is still a valid answer.

`ca` here is not `clientCa` there

The third key means the opposite direction in the two blocks, so it is worth reading twice:

Key	Direction	What it does
`web.tls.clientCa`	server verifies clients	Requires a client certificate signed by this CA. It is the caller allowlist.
`state.jobApi.tls.ca`	jobs verify the server	Path handed to jobs as `CRONSTABLE_STATE_CACERT` so they can verify the daemon's certificate. It authenticates nobody.

The job state API deliberately does not support mutual TLS. The per-run bearer token already authenticates the caller (it identifies which run is calling, which a certificate would not), and requiring client certificates would mean injecting client key material into every job's environment.

`CRONSTABLE_STATE_CACERT`

When state.jobApi.tls.ca is set, the daemon adds one variable to every job's environment alongside the existing CRONSTABLE_STATE_* set:

Variable	Meaning
`CRONSTABLE_STATE_CACERT`	Path to the CA the job should verify the state endpoint against. Absent when `ca` is unset.

The job CLI picks it up with no configuration. cronstable state, cronstable lock, cronstable secret, and the rest read it the same way they read CRONSTABLE_STATE_URL and CRONSTABLE_STATE_TOKEN:

#!/bin/sh
cronstable state set last-cursor 12345     # verifies against CRONSTABLE_STATE_CACERT

The in-job CLI has no TLS flags at all, and there is deliberately no way to switch verification off from inside a job. Nothing running in a job's environment should be able to downgrade the channel carrying that job's own secrets.

Client configuration

The TUI, the MCP bridge, and cronstable pair take the same four options, flag first and environment variable second, so one exported set of variables serves every client in a shell:

Flag	Environment variable	Meaning
`--cacert PATH`	`CRONSTABLE_WEB_CACERT`	Verify the listener against this CA file instead of the system trust store. Needed for an internally-issued or self-signed certificate.
`--client-cert PATH`	`CRONSTABLE_WEB_CLIENT_CERT`	Certificate to present to a listener configured with `web.tls.clientCa`.
`--client-key PATH`	`CRONSTABLE_WEB_CLIENT_KEY`	Private key for `--client-cert`. Passing it without `--client-cert` is an error. A key alone cannot present an identity, so accepting it would leave you believing you had authenticated when the listener had refused you.
`--insecure`	`CRONSTABLE_WEB_INSECURE` (`1`/`true`/`yes`)	Skip verification entirely. The client warns on stderr, because it still sends the bearer token to whoever answers the connection.

Who takes what:

cronstable tui, cronstable mcp, and cronstable pair: all four, identically named.
A publicly trusted certificate needs none of them. With no option set, a client verifies against the system trust store and presents no certificate.
The in-job CLI takes none of them. It reads CRONSTABLE_STATE_CACERT from the environment the daemon injects; see CRONSTABLE_STATE_CACERT earlier.

--insecure is for a broken-certificate emergency, not a deployment mode. Verification off means an interceptor that answers the connection receives the bearer token, which is exactly what the warning says.

Certificate rotation

An in-place rotation (same paths, new bytes, which is how cert-manager, Vault, and a Kubernetes secret refresh renew) leaves the configuration bytes identical, so the ordinary "did the config change?" check never fires for it. Meanwhile the SSL context, built once at listener start, holds the old certificate in memory. Without a dedicated check, the daemon keeps serving the expired certificate.

The daemon notices it by fingerprint. On each housekeeping pass (at most once per wall-clock minute) it compares an (mtime_ns, size) signature of cert, key, and clientCa against what the running listener loaded. If any of them changed, it restarts the web app so the context is rebuilt from the new material. os.stat follows symlinks, so the daemon picks up the atomic symlink swap behind a Kubernetes mounted secret.

Two properties are worth planning around:

The restart is gated on the new material loading. A rotation is not atomic across the files and can be observed half-written. The new cert/key are dry-run loaded into a throwaway context first. If that fails, the running listener stays up, the daemon logs a warning, and the check runs again on the next reload. A half-written rotation therefore costs a log line, not an outage.
The restart briefly drops open connections. An https:// listener closes before its replacement binds the same port. Requests in progress are dropped, and so are the long-lived SSE log streams the dashboard and the TUI hold open. Those streams reconnect on their own, and the visible effect is a tailing log view that blips and resumes. Time noisy rotations accordingly. http:// listeners in the same listen list keep their sockets through the restart.

The state.jobApi listener has its own in-place rotation check for cert and key. It skips ca, because jobs read that file fresh on each run. On the next reload pass after a change, once the new files load, the daemon restarts that listener alone. If the rebind fails, the endpoint stays down until the state config next changes.

Troubleshooting

A client fails with certificate verify failed: Hostname mismatch (or IP address mismatch). The certificate does not cover the name in the URL. Hostname verification is on by default in every client, so https://127.0.0.1:8443 needs an IP:127.0.0.1 SAN and https://localhost:8443 needs a DNS:localhost SAN. One does not imply the other. Check what the certificate actually carries:

openssl x509 -in /etc/cronstable/tls/web.pem -noout -text | grep -A1 "Subject Alternative Name"

Reissue with the missing SAN, or dial the name the certificate does cover.

certificate verify failed: unable to get local issuer certificate. The client does not trust the issuing CA. Pass --cacert with the CA PEM (for a self-signed certificate, the certificate file itself), or install the CA in the system trust store. This is also what a browser's trust warning means.

The log says web: TLS material is not loadable, so the web API is not starting. The daemon could not load the certificate or key, so the whole web API stays down rather than any port being served in cleartext. The rest of the line carries the underlying error: a wrong path, a key that does not match the certificate, or a file the daemon's user cannot read. Fix it and the next config reload retries. No restart is needed.

The log says Ignoring web listen url ...: no usable web.tls material for an https listener. The daemon skipped a single https:// address because no context was available for it. Parse-time validation normally makes this unreachable (an https:// listen with no web.tls is a ConfigError), so in practice it means web.tls is present but empty. Give it a cert and key, or make the address http://.

The log says web: new TLS material is not yet loadable. The daemon detected a rotation, but the new files do not load yet, typically a renewal caught half-written. The old listener is still up and still serving, and the check repeats on the next reload. If the message persists past a couple of minutes, the new material is genuinely broken rather than half-written. Check the renewal job and that the certificate and key are a matching pair.

An mTLS listener refuses a client during the handshake. With clientCa set, the listener rejects a client that sends no certificate, or one signed by a different CA, before any HTTP request happens, so there is no status code to read, only a handshake error. Confirm the client is passing both --client-cert and --client-key, and that the certificate chains to the exact CA in clientCa:

openssl verify -CAfile /etc/cronstable/tls/client-ca.pem /etc/cronstable/tls/ops.pem

A job reports a TLS handshake failure against the state endpoint. The message names CRONSTABLE_STATE_CACERT. Either state.jobApi.tls.ca points at the wrong anchor for the certificate state.jobApi.tls.cert serves, or the endpoint's certificate lacks a SAN for the host in state.jobApi.listen. That URL is the configured host verbatim, so it is the name the certificate must cover. A job that reports the endpoint as unreachable instead is a different fault. That one really is a bind, port, or network problem.