Deployment - developerz-ai/ultimate GitHub Wiki
One image, N roles. Build once; the ROLE env var selects behavior. No role-specific Dockerfile, no per-role dependency set, no drift between what you tested and what runs.
As of 2026-08. Stable API โ semver from here (Upgrading). All three artifact targets ship โ x build --target docker, x build --target binary, x build --target static โ plus x build --target prebuilt, the step the app image runs on itself (As of 2026-10), and so do the compose files and the Helm chart. Milestone 11 is ๐ง on one thing (roadmap): the two-platform proof โ the demo app on Compose and on K8s from one image, with a rolling restart invisible to connected clients.
docker build -t myapp . # once
ROLE=web myapp # โ same image
ROLE=sync myapp
ROLE=worker myapp
ROLE=scheduler myapp
ROLE=migrate myapp # pre-deploy hook
ROLE=replicator myapp
| Role | Does | Scales on | Notes |
|---|---|---|---|
web |
SSR + static + RPC (actions/queries over HTTP) | RPS | behind CDN, stateless, N replicas, no local state |
sync |
live queries + fanout over WebSockets | concurrent connections | stateless, no sticky sessions โ a client may reconnect to any node |
worker |
jobs + steps | queue depth | one pool per queue served; WORKER_QUEUES=default,integrations claims exactly those and nothing else, unset claims jobs.queues plus every registered job's queue (Configuration) |
scheduler |
cron dispatch โ enqueue only | fixed 1 | leader election is an expiring row in x_scheduler_leader (postgresLeaseLeader), never an advisory lock โ that grant is session-scoped and the executor is a pool. A second instance is a warm standby, not a duplicate |
migrate |
run-once, pre-deploy | n/a | applies migrations through the ledger and exits; never binds a port. Holds the migration advisory lock on one pinned session for the whole run, so overlapping deploys serialise โ the second waits, polling once per 500ms for up to 60s, then exits non-zero with X_MIGRATE_CONCURRENT rather than hanging the rollout (As of 2026-08; 1.2.0 waits forever โ Known gaps) |
replicator |
logical replication โ change feed โ matcher โ NATS | 1 per database | owns the replication slot; a second instance would double-deliver, so it takes an advisory lock โ a container that loses it stays up, /readyz 503, and takes over when the holder goes (x dev --role replicator refuses with X_REPLICATOR_SLOT_HELD instead) |
- No role holds durable state. Everything survivable is in Postgres, NATS, or object storage.
- Only
migrateexits non-zero with a typed error when its lock is held (X_MIGRATE_CONCURRENT), andx dev --role replicatorrefuses (X_REPLICATOR_SLOT_HELD). Aschedulerorreplicatorcontainer that does not hold its lease or lock stays up as a standby โ unready, retrying โ rather than running degraded or double-delivering. -
There is no
ROLE=all. Those six names are the whole set; anything else isX_ROLE_UNKNOWNat boot. For dev,x devco-locatesweb,sync,workerandschedulerin one process โ role isolation is simulated, not skipped, and--roleopts thereplicatorin.
PORT selects the bind port, default 3000. Empty or whitespace falls back to the default; anything else must be an integer in 0โ65535 or the process refuses with X_PORT_INVALID rather than quietly binding 3000 and failing the platform's health probe with nothing in the log that names the cause. The production entry is the scaffolded apps/web/server.ts โ CLI reference.
Every serving role answers both. web and sync on their HTTP server; worker, scheduler and replicator construct none and answer them on the metrics listener (METRICS_PORT, default 9090, packages/cli/src/metrics-endpoint.ts) with the verdict only โ { state, ready, role }, never the check names. There /readyz is 503 until the role has started and from the first instant of a drain, which is what the chart's readiness probe on those roles reads. Both endpoints return a body, never a bare 200 OK.
| Endpoint | Answers | 503 when | Consumer |
|---|---|---|---|
/healthz |
"is this process alive?" | the lifecycle is stopped. Ignores the readiness checks on purpose: a database outage that failed liveness fleet-wide restarts every pod into the same outage, cold |
liveness probe โ restart |
/readyz |
"should traffic come here?" | starting, draining, stopped, or any registered check answers failing
|
readiness probe โ remove from rotation |
The body is { state, ready, role } for everyone, and { state, ready, uptimeMs, inflight, buildId, checks, registered, role } โ checks a map of name โ ok / failing โ for a peer configureHttp({ healthDetailPeers }) lists (default ['loopback']: kubectl exec, a port-forward, the container's own healthcheck). The status code is the same for both, and is all a probe reads. A published container port is NOT loopback: the caller arrives from the bridge gateway, so list 'private' to read the detail through -p. The sync role's own port answers by the same list, and tells a request carrying Forwarded or X-Forwarded-For the verdict only.
| Registered check | When it exists | What it reads |
|---|---|---|
database |
always | the previous db.ping(), refreshed on read โ never awaited inside the probe |
transport |
only when the transport can report a connection: NATS can, the in-process bus cannot |
transport.connected. failing on sync and the replicator; degraded on web, worker and scheduler, which only publish to the bus and serve without it (As of 2026-10) |
| anything per-role |
never As of 2026-08-22
|
this table used to list five โ replication lag, migration-version skew, "one pool claiming", "holds the leader lock", "slot active" โ and none was wired. The one since is replicator (As of 2026-10-02: failing while its stream is down). A scheduler standby does not report not-ready: its /readyz answers exactly as the leader's |
health: { readiness: 'process' } in app.config.ts (As of 22.5; default 'dependencies'). Every replica shares the database, so one blip fails database everywhere at once and the ingress answers "no available server" for the whole site โ pages that never touch the database included. In 'process' mode /readyz is 503 only while starting, draining or stopped; a failing check stays 200 with ready: true and the check still named failing in checks. /readyz?deep=1 always answers in 'dependencies' mode โ point monitoring there. Web and sync roles alike; liveness is unchanged in both modes.
A check has three answers: ok, degraded, failing (As of 2026-10). degraded is a dependency the role serves without โ today, the bus on a role that only publishes to it. It never turns /readyz into a 503, in either mode, so a NATS outage cannot pull every web pod from the ingress; /readyz?deep=1 is strict and answers 503 for it. An app registers its own with registerReadinessCheck(name, check, { onFailure: 'degraded' }).
Which build answers is a header, on every health answer (As of 27.2.3). /healthz, /readyz and /readyz?deep=1 send x-ultimate-build โ the same header every page sends โ on the web port, the sync port and the metrics listener alike, whoever asks. The body does not change: a stranger still reads { state, ready, role }. A deploy check that waits for the new build polls curl -sI https://<host>/readyz | grep -i x-ultimate-build rather than a page. Two cases send none: a web role whose buildId is null (no BUILD_ID and none from the boot), and the metrics listener in the moments before its boot has computed the id โ a container opens it first, so a cold pod's probe gets the status without the header.
registered: 0 is a real answer, not an error. An empty registry is still ready, so a 200 means no more than "the socket is bound" โ which is what the chart's and compose's healthchecks route traffic on. Read registered before you trust checks: {}.
Identical in every role. Framework behavior, not a deployment guide.
SIGTERM
1. /readyz โ 503 (web, sync: the listener stays open for drain.readinessGraceMs โ
5 s outside development/test โ so the LB stops routing here
first; every other role: no grace)
2. stop accepting new work (HTTP: close listener; worker: stop claiming; sync: refuse new upgrades)
3. finish in-flight work (bounded by drain.deadlineMs in app.config.ts, default 25 s)
4. role-specific handoff (see table)
5. flush OTel spans + logs
6. close pools, release advisory locks, exit 0
Both numbers are app.config.ts's drain section, applied to every role by the production boot
(Configuration). The budget is what a running job gets on a deploy, so an
app with long jobs raises it there: drain: { deadlineMs: 600_000 }. The platform's kill timer
must outlast grace + budget โ the Helm chart derives each role's terminationGracePeriodSeconds
from the values x deploy --method helm passes it; on Compose, stop_grace_period is yours.
| Role | Step 4 handoff |
|---|---|
web |
let in-flight requests and streaming responses finish; a stream past the deadline gets a typed truncation, not a socket reset |
sync |
send every client a reconnect frame with a per-client backoff delay (see below), then close cleanly |
worker |
stop claiming at step 2 and cancel nothing: every held job keeps running, and one that finishes inside the budget is acked. At the cut-off โ the worker's budget (drain.workerDeadlineMs when declared, else drain.deadlineMs) less a margin (2 s, or half of a budget under 4 s) โ a job still running has its ctx.signal aborted with X_DRAINING; a step that completes after that is still recorded. Half a margin later every claim still held is handed back uncounted (countsAsAttempt: false), so the replacement worker claims it at once โ not after the visibility timeout โ and replays every completed step instead of re-running it. A worker started after the signal (a boot still in progress) claims nothing |
scheduler |
delete the lease row immediately so the standby promotes on its next round rather than waiting out the 30s TTL |
replicator |
flush the change feed to NATS up to the last confirmed LSN, then release the slot |
A hook or request still running at drain.deadlineMs (the worker's: drain.workerDeadlineMs when declared) is abandoned โ logged as X_SHUTDOWN_TIMEOUT naming it, never thrown, and the process exits clean; requests arriving during the drain get X_DRAINING.
The SIGTERM path above already finishes held jobs โ inside the worker's drain budget: drain.deadlineMs (at most 1 h), or drain.workerDeadlineMs (at most a day, the worker alone) when declared. A worker that must outlast it โ a job whose side effect may happen at most once, on a pod with a 2 h grace โ is retired instead: send it SIGUSR2 (As of 25.0.0, ROLE=worker only, packages/cli/src/serve-retire.ts).
| Step | What the worker does |
|---|---|
| SIGUSR2 |
isRetiring() (@ultimat3/core) turns true for the rest of the process; logs jobs.worker.retiring (inFlight), stops claiming |
| held jobs | every one runs to the end, however long โ nothing is aborted, nothing handed back |
| done | logs jobs.worker.retired, then the ordinary drain, then exits 0
|
It exits rather than holds: the only thing a preStop can observe is PID 1 going away, and a worker that claims nothing has nothing left to serve. A second SIGUSR2 joins the first. One during boot waits for the worker, then retires it. A SIGTERM that lands mid-retire is the ordinary drain: the wait is bound to the worker's budget (drain.workerDeadlineMs, else drain.deadlineMs) and cut off at the margin, as in the table โ so a worker whose jobs outlast the hour declares drain.workerDeadlineMs. isDraining() stays false until that drain; the app's own code (a heartbeat, a status row) reads isRetiring() to say "finishing, about to exit" โ x dev's restart, which retires the same way, sets it too. Outside a pod deletion an exit 0 is restarted by the Deployment like any other: a fresh worker.
The Helm chart wires it behind one value: roles.worker.retireSeconds (0, off, by default). Set it to the longest a held job may run, and the worker's preStop sends its PID 1 SIGUSR2 and waits for it to exit; the same seconds are added to its terminationGracePeriodSeconds, which the kubelet counts the preStop against. The hook is bun -e, the one binary every app image carries โ the image's ENTRYPOINT ["bun", "apps/web/server.ts"] is exec form, so bun is PID 1. Outside the chart, the target is the same: kill -USR2 1 inside the container, or docker kill --signal USR2 <container>.
helm upgrade app docker/helm --set roles.worker.retireSeconds=7200 \
--set drain.workerDeadlineSeconds=7500 # 2 h worker; x deploy derives the second itselfWindows has no SIGUSR2: there the retire installs nothing and the worker stops only through the drain above (Ctrl-C, Ctrl-Break). A native Windows process is for development and CI.
Windows never sends SIGTERM to a console process, so the set is per platform โ drainSignals(platform) in @ultimat3/core, installed by every role As of 2026-10-05:
| Host | Drains on | Not a drain |
|---|---|---|
| Linux, macOS, any container (Docker Desktop on Windows included โ the image is Linux) |
SIGTERM, SIGINT (Ctrl-C) |
SIGHUP: a terminal hang-up, not a stop |
native Windows (x dev, a local x build --target binary run) |
SIGTERM, SIGINT (Ctrl-C), SIGHUP (the console window closing), SIGBREAK (Ctrl-Break) |
taskkill /F and Task Manager's End task: no signal, an outright kill |
Stop a native Windows process with Ctrl-C or Ctrl-Break, and the drain log line names the signal ({"msg":"draining","signal":"SIGBREAK"}). Closing the console window also drains, inside the few seconds Windows allows before it kills the process. A native Windows binary is for development and CI. Deploying on a Windows host means the Linux image under Docker Desktop, where SIGTERM works as on any other host.
Closing 50,000 sockets at once means 50,000 simultaneous reconnects, all resubscribing, all asking "what changed since my LSN?" โ a self-inflicted DDoS landing during a deploy when capacity is already reduced, and it is fractal: surviving nodes overload, drop connections, and the herd re-forms.
{ "type": "reconnect", "v": 3, "afterMs": 1830, "reason": "drain" }
ReconnectFrame in packages/realtime/src/sync-protocol.ts โ four fields, built by
reconnectFrame() (thundering-herd.ts); v is PROTOCOL_VERSION, reason one of drain,
overload, rebalance.
| Property | Effect |
|---|---|
Per-client afterMs, jittered over a window |
reconnects arrive spread out, not as a spike |
The window is syncNode({ drainSpreadMs }), default 30 s |
each socket draws its own delay inside it (drainPlan) |
| The resume point is the client's, not the frame's | the frame carries no cursor. On reopen the client sends hello, then one subscribe per live query carrying that query's own cursor, and re-announces every channel from its own; the node replays from its change buffer when the cursor is inside it and serves one snapshot when it is not (live-resume.ts) |
| Clients redistribute | the LB places them across remaining nodes; no sticky session to honour |
| Client-side backoff is a floor, not the mechanism | a client that loses the socket without a frame still backs off exponentially with jitter; one that got a frame waits its afterMs instead |
sync takes both shutdown phases, and they answer different questions As of 2026-08. The accept phase calls stopAccepting(): /readyz flips to 503 and an upgrade arriving anyway is shed with retry-after-ms, while every socket the node already holds keeps its patch stream โ sockets are untouched and no reconnect frame has been sent yet. The close phase is the drain below, then stop(). Registered with no phase, the whole thing landed in close, and until that last phase ran the node went on upgrading new websockets onto a process that was going away.
There is no realtime.drain config key โ the spread window is syncNode({ drainSpreadMs }), default 30s, and the grace is drain({ graceMs }), default 5s (Configuration). The reconnect benchmark that gated topology now exists: 50,000 sockets, a SIGKILLed sync node with no drain and no reconnect frame โ all 50,000 reconnected on their own backoff, 49,981 received a channel patch inside the window at p50 54.0s / p90 105.5s, 156,851 connect attempts shed before any query path. That is time to the first patch on the reconnected socket โ reachability, not consistency โ and it is the floor this section's frame is meant to beat. Measured on one node (Realtime).
x build --target docker # one image, all roles (default)
x build --target binary # a Bun-compiled launcher: no Bun on the host, but the app tree beside it
x build --target static # site/ output only: HTML, assets, sitemap, feeds
x build --target prebuilt # the line the Dockerfile runs INSIDE the image build
| Target | Output | Use |
|---|---|---|
docker |
one OCI image, ROLE selects behavior |
the normal path |
binary |
.x/app โ bun build --compile of apps/web/server.ts, ROLE selects behavior. A launcher, not a self-contained artifact: it embeds the Bun runtime, and at run time reads the app's package.json and tsconfig.json, imports app.config.ts and apps/*, and loads @ultimat3/* from the app's node_modules (kept external โ binaryArgs, packages/cli/src/cmd-build.ts). Start it from the app root with the source and installed node_modules beside it. What CI proves As of 2026-10-06: the required windows job builds a scaffold with x build --target binary, starts .x/app.exe from the app root and needs 200 from /healthz (.github/workflows/ci.yml); on Linux, packages/cli/e2e/binary-launch.e2e.test.ts compiles a probe tree with the same binaryArgs and runs it โ a probe, not a scaffold. Not yet served from a bare VM (Known gaps) |
VMs, systemd, air-gapped, a CLI-shaped product |
static |
.x/static โ 0kb-JS pages, hashed assets, sitemap.xml, robots.txt, feeds |
CDN / object storage, deployed independently |
prebuilt |
node_modules/.cache/ultimate/ โ every island chunk and every compiled stylesheet. No gate, no subprocess, takes neither --tag nor --out
|
never by hand: it is docker's other half, a RUN line in the image |
A pod builds nothing at boot. As of 2026-10-01. The scaffolded Dockerfile carries, after COPY . .:
RUN bun node_modules/@ultimat3/cli/src/bin.ts build --target prebuilt| Rule | Why |
|---|---|
| it runs in the image build, never on the host | the store is valid for one Bun, one framework version and one set of absolute paths โ the image's |
above ENV NODE_ENV=production
|
it imports the app with no deployment environment; in production app.config.ts asks for values no image build has |
above ARG BUILD_ID
|
two images differing only by their stamp share the layer |
it writes under node_modules/, not .x/
|
.x/ is state: every ignore file drops it and the compose topology mounts a tmpfs over it |
| a module that will not import fails the image build | its stylesheets would be missing, and every pod would compile them |
An image without the line still serves. Every boot then runs Babel over every island and Sass over
every stylesheet, and logs one error line saying so โ X_IMAGE_NOT_PREBUILT, with the counts,
the milliseconds and the line above. Measured on the demo app's image, ROLE=web, four boots each:
| time to ready | boot CPU | peak RSS | settled RSS | |
|---|---|---|---|---|
| no store | 4.2โ4.8 s | 8.3โ9.5 CPU-s | 244โ269 Mi | 215โ220 Mi |
| prebuilt | 1.6โ1.7 s | 1.8โ2.0 CPU-s | 141โ147 Mi | 97โ102 Mi |
An app scaffolded before 23.0.0 does not have the line โ its docker/Dockerfile is the app's
own file. Add it; the boot log names it until you do (Upgrading).
All targets share one build ID (content hash), stamped into the image, the HTML, the assets, x.manifest.json โ and sw.js, As of 2026-08: the worker's cache names carry the build id, so a deploy retires the previous build's caches instead of serving them (#390). Every target that serves or writes a document emits manifest.webmanifest, sw.js and x-sw-register.js alike (PWA and offline).
$ x build --target docker
โ typecheck + boundaries โ site/ 12 routes static 0kb js
โ app/ 31 routes stream โ static assets avif+webp, 214 files
โ manifest + openapi emitted โ image myapp:8f2a1c9 118MB
build id 8f2a1c9
x build runs six of x verify's steps first โ typecheck, lint, boundaries, filesize, package-shape, errors โ and produces no artifact if any fail. It also refuses before spawning the builder when the target's entry file is missing (X_BUILD_ENTRY_MISSING) โ CLI reference.
# docker/docker-compose.dev.yml โ only needed for parity checks; `x dev` needs none of this
services:
db: { image: postgres:17-alpine, ports: ['127.0.0.1:5432:5432'] }
nats: { image: nats:2-alpine, command: ['-js'], ports: ['127.0.0.1:4222:4222'] }
s3: { image: versity/versitygw:v1.8.0, ports: ['127.0.0.1:9000:9000'], volumes: ['s3bucket:/data/myapp'] }x dev uses embedded Postgres, in-process NATS, and a local directory for S3 โ Docker is not required to develop. This file exists for parity debugging: three backing services and no app service, because the app in development is x dev on the host, pointed at them by environment (the file's header is the line to paste).
| Service | Rule |
|---|---|
| every port | binds 127.0.0.1 โ the credentials are in the file |
s3 |
Versity S3 Gateway over a directory. A directory under /data is a bucket, so the volume mounted at /data/<app> is the bucket: ready on the first up, no CreateBucket call |
S3_REGION |
not needed against it: an unset region signs for auto, and the gateway is started with that region. An endpoint that expects another refuses the first write with X_CONFIG_INVALID, naming the value to set |
| production | any S3-compatible endpoint. This file picks a server for a laptop, never for a deployment |
# docker/docker-compose.prod.yml
x-app: &app
image: myapp:${BUILD_ID}
env_file: [../.env.production] # what the CONTAINERS see โ not what `${VAR:?}` reads
restart: unless-stopped
services:
migrate: { <<: *app, environment: { ROLE: migrate }, restart: 'no' }
web: { <<: *app, environment: { ROLE: web, PORT: 3000 }, ports: ['3000:3000'],
deploy: { replicas: 1 },
depends_on: { migrate: { condition: service_completed_successfully } } }
# PORT is the WEB port even here: `sync` binds PORT + 1, so 3000 is a listener on 3001.
sync: { <<: *app, environment: { ROLE: sync, PORT: 3000 }, ports: ['3001:3001'],
deploy: { replicas: 1 } }
# WORKER_QUEUES is the EXACT set a worker claims (25.0.0): `integrations` runs on its own
# service, so the other worker names every remaining queue itself. Unset, a worker claims
# jobs.queues plus every queue a registered job is enqueued on; set, `jobs.worker.queue-unserved`
# at boot names the registered queues this one leaves to another.
worker: { <<: *app, environment: { ROLE: worker, WORKER_QUEUES: 'myapp-default,default' },
deploy: { replicas: 4 } }
worker-integrations: { <<: *app, environment: { ROLE: worker, WORKER_QUEUES: integrations },
deploy: { replicas: 1 } }
scheduler: { <<: *app, environment: { ROLE: scheduler }, deploy: { replicas: 1 } }
replicator: { <<: *app, environment: { ROLE: replicator }, deploy: { replicas: 1 } }| Rule | Reason |
|---|---|
migrate completes before web/sync start |
a new schema must exist before new code reads it |
web and sync at 1 replica |
each publishes a host port, and a host port has exactly one binder |
PORT: 3000 on sync, published as 3001:3001
|
the sync role binds PORT + 1; naming 3001 opens 3002 and publishes a socket nothing in the container ever opened |
SYNC_URL=ws://<host>:3001/_x/sync in .env.production, read through --env-file
|
a page dials /_x/sync on its own origin by default, and on this rung that is web's port, which does not serve the socket. With no proxy in front, only the deployment knows where sync is published. The shipped compose file refuses to start web without it (${SYNC_URL:?โฆ}). Must be ws:// or wss://, else X_CONFIG_INVALID at boot. Since 21.0.0. |
APP_URL=http://<host>:3000 in .env.production, reaching sync
|
the sync node admits a socket from APP_URL's origin and no other once one is declared โ scheme, host and port, exactly; with none declared, from the origin it was reached on. On this rung the page is :3000 and the socket :3001, two origins, so without it every upgrade is 403 X_SOCKET_ORIGIN_REFUSED. The shipped compose file refuses to start sync without it (${APP_URL:?โฆ}). One proxy serving both on one host needs none (a PaaS: a node reached over plain http admits the https spelling of that host and port), and the Helm chart sets APP_URL on the sync role from ingress.host
|
POSTGRES_PASSWORD in .env.production, read through --env-file
|
x new's compose file runs its own db service and refuses to start it without one (${POSTGRES_PASSWORD:?โฆ}) |
every command passes --env-file .env.production, before -f
|
Compose interpolates ${VAR:?โฆ} from the shell and --env-file only, never from a service's env_file:. Without the flag a value set only in .env.production reads as missing and the parse fails. x deploy passes it on every step (packages/cli/src/cmd-deploy.ts, PROD_ENV_FILE); a variable set in the shell still wins |
scheduler and replicator at 1 replica |
leader lock makes a second one a standby, not throughput |
stop_grace_period โฅ drain.readinessGraceMs + drain.deadlineMs + 10 s (40s ships, for the defaults); the worker service's โฅ drain.workerDeadlineMs + 10 s when declared |
otherwise SIGKILL truncates the drain and the reconnect fanout. Raise it with drain.deadlineMs (the worker's with drain.workerDeadlineMs) โ nothing derives it on this rung |
| roles that publish no host port are rolled start-first |
x deploy scales the new container up beside the old with --no-recreate, waits for its healthcheck, then stops and removes the old one; web and sync publish one and are recreated stop-first until a proxy fronts them (docs/ops/README.md) |
Health probes from /readyz
|
never from a TCP check โ a process can accept sockets while unable to serve |
x deploy --method compose applies this against the committed docker/docker-compose.prod.yml; it is a plain compose file you can read, diff, and run by hand โ from the app root, as docker compose --env-file .env.production -f docker/docker-compose.prod.yml up -d.
webandsyncare one replica each, and the file says soAs of 2026-08. Both publish a host port, one host port has exactly one binder, so both declarereplicas: 1โ a ceiling that is declared rather than discovered when the second container dies onport is already allocated.worker,schedulerandreplicatorpublish nothing;workerscales freely.
Want Do more web/syncon the same boxdelete their ports:lines, add a reverse proxy of your choosing to the compose file, point it at the service names โ compose DNS resolves each to every replicamore web/sync, full stopthe Helm chart below, which carries a per-role HPA and an Ingress The framework ships neither proxy: a proxy image in
docker-compose.prod.ymlis a dependency every app inherits and a second answer to "how does traffic reach a role" beside the chart's Ingress. This is the rung-1 ceiling โdocs/idea/17-scale-ladder.md.
x new writes it, As of 2026-08-19 โ docker/helm, one Deployment per role, 8 files: Chart.yaml, values.yaml and 6 templates. There is still no --helm flag on x build; the chart is a scaffold artifact like docker/Dockerfile, not a build target.
| Role | HPA metric | Typical range | Notes |
|---|---|---|---|
web |
requests/sec (or CPU as fallback) | 3โ50 | behind Ingress + CDN; terminationGracePeriodSeconds derived per role from drain.* in values (45 s web/sync, 35 s the rest, by default) |
sync |
active WS connections (custom metric) | 2โ100 | no session affinity; connection count is the only honest signal |
worker |
queue depth per named queue (custom metric) | 2โ200 | one Deployment per queue when isolation matters |
scheduler |
none โ replicas: 1
|
1 |
PodDisruptionBudget maxUnavailable 1, leader lock covers overlap |
replicator |
none โ replicas: 1
|
1 | owns the slot; strategy: Recreate, because a surged pod can only stand by unready and maxUnavailable: 0 would never retire the holder |
migrate |
n/a | โ | pre-install/pre-upgrade Job hook; blocks the release on failure |
CPU autoscaling is wrong for sync and worker: a node holding 80k idle sockets is near-zero CPU and near-capacity, and a worker blocked on a slow HTTP call is idle CPU with a growing backlog. The framework declares both series โ connections and queue_depth, with rps derived from the monotonic http_requests_total โ and SCALING_METRICS maps each role's signal to its series so the chart and the role table cannot drift. As of 2026-08 every role serves /metrics on METRICS_PORT (default 9090) and http/realtime/jobs call the recorders, so the signals exist. The chart's half is closed in 2.0.0: values.yaml declares metricsPort: 9090, _helpers.tpl emits a container port named metrics on every role but migrate, service.yaml publishes it by name and templates/servicemonitor.yaml ships the scrape target. Two things remain, and neither is the chart's: serviceMonitor.enabled defaults false, because a cluster without the Prometheus operator has no such CRD and helm install would fail on an unknown kind; and turning scraped series into the Pods metrics an HPA reads needs a custom-metrics adapter. Do not hand-add a metrics container port โ the chart already emits one and a duplicate is rejected by the API server โ Observability.
x deploy --method helm works in a fresh app As of 2026-08-19: the command implements helm
completely and its X_NOT_IMPLEMENTED branch โ which claimed the build did not implement it, over
a build that did โ is deleted. An app that deleted its chart now gets helm's own error. The image
goes in as --set-string image.repository=โฆ --set-string image.tag=โฆ As of 2026-10: a plain
--set typed an all-digit tag (a build number) as an integer and rendered app:%!s(int64=โฆ).
As of 2026-10-05 it also passes the chart values --set drain.deadlineSeconds=โฆ --set drain.readinessGraceSeconds=โฆ,
read off the app's own drain config, so the chart's grace periods follow a raised budget. Every role carries a readiness probe on /readyz (worker and scheduler on
the metrics port), every Deployment minReadySeconds: 10, and every rolling role keeps
maxUnavailable: 0.
The framework repo's own docker/helm
carries two templates the scaffold does not โ pdb.yaml and servicemonitor.yaml. Neither ships in
an npm tarball, so taking them is a git clone of this repo. Its image.repository is a placeholder
(registry.example.com/your-app) โ the framework publishes no app image โ and an emptied one is a
render error naming the key. On 3.0.0 and below, x new writes no
chart at all and --method helm exits X_NOT_IMPLEMENTED: copy the chart in, or use
--method compose.
As of 26.1.0. A proxy in front of both roles (an Ingress, a Cloudflare tunnel, nginx, an ALB)
sends the sync role exactly one path family, and everything else to web:
| Route to | Exactly | Regex |
|---|---|---|
sync |
the socket, /_x/sync โ and anything under /_x/sync/, which the framework reserves for it |
^/_x/sync(/.*)?$ |
web |
everything else โ pages, actions, queries, /_x/assets/โฆ, /islands/โฆ, /healthz
|
โ |
Anchor the rule. ^/_x/sync with no end anchor is the rule an operator writes first, and it is
safe as of 26.1.0 only because nothing else the framework serves starts with that string. Through
26.0.0 the sync worker lived at /_x/sync-worker/<hash>.js: that rule sent it to the sync role,
which serves only the socket, so no page got its shared worker (no realtime), and the 404 came back
with no cache-control, which a CDN pinned for its default TTL (Cloudflare: four hours).
Every framework-served, content-addressed script lives under one asset prefix, /_x/assets/ โ
the Rails rule: <prefix>/<name>/<sha>.js, served immutable, a new URL per build:
| Script | URL |
|---|---|
| the sync worker | /_x/assets/sync-worker/<hash>.js |
| the page boot | /_x/assets/page-boot/<hash>.js |
| the client router | /_x/assets/navigation/<hash>.js |
Island chunks (/islands/โฆ), surface stylesheets and asset() files have prefixes of their own; no
live endpoint shares any of them. A miss on any content-addressed path is 404 with
cache-control: no-store โ a CDN never keeps it โ and so is every answer of the sync role that is
not the socket (a stray path, a plain GET of the socket path, a shed upgrade). Old /_x/sync-worker/,
/_x/page-boot/ and /_x/navigation/ URLs are not served: each named one build's file, which no
other build serves either, and a tab mid-roll that already started its worker keeps it.
TRUSTED_PROXY_HOPS counts the proxies that append to x-forwarded-for between the client and
web/sync โ Configuration. Unset trusts no proxy header. Proxies treat the two
headers differently, and the reader follows that:
| Header | What proxies do | What the framework reads |
|---|---|---|
x-forwarded-for โ ctx.ip
|
append their peer (nginx $proxy_add_x_forwarded_for, an ALB, Envoy) |
the entry hops from the right. A shorter list is not the declared chain: nothing is trusted and the socket address is used |
x-forwarded-proto โ ctx.https (HSTS, the CSRF self-origin) |
usually overwrite (nginx $scheme, an ALB, Traefik); some append |
a full list: the entry hops from the right. A shorter list: its first entry โ an overwrite erased what the client sent, so what is left was written by a trusted hop |
As of 2026-10, two hops with an overwriting proto proxy read https, so HSTS goes out. Before, a
one-entry proto header behind TRUSTED_PROXY_HOPS=2 counted as "not the chain": ctx.https was
false and no HSTS was sent.
Make sure at least one trusted hop writes x-forwarded-proto. If none does, the client's own
value reaches the app, exactly as it always has with TRUSTED_PROXY_HOPS=1. The edge that ends TLS
is the one to set it: nginx proxy_set_header X-Forwarded-Proto $scheme;. An inner proxy reached
over plain http must pass the edge's value on rather than writing its own $scheme โ on
ingress-nginx, use-forwarded-headers: "true".
x build --target static # dist in .x/static โ upload it with your CDN's own tool
| Property | Consequence |
|---|---|
| Static build does not include the app image | a copy change, a new blog post, a pricing tweak does not redeploy the API |
| Independent version, shared build ID namespace | assets stay resolvable across N deploys (PWA and offline) |
| ISR pages regenerate server-side and push to the CDN | no full rebuild for one changed record |
| Rollback is a pointer swap | seconds, no container churn |
| Cache purge | tag-driven, one hop from the write (Caching and invalidation) |
The only requirement: something that runs containers, plus Postgres. NATS and object storage are optional in small deployments โ Postgres covers queue and pubsub; a local volume covers files.
| Target | How | Notes |
|---|---|---|
| Hetzner + Compose |
compose.prod.yml on one or two boxes |
cheapest credible production; one node runs all roles |
| Fly.io | one app per role, or process groups | drain semantics map cleanly to Fly's SIGTERM handling |
| Railway / Render | one service per role, same image | set ROLE per service |
| AWS ECS / Fargate | one task definition per role | ALB for web, NLB for sync
|
| Any Kubernetes | the generated Helm chart | EKS, GKE, AKS, k3s โ no cloud-specific resources |
| Bare VM |
--target binary + systemd units per role |
no container runtime at all |
Not supported, by design: vendor edge runtimes, serverless-function-per-route, vendor KV / queue / cron primitives, proprietary image loaders. Each would need a second implementation of a framework primitive, and the second implementation is where behavior diverges.
x verify # the gate โ green means shippable
x build --target docker
ROLE=migrate <image> # pre-deploy, must exit 0
<roll web + sync> # drain-aware; clients reconnect with backoff
x build --target static # independently, whenever copy changes
x doctor --json # env, versions, drift, ports
The framework depends on none of this; it is the operations manual for the app you deploy with it. Recommendations, not contracts โ nothing in packages/ reads a word of it.
| Doc | Answers |
|---|---|
docs/ops/README.md |
the PaaS โ Compose โ Kubernetes ladder, and which rung you are on |
docs/ops/01-kubernetes.md |
the chart, one Deployment per role, probes, PDBs, HPAs |
docs/ops/02-secrets.md |
env or a mounted file, and nothing vendor-shaped |
docs/ops/03-observability.md |
what to scrape, what to alert on, what is not implemented yet โ Observability |
docs/ops/04-datastores.md |
sizing Postgres, NATS and object storage for a given rung |
docs/ops/05-disaster-recovery.md |
backups, PITR, restore drills, replication-slot recovery |
docs/ops/06-runbooks.md |
symptom โ page โ action, per role |
Two design-only companions โ specification, not shipped behaviour: docs/idea/16-app-targets.md (three targets, one backend, two view layers) and docs/idea/17-scale-ladder.md (why the app code is identical at rung 0 and rung 4).
Redeploy the previous image tag. The last 3 builds' assets stay served โ retentionPlan(deploys, keep = 3), a count of deploys with no time component and no pwa.retention config field โ so a rollback one release back does not 404 anyone mid-session. Version-skew handling: Upgrading. Failure symptoms: Troubleshooting.