Running Indexterity

Everything about working on this and operating it: the development loop, how a release is cut, the images and the three topologies, the environment variables that matter and why, the metrics worth alerting on, and what error reporting does and does not send.

Moved out of the README so that page can be read in a minute. What the product decides is in Architecture, what holds it shut is in Security, and the scoped user it needs is in Connecting a cluster.

Every key, component and the things that turned out to be load-bearing: Architecture §14.

Develop

cp .env.example .env      # then fill secrets
npm install
podman-compose up         # postgres + api + web, hot reload
# or npm run up           # the same, and recovers from stale container state

No mongod in the stack — connect whichever cluster you are working against. Which throwaway you want depends on who has to reach it:

# the integration and e2e suites, which run on the HOST (they take MONGO_URL)
podman run -d --rm --name mongo-test -p 27017:27017 docker.io/library/mongo:8

# a cluster you connect IN the stack — then use mongodb://mongo:27017
podman run -d --rm --name mongo --network mongo_optimizer_default \
  --network-alias mongo docker.io/library/mongo:8

The second one has to be on the compose network: the api is a container, so localhost is itself, and the host is only addressable as host.containers.internal, which resolves into 169.254.0.0/16 — a range src/engine/net-guard.ts files as FORBIDDEN and never dials, even with ALLOW_PRIVATE_CLUSTER_TARGETS, because the cloud metadata endpoint shares it. Clusters registered before the demo mongod was dropped from compose already hold mongodb://mongo:27017, and the api logs cluster <id> unreachable — skipped every tick until something answers to that name.

Anything topology-shaped needs a replica set, since $indexStats and $collStats latencyStats are both per-member — most easily several mongods as several processes in one host-network container.

npm run build · npm run typecheck · npm run lint · npm run test
npm run db:generate · npm run db:migrate
npm run db:deploy -w @repo/api        # production migrations, compiled migrator
npm run version:set 0.2.0             # workspaces + the chart + the lockfile
npm run version:check                 # assert every file that states it agrees

Releases are cut by hand, and the tag is the only trigger. Work integrates on dev: npm run version:set X.Y.Z in a chore: release X.Y.Z commit writes the version to the seven manifests, both chart fields and the eight places package-lock.json states it. A dev → main pull request promotes that, the merge is tagged vX.Y.Z, and the GitHub Release is written from the commits it carried. release.yml waits on the tag and nothing publishes from a branch, so what a cluster pulls cannot move because something merged.

A release that adds a privilege to a provisioned role says so in its notes, because the owners of every cluster connected before it are being asked for something. The notes name the privilege, what it turns on, and that nothing breaks without it (D184). version:set prints the list when it numbers the release, and refuses a change in mongo/role.ts labelled for a later release than the one being cut: everything in the tree ships in it.

version:check is what makes it safe rather than careful. One version lives in seventeen places, and release.yml runs the check against the tag before it builds or installs anything — so a tag cannot claim a version the tree does not carry. It earned that: 0.3.0, 0.4.0 and 0.5.0 each shipped with a stale package-lock.json, a tree npm ci refuses, while the one command whose whole job is reporting version drift said everything agreed (#186, D78).

npm run up is a convenience, not a requirement — podman-compose up works directly. It recreates containers whose crun state a logout cleared (cannot open .../exec.fifo; named volumes untouched), and drops node_modules/.bin from PATH, which matters only when compose is reached through npm — @vercel/nft installs an nft binary there and podman's network backend shells out to nft meaning nftables.

npm run lint is Biome plus two checks Biome cannot express. scripts/lint-tailwind.ts fails the build on a Tailwind arbitrary value that has a canonical utility (w-[220px] is w-55); vendored components/ui is exempt. scripts/lint-migrations.ts fails it on a migration that adds an enum value and then uses it — Postgres refuses that inside one transaction, and drizzle-kit runs one file per transaction, so the ALTER TYPE has to commit on its own and the backfill goes in the next migration. Both scripts verify their own cases on every run, because the way a check like that fails is by matching nothing.

Migration installs two schemas — public for Drizzle and graphile_worker for the queue — installed before the api serves, because a request that queues a job would otherwise race a schema that is not there yet.

Four test layers, currently 1,261 api unit, 548 web unit, 205 integration and 41 end-to-end.

layer	what it needs	what only it catches
npm run test	nothing	the pure decision engine; components in jsdom with the api client mocked at ~/lib/api
npm run test:int -w @repo/api	migrated postgres + a mongod	CI runs it against 6.0, 7.0, 8.x and 9.0 — they take different paths through the workload collector, and 9.0 keeps $queryStats' plan metrics somewhere else again
npm run test:e2e	both apps built	a real browser all the way to postgres and mongo, with no proxy in front, so the passthrough is the path under test. Two projects: chromium for the suite, and touch (a Pixel 7) for two specs that are only meaningful without a pointer
deploy/kind-test.sh	Kind	the chart actually installing, helm test, sign-up over cluster DNS; fails if any pod logged a warning. TOPOLOGY= runs the same assertions against each packaging, which is what holds them to being blind to it

Preloading will tell you a loading change is fine when it is not. defaultPreload: "intent" runs a route's loader 200ms after the pointer settles on a link, so on a development machine a click usually lands on data that is already there — which is how a loader that blocked navigation for the length of its reads survived as long as it did (D117). A tap is faster still: Link sends hover and keyboard focus through one delayed timer and onTouchStart straight to doPreload(), with no delay at all.

So do not judge a loader change by clicking it. Either watch it with the read held open (page.route with a timeout, as navigation.spec.ts and touch.spec.ts do), or throttle the network in devtools. The measurements behind this are in #401.

The top ones are not decoration. The e2e suite found the api's session cookie being percent-encoded a second time on its way through the web server — both sides correct on their own, the defect in the hand-off — and the dashboard drawing itself twice, which only a browser could see. The Kind run found that the chart could not install at all, while helm lint passed throughout.

A superseded CI run is cancelled, not finished. Pushes to main and dev are exempt — those runs are the record of whether an integration branch is sound. The release workflow has no such rule at all.

House rule: the api and the web app run clean. No errors and no warnings in server logs, build output, or the browser console. A warning is a defect — fix

is deliberately not done: Security.

Deploy

The api boots from one file (D176, #567). On the 0.1 CPU of a free hosting tier, the all-in-one container used to take 52–54 s before its public /api/health answered, and the host waits for that on every deploy and every wake. A CPU profile put most of the api's boot in node's module loader: resolving, stat-ing and reading swc's ~220 files and their package graph. So the build bundles dist/main.js with esbuild after swc (apps/api/scripts/bundle.mts). The container now answers in about 31 s at 0.1 CPU, and the api alone takes 2.4 s on one CPU, down from 5.3. Beyond that it is CPU: at 0.5 CPU, the size of a small paid instance, the same container answers in 7 s (11 s before the bundle).

Small images, and small on purpose. The api's images ship no node_modules at all (D182): every entrypoint they run — the api, and its migrate.js, rotate-key.js and set-plan.js CLIs — is one esbuild bundle, so the runtime stage copies those four files, the api's package.json and its migrations, and nothing else. The dashboard ships its self-contained .output the same way. The base is most of what remains, so it is node:26-alpine (181 MB) instead of node:26-slim (252 MB). Nothing shipped links against the C library — no .node file anywhere, and the tunnel binary is static — so that is a base swap and not a musl rebuild — and mongodb+srv://, the one lookup that would care, is tested on musl rather than assumed (D76). api 216 MB (was 510), all-in-one 238 MB (was 714), web 193 MB (was 264):

docker build -f apps/api/Dockerfile -t indexterity-api .
docker build -f apps/web/Dockerfile -t indexterity-web .

There is a third, for hosts that run one container per service and have no pods at all — Fly, Render, Cloud Run, a single VM. Same builds, same artefacts, both processes (three with the tunnel service) under a supervisor that is PID 1, forwards SIGTERM to each and exits non-zero if any dies:

docker build -f deploy/all-in-one/Dockerfile -t indexterity-all-in-one .
docker run -p 3000:3000 -e DATABASE_URL=… -e MASTER_KEY=… indexterity-all-in-one

Port 3000 is the only one that has to be published: the dashboard answers /api itself, so the browser gets everything from one origin. The Helm chart can deploy this shape too — topology: single-container.

Configuration

Every process validates its environment before it serves anything, against one schema per process rather than reads scattered through the code. absent is fine, malformed is fatal: an unset optional knob takes its default, and a malformed one is a boot failure naming the variable and what it expected. So AUTH_RATE_LIMIT_MAX=2O (letter O) refuses to start instead of quietly reading as 20, and a garbled TRUST_PROXY refuses instead of quietly meaning "no proxy in front" — which used to collapse every per-client rate limit into one shared bucket.

Three schemas, because three validation surfaces are given different things (apps/api/src/config/schema.ts):

Process	Demands	Because
migrate	DATABASE_URL	The pre-install Job talks to Postgres and exits
worker	+ MASTER_KEY	The pipeline's subset, and what the rotate-key CLI validates — it unseals stored credentials without serving HTTP
api	+ BETTER_AUTH_SECRET	Only the api serves auth (and it validates the pipeline's variables too, since it runs the jobs)

DATABASE_URL has to be the direct endpoint, not a transaction-pooled one — Neon's -pooler host, PgBouncer or Supavisor in transaction mode. LISTEN/NOTIFY does not survive transaction pooling: the LISTEN is accepted and the notification never arrives, because the listener and the notifier land on different backends. Nothing errors, and the dashboard's SSE quietly never fires — panels only refresh on their own staleTime. Since a shape of failure that logs nothing is worse than one that refuses, the api runs a NOTIFY round trip at boot (apps/api/src/db/notify-probe.ts: LISTEN a throwaway channel, pg_notify it from a second connection, wait two seconds, close both) and refuse to start when it does not come back — three attempts, so a Postgres that was restarting is a slow boot rather than a crashloop. The round trip is the test and the hostname only a hint: measured on PgBouncer 1.25, pool_mode=transaction loses the notification and pool_mode=session delivers it. A pooler buys this app nothing either way — it holds about 20 connections at its defaults (D89).

A registered VPN peering is carried by a service beside the api (wireguard-go and gvisor's netstack, apps/tunnel). It needs no capability and no /dev/net/tun, so it changes nothing about what the pod is allowed to do — it runs under the same drop: ALL and nonroot the app container does, with a read-only root filesystem, from a scratch image holding one static binary.

TUNNEL_PORT is the whole of the configuration, and the same variable turns on both halves: the api looks for the service on that loopback port, and the service binds it. Unset means the VPN feature is off — the api says so and the dashboard shows it as unavailable, which is a supported state rather than a missing setting. There is no host and no secret because the service is always in the api's own network namespace, so the peering's private key never crosses a network.

Where that container comes from depends on the deployment: the all-in-one image's supervisor starts it when TUNNEL_PORT is set, the chart adds it as a sidecar when tunnel.enabled is on, and docker-compose runs it with network_mode: service:api. On with no container in the pod is the one misconfiguration to watch: every tunnel open then fails against a port nothing is listening on, per tunnel rather than as a boot failure — a tunnel nobody has used yet cannot be checked at boot.

sslmode in DATABASE_URL is node-postgres's, not libpq's, and the two do not mean the same thing. require and verify-ca both mean FULL verification here — hostname included — and an absent sslmode means no TLS at all rather than opportunistic TLS. Node deprecated that aliasing and now warns on every connect, asking for the intent to be written down, so a production URL should say sslmode=verify-full outright. require still works and still verifies, but it warns each time and would silently become libpq's weaker meaning the day node changes its default. uselibpqcompat=true&sslmode=require is the opposite choice, spelled out — encrypt without checking who answered.

The dashboard server has its own, built on @t3-oss/env-core (apps/web/src/lib/env.ts), where the interesting part is the split: every variable is a server one, so reading one in the browser throws rather than returning undefined, and a value the browser genuinely needs has to be declared with the VITE_ prefix that makes it public by construction. There are none.

The schema is also the source of truth for the four places a variable has to be registered — .env.example, docker-compose.yml, the test suites, and the Helm chart. config/homes.test.ts walks it and holds all four to it, in both directions: a required variable the chart never sets fails the test rather than the deploy, and a variable the chart sets that no schema knows is a typo nothing else would have caught.

One web image serves every environment — API_URL and WEB_ORIGIN are read at runtime, and nothing about the api's address is baked into the browser bundle. The api runs the whole job pipeline itself (#232 retired the separate worker process): no resident runner, no LISTEN — a 30-second in-process tick claims what became due and drains it with runOnce, so between ticks an idle api holds nothing. Replicas are safe: passes are claimed per occurrence, so concurrent tickers cannot double-dispatch. The interval still queries twice a minute — a database that suspends on idle wants the external clock below instead.

The clock outside (RUN_CRONJOB=false). The api always executes jobs; this flag hands over the decision of when passes become due. Nothing then fires on its own — an external scheduler you own fetches the tick:

GET /api/internal/tick     Authorization: Bearer $CRON_TRIGGER_SECRET
  → {"dispatched":["scheduleApply","scheduleProbe"],"alreadyClaimed":[],"drained":true}

The request claims and drains — the same tick the in-process interval runs, because nothing LISTENs any more, so if this request did not drain, nothing would. It is bounded: past 25 seconds (under the 30-60s platform proxy floors) the response says "drained": false while the drain carries on in-process, and pinging again resumes a partially drained queue rather than duplicating it — which is what makes even a dumb every-five-minutes ping a working scheduler.

Choosing the interval on free serverless tiers. A five-minute ping works as a scheduler, but on a database that bills compute by the hour, five minutes is the wrong cadence for the work. Two thresholds decide it, and they pull in opposite directions:

• Neon suspends a compute only after 5 minutes with no queries. Every tick that reaches the database queries it, so a five-minute cadence leaves no five-minute gap and the compute never sleeps. That comes to about 730 hours a month, roughly 182 CU-hours at the smallest size, against the Free plan's 100. It runs out around day 16, and Neon then suspends the compute until the next billing period.
• Render's free tier spins a web service down after 15 minutes without inbound traffic, and waking takes about a minute, which is longer than most URL pingers wait. The ping has to stay well inside that.

So the ping and the work come apart. Keep the external ping at five minutes and set FAST_PASS_INTERVAL_MINUTES=15. The fast passes (apply, the read-pressure probe, and the stale-lock repair) then fall due once a quarter hour, and a ping that finds nothing due answers from memory without a single query:

  → {"dispatched":[],"alreadyClaimed":[],"drained":true,"idle":true}

Two pings in three look like that, and the database sleeps between the ones that do work. Expect roughly 60–80 CU-hours a month, depending on how long each drain keeps it awake. That is an estimate, not a measurement, so check Neon's usage page after a few days. A ping still goes to the database in three cases:

• the last drain did not finish;
• a request queued a collect ("collect now", or the first collect after connecting a cluster);
• a failed job is waiting on its retry.

The cost is cadence, and it is worth stating plainly. An approved drop hides up to 15 minutes later, and a missing index shows up in read latency up to 15 minutes later. The hourly passes are unaffected. With the cadence raised, three things the process cannot see also wait for the next quarter hour rather than the next ping:

• a collect queued on another replica;
• a retry whose backoff outlived the ping after its failure;
• a watermark edited by hand.

A single replica, which is what a free host runs, sees none of them. At the default of five, every ping asks the database, exactly as before. The value must divide 60, because occurrences count from the top of each hour, and the boot refuses anything else. The api needs nothing more for the database to sleep: idle pool clients close after pg's default 10 seconds, long before the compute does, and the dashboard's LISTEN is held only while somebody is watching (#223), in a tab that is actually shown (#548).

Measured on the hosted deployment in October 2026, 15 was not enough (#614). Neon showed 11.5 CU-hours after three and a half days, about 100 a month. Each of the four fast-pass occurrences an hour keeps the compute up for five minutes after it ends, and the hourly passes add their own length on top. With FAST_PASS_INTERVAL_MINUTES=30, the fast passes coincide with the hourly ones at :00 and :30, and the estimate falls to about 50 CU-hours. The cost is an approved drop hiding up to half an hour later.

Egress and the host's outbound transfer are limits of their own (#614):

• Neon allows 5 GB a month of egress. The overview's two latency reads are folded in postgres. Before that, they shipped every reading of every collection over their window, nearly 5 MB a load on a cluster of a hundred MongoDB collections.
• Render's Hobby workspace allows 5 GB a month of outbound bandwidth. Without a payment method on file, Render suspends the free services for the rest of the month when it runs out. The dashboard compresses what it sends. The CDN in front compresses for the browser, but Render bills the bytes it sends to the CDN, and those used to go out raw.

After every deploy, the first collect of a large SQL Server can time out. Each Query Store plan's tables are read off its XML once and remembered in the process (#478), and a restarted process starts from empty. What a timed-out collect did read is kept, so the next one or two finish.

Free-tier storage is the next limit after compute: Neon's is 0.5 GB, and the sweep keeps each org's own plan window, 90 days on FREE and 183 on PRO. See Plans and policy → History is kept for the plan's window, and no longer before relying on a free database for long.

GET, and only GET — POST is not routed and answers 404. The endpoint changes state, so by the letter of HTTP it wants POST, and that is what it was; the method turned out to be the thing standing between an operator and a working schedule. The audience for an external clock is the host that sleeps, and what such a host comes with is a URL pinger — an uptime monitor, a platform cron field that takes a URL, a status-page checker — most of which send GET and nothing else. What a safe method actually promises an intermediary is that it may repeat the request or reuse the answer, and both are already true here: repeating is free by construction (below), and reuse is refused outright, because every response this api sends carries Cache-Control: no-store. A cached 200 would otherwise be this endpoint's worst failure — a scheduler being told the pipeline ran while nothing runs at all.

Safe to call as often as you like, and that is a property rather than a promise — it is also what makes serving this over GET unremarkable: passes are claimed against their occurrence in worker_watermarks, so a hundred calls inside one five-minute bucket enqueue at most one scheduleApply. Two crons firing a minute apart compute the same occurrence and only one claim wins — no lock. A host that slept through eleven buckets owes one run per pass, not eleven, so catching up never dials the fleet eleven times over; retention stays on 03:00 and the weekly digest on Monday 09:00, which an interval reading would have drifted. CRON_TRIGGER_SECRET is required when RUN_CRONJOB=false — the endpoint authorises the whole pipeline and there is no user session behind it — and the boot refuses without it, because a deployment with no interval and no way in is a pipeline that is silently dead.

Hosts that sleep are a supported topology, with caveats named. On a host that sleeps when idle, or a database that suspends its compute, a resident schedule does not slow down, it stops — which is why the external clock exists. Any scheduler that can fetch a URL with a bearer header drives the pipeline in request-sized bites: the work happens inside the request window, which is exactly what a platform that freezes the container after the response needs. The one loss on such a platform is the post-deadline tail of a long drain — a frozen mid-flight job waits out graphile-worker's lock before retrying — so keep the ping cadence at a few minutes and the tail stays short. Ticked every fifteen minutes, a five-minute pass is a fifteen-minute pass; that is a latency question, not a safety one — the observe windows the drop pipeline runs on are measured in days.

On-demand actions ride the same tick: the dashboard's collect button and an approval enqueue, and the next drain — at most 30 seconds away with the interval, or your ping cadence without it — picks them up.

The api holds no database connection when nobody is watching. Its SSE listener needs a dedicated session (LISTEN binds to one, so it cannot come from a pool), and that session used to be opened at boot and held for the life of the process — which after burst mode was the only thing an idle api held, and the one thing that pins a database that suspends when idle. It is lazy now: the session opens on the first subscriber and closes thirty seconds after the last one leaves. Measured on a built image, forty seconds idle, no dashboard open: zero client backends. One while a stream is live, which the integration suite asserts against pg_stat_activity rather than against the service's own opinion of itself. Thirty seconds rather than zero because the SSE route caps a stream at five minutes to re-check ownership, so an open dashboard re-subscribes on a schedule and would otherwise churn the session (D87).

One thing the tick design forced everywhere: the alert cooldown is a Postgres claim rather than a Map in process memory, so restarts and replicas cannot re-mail a cluster's owners on every tick of a bad day (D86).

One origin, guaranteed by the app. The browser calls the api itself, so the session cookie only works if both answer on one origin. That is arranged twice over, and the difference is a hop rather than whether it works — a proxy in front routes /api to the api (zero hops, what the ingress does), and with nothing in front the dashboard server answers /api itself and forwards it (src/lib/api-passthrough.ts, one transparent hop, no configuration). So helm install without an ingress works, a port-forward works, npm run dev works, and compose works; indexterity_web_requests_total{kind="api"} is non-zero exactly when you have not set the proxy rule, which is how a missing ingress rule stops being silent. The cost is that the api is publicly reachable, so its rate limiting, origin checks and auth guard are now the only line of defence rather than the second.

Those limits are worth reading precisely, because two limiters answer for them. RATE_LIMIT_MAX (300/min) is counted in each api process's memory, so it is per replica and a rolling deploy resets it. AUTH_RATE_LIMIT_MAX (20/min) is read twice: the same per-replica way for /api/auth/*, and by better-auth for the credential endpoints, which counts in Postgres — so that half is one budget for the whole deployment. Both count per client address, and TRUST_PROXY is what decides what that address is — for both, since the api resolves it once and hands better-auth the answer rather than letting it read the header itself. Set it to the CIDR ranges of whatever sits in front; on a managed host that does not publish ranges, leave it unset once and read the request log's remoteAddress, which is the address the platform connects from. Behind a CDN, name the header it sets rather than trusting its published ranges, which its own customers can send from: on a host whose proxy appends rather than replaces — Render, behind Cloudflare — set CLIENT_IP_HEADER=cf-connecting-ip beside it, with TRUST_PROXY=127.0.0.1 for the all-in-one image, whose dashboard forwards /api over loopback. true is the one to avoid: it takes the first entry in X-Forwarded-For, which a caller can write, so anywhere the platform appends rather than replaces it lets each attempt choose the bucket it spends. The chart requires it whenever its ingress is on and infers nothing — it derived the value from ingress.enabled once, which chose true with an ingress and false without one, and a proxy routed to the Services from outside the chart is invisible to that test. Architecture §10.5 has the table.

A Helm chart is in deploy/helm/indexterity — api (pipeline included) + dashboard, a pre-upgrade migration hook, ingress, and a helm test. Bring your own PostgreSQL. topology folds the two workloads into one container (single-container) for an install where two Deployments is more than it needs; the Services, the ingress and the app are identical either way, so nothing in front of the chart has to know which one is installed.

Metrics

METRICS_ENABLED=true serves Prometheus metrics on port 9464 (METRICS_PORT) from both workloads. Off unless asked for — an exporter costs memory in every process, and an install with nothing scraping it pays that for nobody. The chart can turn it on and install a ServiceMonitor per workload; compose has it on and publishes 9464 (api) and 9466 (web).

Each answers for what only it can see, so scrape both. The seven things a service that drops other people's indexes has to be able to state:

question	metric
how many clusters can we not reach right now	indexterity_clusters_unreachable
how long is the job queue	indexterity_jobs{state="queued"}, indexterity_jobs_oldest_queued_age_seconds
how many drops are mid-observe	indexterity_recommendations{state="HIDDEN"}
how often does the regression gate fire	indexterity_regression_gate_decisions_total{verdict="regressed"}
what is the dead-letter rate	rate(indexterity_job_runs_total{outcome="dead_letter"}[1h]), backlog in indexterity_jobs{state="dead_letter"}
is run-length storage still saving anything	indexterity_evidence_writes_total{outcome="extended"} against {outcome="inserted"}, by engine and table
does anything we say ever leave the building	indexterity_mail_sends_total{outcome="refused"} and {outcome="disabled"} against the total, by purpose

An unreachable cluster is a handled condition — the tick is skipped and retried, nothing fails — so the queue counters cannot see it and indexterity_cluster_task_runs_total{outcome=...} is where the six ways a tick can end are told apart.

The last row is a saving rather than a failure, and it is here because it stopped working once without anything noticing. index_snapshots and latency_samples write one row per distinct counter state rather than one per collect; a reading that never repeats byte for byte folds nothing, and the tables go back to a row per collect per index with no error and no log line. A ratio of extended to inserted near zero on an engine is that condition.

The mail row is the other one that is here because its failure looks like nothing. A deployment with no SMTP, a transport refusing every send, and a week where no cluster went wrong produce identical silence — no exception, no failed job, no empty panel — and outcome="disabled" is the branch that was invisible: sending is a no-op, the alert cooldown correctly treats it as settled because no retry could do better, and the mail is simply gone. Set metrics.prometheusRule.mailRequired on an install that means to send.

What none of it can see is a message the transport ACCEPTED and the recipient's server then rejected. Those count as sent, because acceptance is the strongest claim the sending process can make — the rejection happens somewhere this deployment never hears about. The only thing that reports them is DMARC, so if you send from a subdomain of a strict-DMARC parent, publish a _dmarc record with a rua= address at the sending domain before the first send. An inherited sp=reject rejects silently: no bounce, no log line, and every alert, invite and password reset gone with nothing anywhere to say so.

The endpoint has no auth, which is why it is a second port instead of a route on the app. metrics.prometheusRule.enabled=true installs 20 alerts with the metrics, because a scrape endpoint nobody wrote queries against is a scrape endpoint nobody reads. deploy/helm/indexterity/README.md has the thresholds and the label-selector gotcha; what the dashboard server reports that the api cannot, and the two alert traps worth knowing, are in Architecture §15.

Errors

The metrics above say how often something failed. SENTRY_DSN says what. Every process reads that one variable; leave it empty — the default here, in compose and in the chart — and neither app initialises the SDK. The dashboard does not even load it: importing @sentry/tanstackstart-react costs 17.3 MB of heap, measured, and a dynamic import() there is real ESM, so a default install skips it entirely. The api loads @sentry/nestjs (12.5 MB) either way — a deliberate tradeoff, not an oversight, kept for full static typing over that saving. See D75 for why the two apps land differently.

The DSN is yours. A self-hosted install reports to your own Sentry organisation or your own self-hosted Sentry, never to us.

Two projects. The api — pipeline included, one process since #232 — and the dashboard, which is a separate app with a separate release. In compose that is SENTRY_DSN_API and SENTRY_DSN_WEB; in the chart, errorReporting.dsn and errorReporting.webDsn.

workload	what is reported
api	unhandled 500s from AppExceptionFilter, tagged with the request id already in the log line and the response body — and a job that burned its last retry (the dead-letter transition), tagged with task, attempt and cluster
web	server-side render failures, and anything the passthrough throws

Plus the unhandled-rejection and uncaught-exception sinks the SDK installs in every process, which is the half that had no home before.

Reported once each, and only for things that are actually faults. A retry is not an event (the counters above already say how often jobs retry), and neither is a 429, a 413, a 404, or a cluster whose owner's firewall is doing its job — an unreachable cluster is a handled condition, so paging about it would be paging about somebody else's network.

Nothing leaves with a connection string in it. packages/errors strips MongoDB and Postgres URIs whole — hosts included, because a replica set's member list is the customer's infrastructure and +srv resolves to it — plus the credentials of any other URI, any field named like a secret (a superset of the Fastify logger's redact list), the same secrets encoded inside a JSON string, where key matching cannot see them, and the values of a failed query — the params: line better-auth logs on a database error, which reaches Sentry as a breadcrumb. It walks the entire event rather than a list of known fields, because the way a field list fails is silently, on the day the SDK adds a field. Unit-tested against a seeded event carrying the string in every place one could hide.

The request body is dropped entirely, not scrubbed: POST /clusters carries a customer's connection string as a field. sendDefaultPii: false does not cover this — in SDK 10.69 bodies, stack locals and database payloads are governed by dataCollection, and its defaults collect all three. All are off, and the body is removed in beforeSend as well, because setting httpBodies: [] did not stop it arriving. That was measured against the running stack, not read off the docs.

Tracing, profiling, session replay and Sentry's own metrics are all off: the