How Indexterity is put together and why. This is the source of truth for the shape of the system; it describes what is there now, not history.

The decision log that used to close this page lives in the repository instead — docs/decisions.md. It is versioned with the code on purpose, so a decision is reviewed in the same pull request as the change that makes it.

Update this page in the same change as the code, not afterwards.

A SaaS that continuously monitors MongoDB indexes and safely manages them: create missing indexes, drop unused ones, remove redundant ones, and merge overlapping ones — while proving the improvement in hard numbers.

Problem. Teams add indexes over time and almost never remove them. Every index slows writes (each write updates every index), consumes RAM (working set) and disk, and nobody audits $indexStats. The cost is a slow burn, so it is rarely acted on.

Wedge / differentiation.

• MongoDB Atlas Index Autopilot only auto-creates indexes. It recommends removals but never auto-drops them (dropping is dangerous), and it is Atlas-only.
• Our edge is the part Atlas leaves manual: safe auto-cleanup (drop unused / redundant, merge), self-hosted + multi-cloud coverage, and a cross-provider ROI dashboard.
• Second edge: an index-only permission model means we never read customer data rows — a strong trust story Atlas does not need to make.

Do not compete on auto-create on Atlas. Compete on safe cleanup everywhere.

1. Safety is asymmetric. Keeping a useless index is mild waste; dropping a needed one is an outage. Every ambiguous call biases toward keep.
2. Read-only by default. Demo/read-only is structural, not a UI toggle. The data plane refuses all writes to a cluster unless that cluster is explicitly taken out of demo mode. Structural also means said out loud: because the mode is enforced at the write, everything upstream of the write has to refuse in its own words rather than let a reader queue something that will never happen. approveRecommendation refuses with the mode named, and the dashboard's Approve cell says read-only cluster instead of drawing a button — an approval on a read-only cluster used to be accepted and then sit at APPROVED forever, with no action row and nothing anywhere explaining it (#257).
3. Deletion is a confirmed, reversible pipeline — never a single command.
4. No customer data exposure in the cleanup path. Index metadata and usage stats only; no find.
5. ROI must be visible. Every action reports before/after. No visible ROI, no renewal.
6. Fully typed. No any, no as overrides, no linter-ignore comments. Latest stable versions of every dependency. Comments stay short.

Postgres stores our state. The databases we manage are the customers' MongoDB clusters, reached via the data plane. Clean separation.

Versions are pinned to latest stable at scaffold time — this doc does not hard-code version numbers so it does not go stale.

The spine of the system. The control plane (our SaaS) never touches customer MongoDB directly in agent mode; it decides and displays. The data plane is the only component that touches MongoDB, using an index-only role.

 CUSTOMER INFRA (private)            OUR SaaS (control plane)          BROWSER
┌───────────────────────┐          ┌──────────────────────────┐     ┌──────────┐
│  MongoDB replica set  │          │  apps/api (NestJS+Fastify)│     │ apps/web │
│  ┌─────┐ ┌─────────┐  │          │   - accounts / tenancy    │◄───►│ TanStack │
│  │prim.│ │secondary│  │          │   - recommendations       │oRPC │  Start   │
│  └──┬──┘ └────┬────┘  │          │   - apply orchestrator    │     │ +shadcn  │
│     │        │        │          │   - audit + ROI           │     └──────────┘
│  ┌──▼────────▼──────┐ │  HTTPS   │   - job queue (graphile)  │
│  │  DATA PLANE      │ │◄────────►│                           │
│  │  collector +     │ │ outbound │  PostgreSQL (Drizzle)     │
│  │  executor        │ │  only    │  secrets (envelope enc)   │
│  │  (index-only role)│ │(agent mode)                         │
│  └──────────────────┘ │          └──────────────────────────┘
└───────────────────────┘

• Hosted-direct — v1. Customer pastes a MongoDB connection string (SRV URI, index-only role) into the dashboard. Our servers connect out over TLS. Fastest onboarding; self-serve. Requires their cluster be reachable from us (public endpoint or IP allowlist) and that they trust us to hold creds (encrypted).
• Agent — phase 2. Customer runs a small container inside their network. It talks to MongoDB locally and phones home outbound-only. MongoDB is never exposed; creds never leave customer infra. Enterprise trust tier.

The data plane is defined by one interface; hosted-direct and agent are two implementations. Building hosted-direct first does not preclude the agent — it is an additive second implementation, not a rewrite.

Reachability is a third axis, not a third mode. Hosted-direct requires the cluster be reachable from our egress, and a VPC-only or on-prem database is not — at any price, which makes it unonboardable rather than merely awkward. A tunnel changes the route without changing the mode: the adapters still open a socket to a real server and still speak the same interface.

An owner pastes the wg0.conf their network already uses and the control plane terminates that peering in userspace, needing no capability and no agent on the customer's side: a small Go service beside the api, wireguard-go for the protocol and gvisor's netstack for IP and TCP (apps/tunnel/, D113 amending D111). Still no NET_ADMIN and no /dev/net/tun.

It is a service in the api's own network namespace rather than a process the api spawns — one container in the all-in-one image, a sidecar in the api's pod, network_mode: service:api in compose — and the api finds it at 127.0.0.1:TUNNEL_PORT, one variable both sides read. Loopback is the design and not a default: a customer's private key never crosses a network, the SOCKS5 proxy into their network is reachable from nowhere else, and there is consequently no shared secret to configure, because there is no listener a stranger could reach. An earlier revision addressed it by URL with a token and TLS; it worked and was strictly worse on all three counts, which is why the port replaced it.

What that gives up is a process boundary: one process holds many devices, so two tenants on overlapping RFC1918 ranges stay separate by stack — each peering gets its own gvisor netstack — while every private key shares one address space and one crash drops every peering. Measured before choosing: 6.4 ms from spawn to a bound listener, ~16 MiB resident per idle peering. The process was never the expensive part; the handshake is, and that is an RTT to the customer's gateway either way.

One control connection per peering, and the connection is the peering's lifetime: it closes, the device goes, and the next dial reconnects. TCP keepalive is load-bearing rather than hygiene — an api killed without a FIN used to close stdin and end the peering in the same instant.

A signal drains the service; it does not stop it (#649, D113). Every deployment signals the api and the tunnel service together: the all-in-one supervisor forwards SIGTERM to every child, and Kubernetes signals every container of a terminating pod. The api finishes its in-flight passes and closes its peerings last. So the service, on SIGTERM:

• closes its control listener, and takes no new peering;
• keeps the open peerings and the SOCKS5 listener serving until the api closes them, then exits 0;
• gives up after five minutes, longer than any default grace period, and ends at once on a second signal.

It used to exit at once. Deploying 0.31.0 caught a pass mid-flight: the old instance's collect of a tunnelled SQL Server dialled a SOCKS5 port that had gone, failed with ECONNREFUSED, and mailed the owner that the cluster was unreachable. The new instance's probe went through the same tunnel nine minutes later.

A deployment with no tunnel service configured is a supported state: the api reports the feature as unavailable and the dashboard says so, rather than offering a form whose every submission fails at the last step.

D110 shipped that runtime in TypeScript over lwIP-in-WASM first, and the reasoning in it still holds for everything except who writes the protocol: the crypto was never ours, and now the handshake and the TCP stack are not either. That implementation is deleted rather than kept behind a flag — a dormant second one keeps the maintenance burden the replacement existed to remove.

The binary deliberately holds no policy. Every CONNECT it is asked to make — including one whose host is already an address — is referred back to the api, which resolves through that tunnel and applies the same assertDialableThroughTunnel the direct path applies. Two pipe round trips per connection, none per packet, and one implementation of the rule rather than a second copy of a security table in a second language. Silence from the api is a refusal, never an allowance. clusters.tunnel_id says which peering reaches a cluster; null is the ordinary case. Each tunnel owns its own netstack, which is what makes two tenants on overlapping RFC1918 ranges a non-question rather than a routing conflict — and why a shared sidecar, which has one routing table per pod, cannot serve the hosted case however many capabilities it is granted. Self-hosted installs that would rather route at the node still can (chart README, D109).

Worth being precise about what a tunnel does not buy, because it is easy to oversell: the connection string is still stored here and still opened here. Only the agent's remote-adapter phase makes "credentials never leave the customer's network" true, and the two claims must not blur.

apps/
  api/          NestJS + Fastify — control plane, job pipeline, engine adapters
  web/          TanStack Start + shadcn — dashboard
packages/
  contracts/    oRPC contracts + zod 4 schemas (shared types api ↔ web)
  config/       shared tsconfig + Biome config
  metrics/      OpenTelemetry provider + scrape endpoint (§15.2); instruments stay in the apps
  errors/       what is stripped from an event before it leaves (§15.4); pure, no SDK dependency

Only what is SHARED is a package. The analysis engine, the Drizzle schema, the auth config and the engine adapters are all apps/api/src/* because exactly one app uses each, and a package boundary around a single consumer buys a build step and nothing else.

Inside apps/api/src, the two halves are layered differently on purpose:

• analysis/, engine/, mongo/, postgres/, mssql/ are framework-independent ports and adapters (§9). No decorators, no container — reachable and testable without booting Nest, which is worth keeping.
• everything else is NestJS feature modules (#333): module → controller → service, and a repository only where a feature's queries earn one. The service layer is where the two halves meet. DatabaseModule and TenancyModule are the shared wiring — imported rather than re-provided, so the pool and the tenancy rules are one instance each.

app.module.ts is an imports list and nothing else — no controller and no provider of its own (#354). That is the checkable version of the sentence above: a controller listed at the root would be a feature that has not been given its module yet.

A provider exists where there is a dependency to inject, and nowhere else. So errors has one for the dial budget (the count lives in postgres), mail one for reading a cluster's owners, billing one for counting seats, metrics one for the gauges that read the api's own pool, and auth one for the two refusals that need the database. jobs holds the six per-cluster passes, so a pass can be handed a service rather than importing one — the queue itself stays graphile-worker, which is a durable postgres queue with cross-replica locking that neither Nest package replaces.

The other direction is the same rule read backwards, and it is why the list in app.module.ts is shorter than the directory listing. analysis/ is pure functions; dependency injection pays for substitutability, lifecycle and interception, and a function with nothing to substitute gets none of the three. config/ is read before a container exists, because validating the environment is what lets one start. orpc/'s Implement is a decorator applied when a class is defined. The three adapters build a connection per cluster and close it with the lease that opened it — a per-lease lifetime, not a container one. Those directories have no module at all, and that is deliberate: an empty one provides nothing and says nothing, so the reasoning lives beside the import list it is absent from.

analysis/ holds all dangerous logic as pure functions: (indexes + usage snapshots) → classified recommendations. No database, no network. Pure means deterministic, testable, and trustworthy. It gets the deepest test coverage in the repo.

Four rules, each settled against a real case rather than adopted from a framework guide. They are worth stating because the obvious reading of "controller → service → repository" gets two of them backwards.

A repository is earned by a rule written twice, not by a query count. A per-table CRUD wrapper is a loss: Drizzle is already thin and fully typed, and the wrapper only costs inference at the seam. What justifies one is a rule that had started repeating itself — insights clamps a caller's window to the PLAN's window, and that clamp had been spelled out in two places, so it is now one method with the entitlement stated once; recommendations asks "this row, but only if the caller's org owns its cluster" in four different column selections, and that is the tenancy boundary for every act in the feature. The two features without such a rule (policy, org) have a service and no repository, and org has sixteen queries.

Repositories also absorb the queries whose comments argue about storage — run-length encoding, microsecond timestamps in max(last_seen_at) — because that argument is not what the view above them is about.

Ownership goes down or stays up depending on how it refuses. A route that answers for a cluster the caller does not own with an EMPTY view is scoping, and belongs in the service with the answer, because each view's empty shape is different (ownsCluster). A route that answers with a contract error keeps it in the controller, because that needs the handler's errors map (assertOwnsCluster).

Contract error maps travel down; they are not rewritten. NOT_FOUND, CONFLICT and BAD_REQUEST are declared on the contracts and the dashboard branches on the code, so a service that refuses takes the handler's own errors map as a parameter — the shape TenancyService.assertOwnsCluster already used. Rewriting them as bare ORPCErrors would work on the wire and lose the reason.

Anything read off the request stays in the controller, and is handed down as a value or a thunk. The security-trail actor (address, user agent) goes down as a thunk, because the trail is only written when a policy actually moved and a form saved twice must not pay for a session read.

No DTO classes. @repo/contracts already is the DTO — zod, validated at the boundary, shared with the web app. A second class-validator schema beside it would be two descriptions of one shape, and the second one drifts.

DatabaseService opens its pool in a field initialiser and drains it in onApplicationShutdown. A feature module that lists it in its own providers therefore gets a second pool nobody counted. db/database.module.ts and http/tenancy.module.ts export one instance each and every feature imports them; app.module.ts imports them too rather than providing them.

Neither is @Global(). Naming the dependency costs one import line per feature and makes the two-pools mistake unavailable.

Nest wiring is invisible to tsc and to the unit suite. A duplicated provider, a missing exports, a controller in no module — all of them typecheck and all 1096 unit tests still pass. The check that catches them is booting the built api and reading its route table: RoutesResolver / RouterExplorer name which controller each route came from, and a wiring fault is an ERROR line before the first request.

$indexStats.accesses.ops is cumulative since that mongod last restarted, and it is per-member. Consequences the engine must handle:

• Snapshot over time. Periodically snapshot $indexStats into Postgres and compute rolling-window usage from the deltas. A single reading is meaningless. A stored row is an interval rather than an instant (§11.2): the counter held this value from capturedAt to lastSeenAt, so the deltas are taken across the gaps between rows, never inside one.

• Detect restarts. Implemented (counterEpochs): every snapshot persists each member's accesses.since — the moment that member's counter started — so classification can see a reset rather than infer one. A reset is noticed when since advances for any member, or when a member's cumulative ops go backwards (which is the only way to catch a SQL Server index rebuild, §7.9). Snapshots written before the field existed carry none and are skipped, so the check never invents evidence.

  A reset ends one counter epoch and opens the next rather than voiding the history. trustedWatchMs sums the epoch spans and the warm-up is judged against that sum — the same number as the plain first-to-last span on a cluster that never restarted. The blind window between epochs runs from our last reading to the restart instant, and since #631 it is the thing gap-between-runs checks. Before, that check measured the whole hole between two runs.

  • A hole the counters ran through hides nothing. They are cumulative, so any use inside it shows at the first reading after it.
  • A counter that started inside the hole leaves it blind from our last reading to that start.
  • A reset with no later start to date it, or no start at all, leaves the whole hole blind.

  The hole is dated by when the counters started, not by the stored restart flag. On production's MongoDB cluster the flag missed every restart across the 22-day outage: its members restart several times a day and do not all answer every collect, so a member is not always there to match. The counters' start still moved, from 09-09 to 10-01.

  A blind window longer than maxGapHours refuses for 30 days after it ends (BLIND_RECOVERY_DAYS). That is long enough for a monthly cadence to come round in the history after it, and the whole window is still read for its cadence.

  Before #631 a blind window refused for as long as it stayed in the plan's window: 90, 183 or 365 days by plan. For PRO that meant production's outage refused 188 of 251 MongoDB indexes until March 2027. It also refused SQL Server's, whose counter had run through the outage untouched.

  The classify pass ships only windows longer than the cutoff, so a cluster that restarts several times a day is not sent its restarts to say nothing. An epoch also begins at the later of when we started reading it and when its counters started, which is the old counters-younger-than-span rule kept as a clamp — a boundary is only visible between two snapshots, so a cluster whose first collect landed after a restart would otherwise look unbroken.

  It used to refuse the whole history instead, and D96 measured whether that was too strict and answered no. That answer was about a different relaxation — replaying the gate against only the stretch since the last reset, which found post-reset tails of 0.16 days against the 7-day warm-up of the time — which is now the auto-apply floor rather than the gate on saying anything (D130). Truncation buys nothing, and the tail arithmetic proving it is exactly the reason not to truncate: the epochs before the reset are just as accountable to a differenced series. Measured on a cluster restarting nightly, the old rule discarded 74.9 hours of observation to avoid 1.6 hours of blindness — and being nightly, would have gone on discarding it forever. D96's other objection is also about truncation: a truncated history cannot see a cadence, so PERIODIC_DEAD (35) would read as FLAT_ZERO (50), more confident on less. Nothing is truncated here — classifyUsage still reads usageSeries over the whole history, and only the trust gate counts epochs.

  What D96 left open, and #277 closed, is that a refusal is invisible to the person it affects. A cluster short of trustworthy history gets an empty recommendations list, indistinguishable from "your indexes are all fine". Each pass now writes an analysis_notes row for the cluster with how many indexes it considered, how many it trusted, and the count per refusal, and the dashboard names the dominant reason above the proposals. PAUSED means nothing cleared the gate: one trusted index means the machinery works and the others are individually short of history, which is the ordinary state of a cluster with new and old indexes. The sentence is composed on read from the same CLASSIFY_OPTIONS the gate used, so it quotes the live thresholds rather than the ones in force when the pass ran, and every one of them ends by saying redundancy findings are unaffected — they are structural and never touched the usage history.

• Aggregate across all replica-set members. Secondaries serve reads with different index usage than the primary. Reading only the primary and dropping an index a secondary relies on is a classic outage. Sum usage across primary + all secondaries before concluding "unused".

• "All members" is two arrays in hello, not one. Electable members are listed under hosts; a priority: 0 member is listed under passives. The discovery in mongo/connection.ts read hosts alone for as long as the per-member fan-out has existed, so a priority-0 member was never dialled — and priority 0 is the standard configuration for a secondary in another region, set precisely so it cannot win an election across a WAN. Priority governs elections and nothing else, so such a member serves secondaryPreferred reads like any other secondary while its counters stayed invisible: the exact outage the bullet above describes, reintroduced one array key later. Measured on a 5-member 8.0 set (primary, 2 × priority 1, 1 × priority 0, 1 hidden) — collectUsage reported from 3 of 5 before, 4 of 5 after.

• Hidden members are knowingly out of reach. They are in neither array, so finding them needs replSetGetStatus or replSetGetConfig, cluster privileges ENGINE_PRIVILEGES does not ask for and every customer would have to re-grant. Drivers never route reads to a hidden member either, so what is missed is direct-connected traffic — BI tools, backups, delayed members. A privilege decision, deliberately not folded into the fix above.

• A member we cannot dial is recorded, and the dashboard names it (#100, D46). MemberConnections keeps every dial's outcome — answered / unreachable / refused (the net guard declining an address the cluster named, a policy fact rather than member health) — and the collect writes the roster to cluster_rosters: one row per cluster, jsonb members + collectedAt, replaced whole each collect. Deliberately not a history table; "which nodes, and did we reach them" is a question about now, and a per-collect log of it would grow with the cadence for a past nobody asks about. Each answering node's role comes from its own hello (no privilege — it is the pre-auth handshake, and the primary's hello names only which host is primary while a node admits to being a secondary itself), so the overview's Nodes panel can say "4 of 5 members answered" instead of summing the gap away. A collect that cannot establish a roster at all leaves the previous one standing with its own timestamp — "as of six hours ago" is honest, an empty panel is not. Down is still normal and still must not fail a collect; it is just no longer silent.

• The per-node split of USAGE needed no storage decision at all (#161, D66). D46 deferred it together with per-node latency, on the grounds that both wanted a decision about the table D39 names as the growth ceiling. That is true of latency, which has no per-member column and would want a row per member per collect. It is not true of usage: index_snapshots.per_member is written on every collect and is the run's own identity — two collects belong to the same row exactly when it is byte-identical — so the split is already stored, already run-length encoded and already free. What was missing was a reader. Every path summed it before the screen (jobs/apply.ts, analysis/classify.ts's totalOps), so the whole point of collecting per member went on making the total honest.

  listRecommendations now carries a usage array beside the rows rather than a field on them, because recommendation is also what approve, undo and un-hide return — those answer about a row, so a usage field would come back null from every mutation and read as "this index has no recorded usage". The reading is the last collect's batch, keyed by last_seen_at for the same reason getCollections is. The roster is not duplicated into the payload: the overview already reads it for the Nodes panel, and the members are one fact about the collect rather than 500 copies of one.

  The rule that makes it safe to ship: a member the collect did not reach is a named blind spot, never a zero, and never folded into the "N of M nodes" ratio. A per-node number that silently omits an unreachable secondary is worse than the total it replaced. The ratio counts members that recorded operations against members that answered; the absences get their own clause ("· 1 not reported") and are named in the tooltip. Per-node latency stays deferred with D46's reasoning intact.

• And it hid something the size of "no SRV cluster ever worked". A member connection is built by rewriting the cluster's string as plain mongodb://, because an SRV seed cannot be pointed at one host — its targets live in DNS and the scheme forbids a port. But mongodb+srv:// keeps two settings outside its own text: the scheme defaults tls to true, and authSource is published in a DNS TXT record the driver reads at connect (VALID_TXT_RECORDS in the driver's connection_string.js). The rewrite kept neither, so every member connection to an Atlas cluster was plaintext against a TLS-only listener, and — once that was fixed — authenticated against the database in the path. Measured against a requireTLS + --auth mongod: as-built refused at the handshake, +tls gave "Authentication failed", +tls +authSource connected. Two breakages in series, each hidden by the same catch, which is why the failure presented as "this cluster appears to be a standalone" for as long as SRV has been supported. Both values are now carried from the LIVE client (MongoConnection.resolved()), not re-derived from the original string — the driver has already done the DNS work by the time members are opened, and repeating it would be a second source of truth. A plain multi-host string was never affected: its options are in its text and the rewrite copies them.

• $collStats latencyStats is per-member too, and it took a second bug report to act on it. Both are aggregations, so the driver routes them by read preference: a customer string carrying readPreference=secondaryPreferred or nearest — the standard shape for a reporting user, and the shape the per-member fan-out above exists for — sent every latency reading to a secondary. A secondary's writes.ops is zero and stays zero however busy the primary is, because replication applies the oplog and the applier is not a client write op. Measured on a 3-member 8.0 set after 2,500 writes: primary writes.ops 2500, both secondaries 0. Reads meanwhile looked completely normal, because a secondary really does serve them — so the write chart alone went blank, which is a far quieter failure than both going blank would have been. The fan-out reads every member and sums by host; the base connection asks primaryPreferred, so a mongos routes to the shard primaries and an election costs a stale write counter rather than the whole collect's reads.

Classify each index from its usage history, not a single number:

Every signal below is activity — how much happened between two readings — and never the value of the counter. $indexStats.accesses.ops is cumulative, so a non-zero reading only says the index was used at some point since that member's counter started, which is true of an index nothing has touched in a year. The classifier differences consecutive readings and treats a counter that restarted under it as a boundary rather than as a drop to zero. The trust gate below is the one place that still reads the counters directly, because since moving and a reading going backwards are facts about the counter itself.

To label a pattern dead (rather than "hasn't fired this cycle yet"), the engine must first learn the period (needs ≥2–3 observed periods of history), then observe N expected occurrences that did not happen. Example: an index that fires around the 1st of each month — if two consecutive month-starts pass with zero ops, the workload is decommissioned → propose drop.

This is both safer (a flat window would wrongly flag a live monthly index during a quiet stretch) and faster (a genuinely dead pattern can be dropped without waiting the full conservative window). Before enough history exists to detect a cycle, fall back to FLAT_ZERO rules.

minActiveHours: 72, recentHours: 12, minHistoryDays: 3, maxGapHours: 48 — all durations, and the first two were counts until Aug 2026. minActiveIntervals: 12 was three days of traffic and recentWindow: 3 was the last twelve hours, but only because a collect interval happened to be six. Change the schedule from 0 */6 * * * to */15 and the same numbers buy an eighth of the evidence: the engine starts calling indexes dead on three hours, with no code change, no config change, and nothing failing a test. Which is why this landed before any cadence change and not alongside one.

recentHours is the one worth staring at. It is the line between PERIODIC_ALIVE and the droppable PERIODIC_DEAD, so as a snapshot count it meant that at a fifteen-minute cadence a nightly job which had simply not run yet today read as decommissioned.

minHistory stays a count on purpose: it is how many data points separate a pattern from a flat line, which is a question about samples and not about time. minHistoryDays beside it is the durational half of the same gate.

minHistoryDays was 7 until Sep 2026, and it was one number doing two jobs: it decided both whether a usage finding could be shown and whether the engine could act on one. Seven is the smallest span covering a weekly cadence, which is the right floor for an unattended deletion — a shorter one calls the weekly batch job's index dead because it happened not to run yet. Applied to disclosure it meant a newly connected cluster spent its first week showing an empty panel and a sentence about why the panel was empty.

Three, and not lower, because minActiveHours: 72 is the real floor. It is the check immediately behind this one and refuses until the collection has actually served reads for 72 hours, so a shorter warm-up would buy nothing but a different sentence — span-too-short becomes collection-idle. The number is chosen to meet the gate that would answer instead of it.

And the score had to move with it, or this would have been a change to what gets deleted. The history term in dropScore is a credit of 25 points spread over a month, so three days of watching earned 2 and a week earned 5 — and policies.auto_apply_score is compared against that same number. A FLAT_ZERO on a gigabyte index scored 50 + 2 + 20 = 72 on three days, above the 70 the engine recommends as a threshold. The first week is now a discount rather than absent credit (thinEvidencePenalty): 25 points decaying to zero at a week of collects, so the same index scores 58, still surfaces, and cannot auto-apply. Zero exactly at a week means nothing past the first week moves and the 95-point ceiling on a drop is untouched. Usage findings only — a redundancy verdict is provable from the index list and rests on no span, so charging it for a short history would be charging it for evidence it never claimed.

evidence_days is nullable, and NULL reads as eligible. That is the finding saying it does not rest on a span — a redundancy verdict, a create — or a row written before the column existed. Classify deletes and re-inserts its own PROPOSED rows every pass, so failing NULL closed would freeze proposals carrying months of evidence for one cadence, to protect against rows that are about to be rewritten anyway.

A hand-approved drop is untouched by all of this: it never reaches promoteByScore. Somebody clicking Approve on three days of evidence is a person deciding, which is exactly the distinction the two spans draw.

activeHours credits each interval at most the series' median gap, measured from the data — a pure engine cannot know what the scheduler is set to, and a cluster's history can span a cadence change anyway. Without that cap a single outage would credit its whole length as traffic the moment the counter had moved anywhere inside it, so one dark afternoon could manufacture the three days a drop needs. What a long interval tells you is that the collection was used somewhere in it, not throughout.

collect runs hourly, and a newly connected cluster is collected immediately — queued from storeCluster, best-effort, so a failed enqueue cannot turn a connect that already committed into an error the reader can do nothing about. The cold start was always a separate problem from the cadence and it has a separate fix; one job on connect settles it without touching steady-state load.

Except for a cluster whose collect does not fit an hour, which is paced (#571, D179). A collect gets five minutes, and one that needed longer used to be abandoned every hour, forever, collecting nothing. Now its budget and its interval stretch by the same factor — ten minutes every two hours, then twenty every four — so the share of the single worker slot it may take never moves; only the twenty minutes can now finish. cluster_pass_timings.tier holds the step, and each collect decides the next one from how it went: a timeout, or a collect that used three quarters of its budget, steps up; one that fits in half the budget below steps down; being unreachable says nothing about how long a collect takes and changes nothing. The dispatcher counts the schedule's hours rather than elapsed time, so a collect that started late is not pushed a further hour. The cap is the analysis's, not the scheduler's: past four hours the change window has six-hour slots that no reading's gap starts in, and the queue alert would fire on a forty-minute collect holding the slot. Two readers changed with it. Active time credits no interval more than an hour, so a paced cluster reaches an "unused" verdict later and never on less evidence — one read anywhere in four hours used to earn four active hours. And the change window spreads a gap's traffic over the slots it covers instead of crediting it all to the one it starts in. A paced collect's lock is released at three of its longest budget, an hour, rather than three base budgets. The Passes panel says when a cluster is paced and why.

suggest is paced the same way, on a tier of its own (#588, D179). The two are slow for different reasons — a SQL Server cluster that collects in under a minute can need half an hour to analyse — so one tier for both would slow the half that fits. A suggest is enqueued three ways (the hourly dispatch, the end of every collect, a probe that finds read pressure), so the pace is checked in the handler itself: one that is not due stands down and records nothing. Its tier is decided by the analysis alone, the part its budget covers, so a long instant build behind it shows as an instantBuild phase and does not pace the analysis.

A paced timeout mails only once pacing has run out (#637). Below the top tier, a timed-out collect or suggest is already answered: the timing it records steps it up, so the next run gets twice the budget at half the cadence. Its block says so (… — paced: the next run gets 10 minutes, every 2 hours), and no mail goes out. It used to mail at the first timeout, about a problem pacing was already solving.

At the top tier, four budgets every four hours, the pass does not fit even at its slowest pace. That is the case an owner has to act on, and the mail says so: <task> does not fit even at its slowest pace. A cluster that genuinely cannot finish therefore mails after about six hours of trying rather than at once. Unpaced budgeted passes (probe, classify) mail as before.

Storage does not stop scaling with the cadence — only half of it does. An earlier version of this page claimed otherwise, and measurement on real data says no:

latency_samples has max(observations) = 1 — not one run has ever been extended, across all 101 namespaces with more than one look. $collStats totals move on any operation, so a live collection differs at every look and there is nothing to run-length. That table therefore grows in step with the cadence, at a measured 717 bytes per row including indexes, while index history grows at roughly a quarter of the rate. It is the table that sets the ceiling, and it is the one behind getLatencySeries — whose read is bounded now (a 30-day window and a top-N, D47), though the table it reads still grows with the cadence, which is what any move to a sub-hourly cadence has to answer for first.

Load is not the constraint. A collect takes 0.66–0.76s against ~100 collections. collectSnapshots walks databases and collections serially with five commands in flight per collection, so against a remote cluster the cost is about one round trip per collection — roughly four minutes at 5,000 collections and 50ms RTT. Hourly leaves ample headroom.

So the cadence is bounded from below by what a reader can see — four points a day is not a trend — and from above by that one table. It stays hourly on those merits, not on a blocker (#178). Half of the blocker this page used to name is gone: getLatencySeries is bounded. But what D47 bounded was the response, not the query — loadLatencyReadings selects every collection's rows inside the window and takes the top 8 in JS afterwards, so a dashboard load scans collections × cadence × up to 30 days either way. Halving the interval doubles that scan, and doubles the bytes held inside the retention window at 717 a row, to buy 48 points a day instead of 24 — against a floor of four, which hourly already clears six times over, for a trend nobody reads that closely. The signal that has to be fast is the five-minute read-pressure probe's, not this one's. What would change the answer is downsampling old latency history and slicing that read in SQL.

A task cannot overlap itself. Each dispatch carries queueName: <task>:<clusterId> as well as its job key. The key alone was not enough: graphile-worker's add_job clears the key of an already-locked job and inserts a new one — its own migration says "in the case of locked existing job create a new job instead as it must have already started executing" — so dedup covers a job still waiting and does nothing about one still running. Two collects racing on one cluster would double-count observations, the number the trust gate reads as how many times we looked, and could write two runs whose spans overlap, which the exclusion constraint then rejects mid-collect. The queue is keyed per task as well as per cluster so the five-minute probe never waits behind a collect walking ten thousand collections.

A running pass is not re-keyed (#454, D138). The clearing quoted above has a second half the migration does not mention: add_jobs also sets the locked job's attempts to its max_attempts, so a pass that was still running when its key came round again had spent every retry it had — if it then failed it was not retried, and if the worker died under it nothing failed it, so the row sat dead with no error text and no owner alert. Thirty-eight such rows accumulated on the production deploy in a week, thirty-seven of them silent, most on the cluster whose passes ran longest. The dispatcher now reads which <task>:<clusterId> jobs are locked (one query per dispatch, runningPasses in jobs/dispatch.ts) and lets a running pass stand for the tick instead of adding behind it. The collect-chained classify and suggest are enqueued the same way — keyed, queued per cluster, five attempts — so a chased suggest dedupes against the hourly one rather than running beside it with the library's twenty-five retries.

How often classify is chased is its own number (#482, #483, D147). It used to be chased by every landed collect, so at an hourly collect it re-derived an index-usage verdict once an hour per cluster — reading the whole retained window each time. That verdict is a claim about weeks: minHistoryDays is three and minActiveHours seventy-two, and the trust gate wants a long series precisely so "unused" means something. One more reading does not move it. Measured on the hosted deployment, which exhausted a 5 GB monthly network-transfer allowance against an 82 MB database: a full round across the fleet is ~13 MB, hourly, ~310 MB a day against an allowance of ~165 MB. D145 cut what one pass ships from 36 MB to 11 MB and was not enough — it cut bytes per row, not how often the pass runs.

There are two thresholds, because a single interval cannot answer both questions:

The first is a product number said out loud: recommendations refresh four times a day rather than twenty-four. The five-minute read-pressure probe is untouched and is what makes that affordable — a missing index shows up as latency within minutes and chases suggest directly, so the signal that has to be fast still is. What slowed down is the re-derivation of a verdict about weeks.

The second case looks skippable and is not, which is the part worth getting right. An idle cluster reports byte-identical counters, so its rows are extended rather than inserted (jobs/runs.ts) and there is genuinely no new evidence for a verdict to turn on. But an extension moves last_seen_at and increments observations, and that is exactly how an index idle since we first saw it eventually crosses those two floors and becomes proposable. A cluster left unclassified until its counters moved would never make that crossing, because nothing would have moved — the drop would be withheld forever with no symptom. So "nothing was learned" buys a longer interval, never an indefinite one, and chaseNotBefore clamps the ceiling to at least the floor so a misconfiguration cannot invert the rule.

The claim lives in worker_watermarks under its own chase:classify:<clusterId> namespace — a compare-and-set, the same primitive the tick's own schedule uses (jobs/classify-cadence.ts). The reading is against the last classify, not the last collect, and that is what stops evidence which landed inside the floor being stranded: it is still unclassified, so the idle ceiling picks it up rather than it waiting for a change that may never come. A cluster with no stamp at all is due, so a freshly connected cluster's first collect classifies immediately.

Saying any of this required collectCluster to stop returning snapshots.length. That number is how many snapshots the collect saw — every index on the cluster, every time — so a completely idle collect returned the same figure as one where every counter moved. It returns inserted-versus-extended counts now, summed across both series, because a moved latency counter is evidence even when no index counter moved: the activity gate reads it, and "the collection served reads and this index served none of them" is exactly the claim it licenses.

suggest is deliberately not rate-limited with it, and a skipped chase is logged rather than silent — "no recommendations changed today" and "the pipeline stopped" must not read alike.

The change-window floor moved to hours with it. inferChangeWindow required three intervals of evidence per bucket, which meant three days only while a collect happened every 6h and each six-hour bucket therefore saw one interval a day. At an hourly cadence the same 3 would have been satisfied in twelve hours — the engine choosing a change window off half a day of traffic and applying elective index changes at what it wrongly believed was the quiet hour. It now requires 3 × BUCKET_HOURS watched hours, read off slot.hours, which counts wall-clock time rather than samples and so means the same thing at any cadence. At 6h it is arithmetically the previous threshold, so nothing about the old behaviour changed. This is the same class of bug #66 fixed in classify.ts, and it survived that pass.

BUCKET_HOURS stays at 6 even though the evidence would now support finer slots. How wide a change window should be is a question about how long a change takes and how much disruption an owner accepts, not about how often we sample; narrowing it is a product decision with its own blast radius rather than a side effect of this one.

An index is redundant when it is a proper prefix of another compound index — e.g. {a:1} is covered by {a:1, b:1}. But prefix match alone is not sufficient; the classifier must confirm option compatibility, or it will drop something that does more than accelerate queries:

• Sort direction of the shared prefix keys must match or be uniformly inverted. Both engines serve a sort from a b-tree when the ordering it needs is the key pattern or its exact full reverse, so (a DESC, b DESC, c ASC) already covers (a ASC, b ASC) — measured identically on SQL Server 2022 CU26 and mongod 8.2.9 with only the wide index present (ORDER BY a and ORDER BY a, b plan with no Sort, walked backward). It is all-or-nothing across the prefix and never per key: a MIXED requirement such as ORDER BY a, b DESC is the one case no walk produces and sorts on both, so a per-key reading would propose dropping a load-bearing index. Unordered key types (text/hashed/2dsphere) have no reverse and need an identical key (D83).
• A unique {a:1} is not redundant against a non-unique {a:1, b:1} — it enforces a constraint. Same for differing partialFilterExpression, collation, sparse, TTL, or index type (text/geo/wildcard).
• The survivor must carry every column the victim covered. SQL Server indexes can hold non-key columns (INCLUDE) that answer a query from the index alone, so a longer key list is not automatically a wider index: (customer_id) INCLUDE (total) answers SELECT total WHERE customer_id = ? and (customer_id, status) cannot answer it at all — 6 logical reads against 1124, measured on 2022 over 200k rows. coversIncludes checks the victim's includes against the survivor's keys and includes, and gates the unique advisory and the re-order replacement as well as the drop (D80).

MERGE is the inverse: when two indexes are each partially useful, propose a single superset index that covers both, then retire the two — via the same apply pipeline.

Hard block. These are never eligible for an automated drop, regardless of usage:

• The _id_ index (mandatory).
• Unique indexes — they enforce a constraint; zero query ops does not mean unused. Dropping one can allow data corruption.
• TTL indexes — they expire documents; low query usage is expected.
• Shard key indexes — the cluster breaks without them.
• Text, 2dsphere and legacy 2d indexes — the index is the access path, so hiding one makes $text or $near fail rather than slow down. See below: this one is not a matter of bias, the gate cannot be run at all.
• Partial / sparse indexes serving a constraint.
• Any index younger than the observation window (insufficient data).
• Any index used on any replica-set member.

Every other member of this list is protected because dropping it does something no latency gate can see. These three are protected because the gate cannot be run on them.

Measured on mongod 7.0.39, each index hidden with collMod:

So the hide stage is an outage rather than an experiment, and every gate that follows reads that outage as confirmation the index was dead:

• $indexStats.accesses.ops stays at 0 — the query never reaches the index. The pre-flight re-check sees zero recent ops and passes.
• $collStats latencyStats does count the failed reads, and they are faster than the baseline they are compared against: 159 µs/op against 245 µs/op on the same server. evaluateRegression returns STABLE.

Nothing in the pipeline observes query errors, only latency, so there is no point at which the un-hide fires. The drop graduates on the strength of the damage the hide did. Closing that — an error-rate signal beside the latency one — is a separate piece of work; this list is what makes it unnecessary for safety.

hashed is deliberately not on the list. Hidden, it degrades to a collection scan like any ordinary index, which is exactly the regression the observe gate exists to catch.

2d is on it, and getting it there needed a fix one layer down: the collector normalized every unrecognized key form to 1, so a legacy geo index arrived as {loc: 1}. That is not merely unprotected — as an ordinary ascending key it also reads as a key-prefix of any wider {loc: 1, …} index, which makes it a DROP_REDUNDANT candidate on structure alone, with no usage evidence required (§6.4). The direction is now carried through (D129).

What the advisory tier does NOT cover any more. It used to report every never-drop index that showed no usage — unique, TTL, shard key — and that was wrong on its own premise (D134). $indexStats.accesses.ops counts query access and nothing else. Measured on mongod 7.0.39: a unique index that enforced its constraint across fifty inserts and refused a duplicate sat at ops 0 throughout, and a TTL index expiring documents does the same. FLAT_ZERO is those indexes' permanent expected state, so "no recorded usage" was never a finding about them — which is what the safety list above has always said in words ("zero query ops does not mean unused", "low query usage is expected") while the advisory tier reported it anyway. On one real cluster that was twelve unique and five TTL rows, every one scoring 72 with identical text.

A text index is the opposite, and that is why this one survives: three $text queries move its counter 0 → 3, so zero means the queries genuinely did not run. The visibility those rows used to provide is now the index inventory's job (§8.2), which shows every index a cluster has with its flags and usage whether or not the engine has an opinion about it.

Where the text/geo finding goes. Into the advisory tier, not into silence: an unused text or geo index is reported as ADVISORY_REVIEW with its score and its own rationale, naming the operator that would fail. promoteByScore excludes ADVISORY_REVIEW at every threshold including 0, so it can never be approved unattended, and the reader is told to drop it by hand if the feature it served is gone. Refusing to report it instead would be the invisible-refusal problem of §6.2 (#277) in a new place.

Bias: when uncertain, keep. The one exception is not an exception to that bias — see below: re-ordering a protected index's key DIRECTIONS removes nothing, and is the only change to this list's members the engine will ever propose.

Re-ordering, and it removes nothing.

A unique index's guarantee is a property of its key set, not of its key directions: {a: 1, b: 1} unique and {a: 1, b: -1} unique enforce exactly the same rule. That is verified against mongod 8.2 rather than reasoned about — both indexes coexist, a duplicate is refused while both are present, and after dropping the ascending one the descending one still refuses it. So a direction change is a swap, not a drop, and the whole of the never-drop argument above (dropping one permits data no latency gate can detect) simply does not apply.

The addressable set is much smaller than "unique indexes", and every exclusion is a fact about MongoDB rather than a policy:

Which leaves exactly one target: a compound unique index whose direction pattern cannot serve a sort the workload performs. Whether that set is large enough to matter on real fleets is a question the engine can now answer — a REORDER proposal is the count — and it was the first thing #127 asked for.

What "cannot serve" means precisely. Not "the key list differs". A field under an EQUALITY predicate is pinned to one value, so the index's direction there constrains nothing: measured on mongod 8.2, {a:1,b:1,c:1} serves find({a:1}).sort({b:-1,c:-1}) straight from the index (equality pins a, and the rest is a full inversion, which is a backward scan) and does NOT serve find({}).sort({a:1,b:-1}) (a partial inversion, which no single scan direction produces). analysis/reorder.ts walks the ESR order, takes only the positions that carry an ordering requirement, and asks whether the index matches them identically or inverts them entirely.

Why it is not free, and what each cost bought:

• A hinted index is a HARD VETO, not a scoring penalty. .hint("a_1_b_1") against an index that is now a_1_b_-1 is an ERROR, not a slower query — and the default index NAME encodes the directions, so a hint breaks by name as surely as by key pattern. Nothing downstream would catch it: the post-build watch measures WRITE latency, and the broken queries would already have stopped running. The signal is the profiler's (collectHintedIndexes), so absence is not proof — it only ever protects.
• Build before drop, never the reverse. Both indexes enforce the constraint while both exist, so there is no instant without it. Dropping first would open one, and a duplicate written in that window makes the rebuild fail and leaves the collection with no constraint at all. The existing create → watch → graduate → retire sequence already works this way; this feature is never allowed a shortcut through it.
• Every option is carried over verbatim — unique, partialFilterExpression, sparse, collation — and they travel ON THE ROW from proposal time rather than being re-read at build time, when the original may already have been changed by somebody else. A build with no options recorded is refused rather than attempted.
• A flip that would break a shape the index serves TODAY is refused. Nothing measures that afterwards, for the same reason the hint veto exists, so which pattern to keep becomes a judgement about the workload rather than a free improvement — and the engine does not make it. It reports an advisory instead.

How the protected index is finally dropped, which is the part that had to be opened up without widening anything. isNeverDrop is untouched and preflightDrop still refuses a protected index on sight — unless the row NAMES the index that replaced it. That claim is then checked against LIVE state at the moment of the drop: same fields in the same order, only directions differing, every option identical, present and not hidden. If the replacement was itself dropped, hidden or rebuilt differently in the meantime, the drop is refused and the original stays. The exemption is therefore tied to a specific replacement rather than to a class of rows, and it cannot leak into the ordinary redundancy path.

Approval-only, whatever the auto-apply threshold says. Not because the guarantee is in doubt — it is provable, and proved — but because a change to a constraint-bearing index is a different felt risk from adding one, and the person who answers for that database is entitled to look at it. Enforced in jobs/apply.ts alongside the advisory exclusion, rather than by scoring it low, so a threshold set to 0 cannot reach it.

A silence this closed on the way. A protected index whose directions did not match used to produce NOTHING: sortOrderAdvisories excluded it from being a blocker, and recommendCreates suppresses the create whenever an index covers the fields, with no such exclusion. Not a create, not an advisory — the engine had a finding and said nothing about it, forever. Advisories now include protected indexes, and are skipped only where a re-order proposes doing it properly.

The drop side looks at every index it can see. The add side cannot: reading a workload and pricing candidate indexes for every namespace on a cluster is work that mostly finds nothing, so jobs/suggest.ts decides up front which collections are worth the pass.

That decision is cost, not size. A scan costs documents times frequency, and a gate on documents alone gets it wrong in both directions — a 900-document collection scanned five hundred times a second is 450k document reads a second and looks small, while a 50,000-row lookup table scanned twice a day looks large and costs almost nothing. The measure is documents walked per week (analysis/severity.ts, weeklyScanCost), summed across the collection's shapes, from docsExamined where the source reports it and from collection size times the execution rate where it does not — a collection scan walks the whole collection by definition, so the estimate is sound.

Two things sit outside the cost gate on purpose. A blocking in-memory sort walks no extra documents and still fails outright at 100 MB, so it earns analysis on its own. And $lookup joins are recorded before the gate, because what an unindexed join costs belongs to the foreign collection, not the one issuing it — the foreign side is then priced the same way, its size times how often the join runs.

Ordering matters here: the size floor has to run before the workload read so the workload can be fetched for every surviving namespace in one call (§9, collectWorkload), while the cost gate cannot run until the workload is in hand. Anything decided in the first pass is therefore limited to what is knowable without the queries.

Every recommendation moves through an explicit state machine. Ordering is part of the safety guarantee.

stateDiagram-v2
    [*] --> PROPOSED
    PROPOSED --> APPROVED: human, or opted-in auto-policy
    APPROVED --> HIDDEN: collMod hidden:true (instant, reversible)
    HIDDEN --> OBSERVE: grace window, watch for regressions
    OBSERVE --> APPROVED: regression detected → un-hide, re-evaluate
    OBSERVE --> DROPPED: clean window + human confirm
    DROPPED --> [*]

1. PROPOSED — shown with rationale and estimated savings.
2. APPROVED — by a human, or by an auto-policy only where the customer opted in for that class of action.
3. HIDDEN — execute collMod { hidden: true }. Instant and fully reversible. The planner stops using the index; the index still exists.

   Note: a hidden index is still maintained on writes and still consumes disk and RAM. Hiding yields no write/space benefit — it is purely a safety probe. Real benefit arrives only at DROPPED.

4. OBSERVE — grace window (default 30d, see §7.3). Watch for query-latency regressions and for failed operations — two separate questions, see below. On any degradation, un-hide instantly (one collMod) and re-evaluate.
5. DROPPED — only after a clean observe window, and always human-confirmed. This is the sole irreversible step.

This page claimed "latency regressions or errors" for a long time before the errors half was true (D131). It is worth being precise about why the omission was not a small one.

Measured on mongod 7.0.39, with a text index hidden and its $text queries therefore failing outright:

So the hide broke the application and every gate downstream read that outage as the result it was hoping for. A latency gate cannot be fixed to catch this; it is answering a different question correctly.

The failure signal is ONE-WAY. Failures seen roll a hide back; failures unseen decide nothing. That is not a hedge — every source is optional and one engine has none, so a gate that required the signal would refuse every drop on every cluster that cannot produce it.

Neither top nor serverStatus substitutes on MongoDB, both probed: per-namespace top totals are total/readLock/writeLock/queries/getmore/insert/update/remove/ commands with no failure field, and serverStatus.metrics.operation carries only specific errors (writeConflicts, temporarilyUnavailableErrors) server-wide.

Four verdicts, for the same reason the latency side has more than two:

• INTRODUCED — the hide is implicated: a query naming the index in a hint failed, or failures a missing index can cause appeared after a clean day (see Only a failure a hidden index can cause counts, below). The only one that acts: un-hide, reject, cool down, and tell the owner in as many words that a latency check could not have caught it.
• INCONCLUSIVE — failures exist but cannot be attributed: they were already happening, there is no before to compare against, there are fewer than three, or the clean before reaches back less than a day. Reported and never acted on: a collection with its own pre-existing errors must not be able to veto every drop on it forever.
• CLEAN — nothing the hide could have caused seen since it. Never "nothing happened": system.profile is capped, so the source reports its own reach and absence is only a claim within it. The reach is the whole ring's oldest entry, since one ring per database holds every collection's operations. And a clean window carries what its source was set up not to record, because that can be the very thing asked about (below).
• UNAVAILABLE — no source, so no question was asked. Written into the audit line like the others, because a gate that ran and cleared the drop must not read the same as a gate that never ran (D19), and it names its cause (#596).

A window can be real and still blind (#596). A MongoDB profiler at level 1 keeps operations slower than slowms, and a failed operation is fast: a hint at a hidden index fails in 0 ms. On mongod 6.0, 7.0, 8.2 and 9.0 a level-1 profiler with no filter recorded none of those failures, so before #596 any namespace with a slow query in the ring read CLEAN over a window that could not have seen a failure. The collector reads the profiler's settings (profile: -1, readable with the engine role) and says what they miss:

SQL Server's equivalent is the capture mode: below QUERY_CAPTURE_MODE = ALL the store keeps the queries it judges frequent, and the line says so.

What the audit trail says, so a skipped check never reads like an error:

Most clusters run with the profiler off, which left the check above with nothing to read: 0 of 36 hides on the hosted deployment ever recorded a baseline. So on MongoDB, for every database holding a drop in flight (APPROVED or HIDDEN), Indexterity turns the profiler on itself — level 1 with a filter of four clauses:

The marker is what makes the watch safe to leave on a cluster. A restore puts back exactly what was there; the filter says whose it is to anyone who reads it; and a person can be told apart from a restart — setProfilingLevel(0) keeps the filter, a restart clears it — so a profiler somebody turned off stays off. Another registration's marker is left alone, slowms 0 is declined (copying it would profile everything), and a level-2 profiler is complete already and untouched. failure_watches lists the databases to give back once nothing needs them; finalize does that hourly. It also remembers what a restart takes off a node (#641, below), and how far back each database's rings reach (#642).

The hide waits a day for a baseline (FAILURE_BASELINE_MS): INTRODUCED needs a clean before, and a nightly job that fails on its own must not read as the hide's doing. It costs nothing in practice — hides happen only inside the daily change window — and nothing at all where the database's own profiler already records everything. Hints are checked hourly while they are recorded, because the hint clause logs every such query too. The wait is not the whole of it: what can be read of that day is what the ring still holds, so the verdict checks the reach itself (#625, below).

The trail says why a drop waits (#630). The waiting line is written by whichever pass keeps the watch, finalize hourly or apply in the window, whenever the trail does not already name the instant the drop waits for. It used to be written only when apply itself began the watch, and production's drops waited for days with empty trails.

A restart is put back across, not started over (#641). A restart clears a node's profiler level and filter, marker and all, and nothing else: system.profile keeps what it recorded. Production's members restart almost daily, and when each restart re-armed the watch from then, every waiting drop lost a day per restart: 24 waiting again lines in two days, and eight approved drops hid only after the first day with no restart. Our own snapshots date the restarts: per-member $indexStats.accesses.since moves when a member restarts. Now:

• failure_watches remembers each watch: every namespace's start, when each node was last seen watching, and the stretches nodes missed.
• A node seen watching within the last two hours (WATCH_RESUME_MS: the hourly pass, and one more it may miss) whose marker is gone is put back with the starts as they were. The drop's day holds, and the trail has nothing new to say.
• The stretch from its last sighting to the re-arm is written into that node's own marker, in the same command that re-arms it, so a pass that fails afterwards cannot lose it. Every reading that spans it names it, in the HIDE line before the hide and in the DROP line after it.
• The verdict does not change for it. A stretch missing from the baseline can only make a failure after the hide look new, which rolls back. A stretch missing after the hide is the blind spot a start-over always left.
• A node unseen for longer starts the day over, and the trail says waiting again: … started over at ….

Sightings are per node, not per database: a member that cannot be dialled is left out of a pass, and a database-level sighting would vouch for a node nobody read.

The dashboard says why an approved drop waits (#642). The recommendations list carries, beside the rows (as usage is, because a mutation returns a row and reads no watch):

• waits: each approved drop's hideableAt, its watch's start plus the day, from a watch the last pass found in place on every node. The table says hides after <time>, or waiting on the change window once that has passed or where there is nothing to wait out.
• watched: each watched database's ring reach, read by the same hourly pass. When a ring holds less than a day, the overview says so above the recommendations and links to the recipe for a bigger one in Connecting a cluster. On production, msb-app's 1 MB ring held about 47 minutes, and that used to be said only inside each HIDE line, after the hide.

One drop that cannot be hidden does not stop the others (#630). Each drop runs on its own:

• if its pre-flight, hide or baseline throws, an index that pass hid but had not yet recorded is put back;
• the trail says hide failed, retried next pass: <error>, once per distinct error;
• the rest of the drops go ahead;
• the pass still fails at the end, so the cluster shows as blocked.

Before this, an exception ended the pass. The next in-window pass met the same drop first, and the block it left was cleared by the next out-of-window apply, which returns ok without doing anything.

Per node. A replica set's members are armed, read and given back one by one: a filter set through the set's connection reached the primary alone, and a failure routed to a secondary landed in that secondary's ring only. Through mongos, profile: 1 is refused (BadValue), so a sharded cluster says why it is unwatched.

What it costs while a window runs: operations slower than slowms are written to system.profile (at level 0 they were only logged), and failures and hints on the watched collections appear as slow-query lines. The privilege is the opt-in — enableProfiler, in the provisioned role and diagnosed as optional; without it the HIDE line says the check cannot be watched and why.

The causes, per engine: MongoDB — the credentials cannot read system.profile, the profiler is off, or it has recorded nothing yet. Through mongos the read reaches the database's primary shard's ring, but profile: -1 answers for mongos itself, which always says level 0 (probed with the shard at level 2), so there the line says the settings cannot be read rather than that the profiler is off. SQL Server — Query Store is off, its settings are unreadable, or it holds no plan that reads the table; PostgreSQL — no source exists, and nothing is hidden there anyway.

Production parked msb-app.exercise · cycle.id_1 — a redundant prefix of three visible compound indexes, so every plan it offered was still on offer — on 3 failed operations since the hide, and none in the 54 minutes of history readable before it. Two faults made that line. Every failure on the collection counted: a duplicate key, a document its validator refused, a client hanging up. And the day the hide waited for was not the day the verdict read: the wait counts from the watch's start, but system.profile is a capped ring, 1 MB per database by default, and that database turned it over in 54 minutes.

Measured on mongod 6.0.28, 7.0.39, 8.2.9 and 9.0.2, with the index hidden:

So the collector sorts what the ring holds, with one $group on {errCode, errName, command.hint} on the server, into three classes, and the verdict judges each on its own terms:

• hinted — BadValue whose hint names the index, as a name, as {$hint: name} or as its key pattern field for field and in order. One is enough, with no baseline: a hidden index refuses every hint at it and a visible one serves them.
• suspect — 50, 292, 291, 27. Three, against a clean baseline that reaches back a full day before the hide; shorter, and the verdict is INCONCLUSIVE with the length in the line.
• unrelated — everything else. Counts against nothing, and is listed in the line, so a reader who sees failures on the collection is told why they did not count.

The reach under our own watch is the watch's start until the ring has turned over. It used to be the later of the start and the ring's oldest entry, but on a quiet database the first operation worth keeping may come hours in, and a ring that had lost nothing read as a short one. Whether it has turned over is read from its fill — dataSize (find on the ring is enough) against the cap in its listCollections options; $collStats is refused to the engine role. A ring that has turned over stays within one entry of its cap (1,048,430 and 1,046,791 of 1,048,576 bytes, probed on 6.0 and 9.0); within a tenth of the cap counts as turned over, the side that understates.

SQL Server's Query Store keeps no error number and nothing of what a failed execution hinted, so all its failures are suspect, and it needs the same day — which its retention usually covers.

Not done: a real day of baseline on a database whose ring turns over in an hour, which would mean sampling the ring every pass and keeping tallies in postgres. A hint is the one break a hide of a redundant or unused index causes outright, and that needs no baseline. A database owner can buy the day instead, with a bigger ring (Connecting a cluster).

Errors are checked before latency, and before the observation window is required to be complete. A slowdown can wait for the window to fill; an outage cannot.

The universal version of this signal was measured and does not exist. A hide is subtractive, so reads it was serving cannot get faster — which suggests a window that measures faster is suspicious, and that check would need no optional source at all. It was replayed against 9,105 judgeable ratios from 1,061 real production namespaces that were never hidden, and the low tail is far too wide: p25 = 0.845, p10 = 0.690, with 14% of untouched windows below 0.75. Against that, a fully broken collection measures 0.88 on mongod 7.0.39 — a quarter of clean windows look more suspicious than total breakage. A failed query still pays query planning (~140-160 µs, the same order as a cheap indexed read), so failure barely moves the average. Requiring ops to be up as well filters almost nothing, and the tail does not narrow with window length. Do not re-file this without a signal that separates the two classes; see D132 for the full numbers, including the misleading 159-against-245 figure it was filed on.

stateDiagram-v2
    [*] --> SCHEDULED
    SCHEDULED --> BUILDING: maintenance window (rolling)
    BUILDING --> ACTIVE
    BUILDING --> ROLLED_BACK: build harms performance → drop
    ACTIVE --> [*]

Creates are scheduled into a maintenance window (a rolling build still drives I/O and can cause replication lag). The build spec is retained as a rollback token: if the build harms performance, drop it.

• Default 30 days. 7 days catches daily and weekly patterns but misses monthly jobs (billing, month-end reports); dropping an index a monthly job needs causes an outage weeks later. 30 days catches monthly. The cost of waiting is only delayed benefit, never lost benefit — outage from dropping too soon is real damage.
• Adaptive down. If ≥90 days of zero-usage snapshots already exist for the index, shorten the window and drop with confidence.
• Pattern-aware. PERIODIC_DEAD (see §6.3) can proceed faster; PERIODIC_ALIVE never proceeds.
• Configurable per policy (a customer who knows there are no monthly jobs may set 7 days).

A rejected index is written to index_cooldowns from three places: the observe gate when hiding it slowed reads, the post-build watch when building it slowed writes, and recordManualVeto when an owner cancels a pending drop or undoes a completed one. The row carries the writer's own sentence for why, how many times the index has now regressed, and the date the engine may propose it again — or a NULL there, which means never.

Two roads in, and they are different decisions (D136).

The engine's road is its own evidence, so it escalates and it lets go. One observe window for a first regression, doubling per repeat, capped — 30 / 60 / 120 / 240 / 365 days at the default window, where it used to be a linear 3x·n from 90. Note the direction of that change: the cap arrives before the doubling overtakes the old line, so the new curve is lower at every step. The engine retries sooner and never parks an index for more than a year on its own.

One window is a floor rather than a preference. The evidence for a drop is a usage history, and the reason it failed is that something used the index during the window — so retrying inside another window re-runs the same experiment against the same workload, hiding somebody's index again to learn what was already learnt.

What holds a repeat offender down is not the park alone. regressionWeight (§6.x) docks the confidence score by the same count and fades it over the same span, so a twice-regressed index comes back eligible and still scores too low to auto-apply. Both read the curve from analysis/cooldown.ts rather than each restating it, or the fade would describe a window nobody was granted.

The human's road is a question, not a default. Cancelling a drop asks how long to leave the index alone: a number of days, pre-filled with what the engine would pick for its own first retry, or never. It used to be a flat 90 days that was neither shown nor chosen. Never is stored as a null until and not a distant date, because never is not a very long time — a sentinel would eventually pass and the index would become eligible on a day nobody picked.

And there is a way out. clearCooldown (POST /clusters/{id}/cooldowns/clear) un-parks an index from the Parked panel. It is a DELETE of the row rather than an expiry backdated to now, which matters because regression_count lives on that row and feeds both the escalation and the score penalty: backdating would make the index eligible again while still docking it for regressions the owner has just said to forget. Keyed on the namespace triple rather than a row id, which is also what lets an owner clear a collection-level park (#282), whose index name is the empty sentinel.

For a long time nothing read it back (#159). The engine consulted it — activeCooldownKeys is what keeps a regressed index out of the next pass — but no controller did, so the only trace a reader got was a clause appended to a recommendation's rationale, and only while that recommendation still existed. A cluster with six parked indexes looked exactly like a clean one, which is the worst version of this: the engine's silence and the engine having nothing to say rendered the same.

listCooldowns (GET /clusters/{id}/cooldowns) is that reader, drawn as the Parked panel on the cluster page. Three things it gets right on purpose:

• Read standalone, never joined to recommendations. A cooldown outlives its cause — a manual veto parks the index for as long as the owner said, and the recommendation that was cancelled is a row the next classify pass rewrites or retention prunes. A join would silently drop exactly the longest-standing entries.
• active is computed on the api's clock, not the browser's, and the headline count comes from the api rather than from parked.length. A laptop an hour behind must not be able to draw a parked index as eligible.
• Expired rows stay on the list, marked back in scope. regression_count has no other home in the product, and "regressed 3×" on an index the engine is now free to propose again is the most useful sentence the table can produce. Zero is not "never regressed": the two owner paths set it to zero deliberately, because nothing regressed, so the panel draws no badge for them at all.

Uncapped, unlike the two reads §11 bounds. The bound here is structural — the table is unique on (cluster, database, collection, index), so it holds at most one row per index, and only per index something has actually parked.

The world changes between recommendation and execution. Immediately before any hide/drop, the executor re-fetches live stats and re-runs the classifier against the current index spec. If the index is no longer unused, its options changed, or it is now unique/TTL — abort and re-propose. Classification is centralized in core; this final re-check runs at the executor.

An outage of days or weeks is the case where a safety gate is most likely to be wrong, because the counters it compares are cumulative since mongod started.

• Restart during the window. The drop side no longer keeps a baseline of its own. It reads latency_samples — the history collect already stores — where a reset nulls the one window spanning it and leaves every other window standing, so a restart costs the stretch it lands in rather than the whole observation. observedWindow sums the windows that produced a reading and the drop waits on that sum, not on elapsed wall clock: a cluster that restarts nightly fills its 30 days a little slower than the calendar rather than never.

  It has to work this way round because the old gate could not be repaired. latencyRatio's denominator is baseline.latencyMicros / baseline.ops, the collection's lifetime average before the hide, and the only record of it was the counter the restart zeroed — re-baselining afterwards compares the hidden period against the hidden period. Aborting was correct, and it never terminated: a 30-day window on a nightly-restarting cluster always contains a restart, so the drop was proposed, hidden, aborted and re-proposed indefinitely.

  Four verdicts now, because "keep observing" and "nothing from before the hide to compare with" are both cannot tell and neither may be spelled STABLE (§D19). OBSERVE_WALLCLOCK_MULTIPLE bounds the wait at both ends: finalize un-hides past it rather than leaving an index hidden for evidence that is not coming, and classify.ts reads the same constant to decline PROPOSING a drop whose collection's duty cycle says it would hit it — surfaced as the unobservable suppression guard rather than as silence. What still decides whether a measurement is owed is the row's own baselineReadOps, which apply.ts records exactly when the index was really hidden; that reading is also the reference of last resort, safe as a rate in a way it was not as a subtrahend.

  The create side works the same way, over write_ops instead — observedWindow takes the metric as a parameter. It used to reset built_at to now on a restart, so on a cluster restarting oftener than the window a built index never graduated: nothing was executed wrongly, but retireSuperseded (its only call site is the graduation branch), the cumulative check, and the release from the watched guard all hang off that event, so a correct finding was never made and the index went on suppressing its own.

  What "give up" means differs between the two, and that asymmetry is the design. The drop side has something to undo, so past the cap it un-hides and re-proposes. A built index does not — it is serving traffic either way — so its watch concludes on whatever it managed to measure and records that it was cut short. Leaving it open is the worse option: it withholds the retirement on a collection quiet enough that nothing was going to be measured anyway. A REGRESSED reading still rolls the index back at any point.

  One subtlety both sides share: a window is excluded only when a reset ate it, never for being quiet. Quiet time is time we watched during which nothing could have been hurt, and minWindowOps gates the ratio rather than the observation count. Filtering it out of the count instead made a collection nobody queries accumulate nothing, never fill its window, and get un-hidden and re-proposed — the same non-terminating cycle, reached from the quiet side.

  Still on the old mechanism: judgeCumulative (#282) differences a stored baseline across a whole run of builds and answers UNOBSERVABLE over a reset, so on a fast-restarting cluster the collection-level reading stays silent. It only ever reports, so the residual is non-destructive.

• No restart, just nobody watching. The counters kept climbing, so the observation is still valid — the window merely lasted longer. Nothing special happens, which is correct.

• Usage history with a hole. Snapshots stop and resume. classifyUsage cannot tell a busy index from a dead one across a gap, so usageHistoryIsTrustworthy (≥ 3 collects, no gap beyond maxGapHours, newest confirmation within it) gates every usage-based finding. Collects rather than rows, and the gap is measured from one run's lastSeenAt to the next run's capturedAt — see §11.2, which is also why a quiet run is not itself read as a hole. Structural findings (redundancy, unique-prefix) are unaffected. This also stops a brand-new cluster from proposing a drop for every index on its first collect.

• Noise. A cluster that stays unreachable fails collect every 6h forever; owner alerts are capped at one per cluster+task per day. The cap is a CLAIM taken before the send, not a check (D86), so a send that reached nobody used to spend the day's alert on a mail that never left the process. Since D121 a failed send hands most of the window back — the stamp is moved to five minutes from now — so an SMTP fault delays the alert rather than losing it, while the claim itself stays taken and two ticks still cannot both mail.

• What the analysis passes READ is a cost too (#474, D145). classify read the whole retained window of latency_samples and index_snapshots back on every pass, and it was chased by every landed collect — so hourly per cluster until D147 gave the chase its own cadence. Two habits made that far more expensive than it needed to be, and the hosted deployment's Neon transfer bill — 4 GB in thirteen days against a 68 MB database — is what surfaced them. A select() with no projection shipped thirteen columns where ten were used, and the three spare ones (id, cluster_id, the generated span) were half the bytes. And the join onto cluster_indexes re-shipped each index's database, collection, index_name and spec once per snapshot — 475 kB of dimension rows arriving as 12 MB, an average of 27.6 copies each — which undoes on the wire exactly what that table exists to avoid in storage. Both fixed: 36 MB to 21 MB on a full pass. The dimensions are fetched by the ids the snapshots reference rather than by cluster, because the join's only condition was on the id and reading by cluster would silently add one. What remains is perMember, 15 MB of the 21 MB, which is load-bearing — the counter-restart rule compares each member's ops and since across runs.

• The latency half of that read is folded in postgres now (#484, #485, D148). D145 and D147 are both constant factors — bytes per row, and how often the pass runs — and neither changes the fact that one pass costs O(retained history). latency_samples is that slope, and it is the table that run-length-collapses nothing (76% for index_snapshots, 0% here), so its read is one row per collection per collect for the length of the entitlement.

  What the analysis needs from it is two scalars per collection — hours the collection served reads, and whether an observe window on it could finish — and both are folds over consecutive readings, which lag() expresses exactly. Measured on postgres 18 against the hosted deployment's shape extrapolated to a full PRO entitlement (144 namespaces, hourly, 183 days, 632,448 rows, a 195 MB table), with explain (analyze, serialize text):

  	server time	serialized output
  ten raw columns	899 ms	75,665 kB
  the fold	3,981 ms	11 kB

  The trade is named rather than glossed: server time up about four times, wire bytes down about 6,900 times, against a pass whose five-minute budget it now uses 1.3% of. That is the right way round for a deployment that ran out of network transfer and not out of CPU — and 601 ms of the old query's 899 was postgres serialising the answer, which is the clearest statement of what was being paid for. The remaining cost is the sort behind the window, not the median: gaps are grouped to one row per distinct interval first, so on a steady cadence that stage sees a handful of rows per namespace.

  The rules stay in analysis/. The query produces activeMs, elapsedMs and drawableMs; activeHoursFrom and observationCanFinishFrom decide what they mean. So what is duplicated is arithmetic that can be cross-checked, not thresholds that cannot — and it is cross-checked, by an integration test that folds the same rows both ways and asserts exact equality over eight shapes (a run, a hole, a counter restart, an even gap count that lands the weighted median on its mean-of-two-middle branch, an idle collection, a lone reading). Four deliberate mutations of the SQL were confirmed to fail it.

• index_snapshots keeps the entitlement window, and that is a refusal rather than an omission (#485, D148). The tempting reading of "bound the engine's read by something other than the entitlement" is read fewer days, so the gates were enumerated first — a gate quietly reading less than it thinks it does is a wrong verdict with no symptom:

  gate	minimum evidence	summarisable
  observation and span floors	counts and summed epoch spans	yes
  the two gap checks	adjacency between consecutive runs	no
  counterEpochs / restartedBetween	adjacent runs, member by member — a stored per-row flag since #534	yes
  classifyUsage	the whole series, because a cadence is only visible in it	folded, never truncated

  Two of the four rows moved, and the last one moved in a way worth stating precisely: classifyUsage is still NOT truncatable, and the fold does not truncate it. It reads the whole entitlement window and returns four numbers per index — total looks, looks that saw the counters move, when the newest run ended, and when the newest run that moved began. The cadence is computed over every day the plan entitles, exactly as before; what stops crossing the wire is a row per collect per index. Measured at 5,351 kB to 28 kB over 29,892 production rows for 190 indexes (#534), and the figure no longer grows with retention, because it is a function of the index count.

  The counterEpochs row moved for a different reason. What a run's counters MEAN — the activity across it, and whether they restarted — is a fact about that run and the one before it, and the collector holds both sides at write time. It stores them (ops_delta, ops_total, counters_restarted, counters_started_at), so the reader no longer reconstructs them and no longer needs per_member at all: 9,510 kB to 5,351 kB over 29,868 production rows, and every input to the epoch rule is now a per-row value. ops_total is stored beside the delta because the window's OLDEST run has no predecessor the window can see and is read in full on purpose; a delta alone would quietly narrow that.

  Which is what let superseded runs drop their copy (#550, D173). Once the reader stopped needing per_member, the JSON was 402 of the 841 bytes each index_snapshots row costs on disk, on a three-member replica set. Only the NEWEST run per index still needs it. The next collect fingerprints it and differences against it, and the live readers read the last collect's batch. So when a collect supersedes a run, it empties the old run's per_member to [], and migration 0067 did the same to the stored history. That cuts total storage growth by about 28%. A run keeps its JSON while any fallback could still read it: its own stored columns are missing, or those of the run after it are, which is only ever a row an api predating #537 wrote. usage-evidence.int.test.ts runs the migration's own file and requires the fold to be unchanged, in postgres and in JS.

  The LAST row has not moved, and it is still what settles the question.

  The last one settles it. A truncated history cannot see a cadence, so a monthly job's index reads FLAT_ZERO — score 50, droppable, the most confident verdict the engine has — where the full series reads PERIODIC_ALIVE, which is not droppable at all. Wrong answer, more confident, on less evidence, and nothing about the resulting finding looks unusual. That is D96's objection and it stands. It is also the cheaper thing to be right about: index_snapshots rows are proportional to how much the cluster changes, not to how long we have watched.

  Carried derived state — #484's primary framing — was not taken, and still has not been: there is no new table and no retention interlock. What #537 added instead is four columns on the run itself, holding what that run's counters MEAN rather than a summary of the series. That is not carried state — it is a fact about one row, written once by the process that already had both sides in hand — which is why it needs no interlock and cannot go stale.

• Visibility. cluster.lastCollectedAt feeds a staleness badge, so figures from before a gap cannot read as current — and since D112 the cluster row also carries WHY the gap exists. Staleness alone was not enough: a paused schedule, a cluster with nothing left to collect and a cluster nobody can reach all produce the same badge, so the one that needs somebody read as the two that do not. runClusterTask writes a cluster_blocks row where it already records the outcome metric — the reason, the sentence, and when the condition started — and the dashboard draws a destructive badge plus a banner naming the reason, how long, the failure in its own words, and what to do about it.

  One row per PASS, and a pass that gets through clears only itself (#462, D142). This was four columns on clusters, which is one slot for six passes: the later failure overwrote the earlier one, and any pass that finished cleared the lot. Measured on the hosted deployment on 2026-09-07 — a collect that had been timing out for 19 hours, and all four clusters reading as healthy, because the five-minute probe beside it kept succeeding and kept clearing the row. The diagnosis was made, written down, and then erased by an unrelated success, which is worse than the absence D112 was written for: it asserts health rather than saying nothing. The key is (cluster_id, task), markUnblocked takes the pass that ran, and since keeps its own clock per pass — so a probe timing out for five minutes and a collect unreachable since Tuesday are two facts with two durations. The cluster's blocked field is a LIST, longest-standing first: the page draws a banner each, and the heading keeps one badge for the oldest, because at that altitude the question is whether the cluster works.

  "Unreachable" is established at the boundary that dialled, not read off a message (#420, D122). The predicate used to match Node's socket-level codes wherever they appeared — ECONNRESET, ECONNREFUSED, ETIMEDOUT, getaddrinfo — and its own comment said why that was general: those codes are Node's and apply to every driver. They apply to pg, which is what this process talks to its OWN control plane with, and measured on postgres 17 a pool whose server goes away re-raises a bare Error: read ECONNRESET from the next query. A pass is postgres-heavy on both sides at once, so a control-plane blip mid-pass was classified as the customer's cluster being unreachable: their cluster marked UNREACHABLE, their owners mailed about a database answering fine, the day's alert claim burned so a genuine outage the same day was silent — and our own postgres fault silenced with it, because a handled skip is recorded as a queue success and never reaches Sentry. Fleet-wide, too: one failover mails every owner with a pass in flight. The broad patterns are now consulted only inside a customer connection, where postgres/connection.ts wraps every dial and every statement it makes into a typed ClusterUnreachableError; outside one, only what a customer driver ALONE can produce counts — the mongo and tedious error names, mongo's Server selection timed out, and the querySrv/queryTxt markers of an SRV lookup, which node-pg never makes. The postgres adapter is the only one that needed a boundary, because it is the only driver shared with the control plane.

  Every customer pool has an 'error' listener, and each driver needed a different one (#420, #424, D122/D126). A pool is an EventEmitter and an 'error' with no listener is an uncaught exception, so this class of bug kills the api process rather than one pass — and sessions are held for five idle minutes (jobs/connection-pool.ts), which is most of the time a cluster is connected at all. All three drivers were measured rather than assumed, and no two agreed:

  driver	idle connection, server gone	request in flight, server gone	listener
  pg	emits 'error' on the pool — crash	client emits on itself	two: pool.on('error') and pool.on('connect', …)
  mssql	emits nothing at all	late socketError into state 'Final', re-emitted on the pool — crash	one: pool.on('error')
  mongodb	monitoring events only	rejects the operation	none needed

  The mssql one is the worst of the three to find by reading, and is why #424 was filed rather than fixed on the guess that it matched pg: the query rejects cleanly (RequestError: socket hang up, ECONNRESET) and then ~1ms later tedious dispatches a late socketError into a connection already in its terminal state, emits on the tedious Connection, and node-mssql re-emits that on the pool. So the caller's error handling ran, looked correct, and the process died anyway. Its two idle cases — server stopped under an idle pooled connection, and KILL <spid> on one with the session verified gone from sys.dm_exec_sessions afterwards — emitted nothing, so the second listener pg needs has nothing here to attach to and is deliberately absent. mongodb needs none at all: it reports transport faults as serverHeartbeatFailed / connectionPoolCleared / connectionClosed and surfaces them by rejecting the operation, never as a bare 'error' on the client. All three findings are written into the file that acts on them, and mssql/client.ts splits buildMssqlPool out of mssqlPool for no reason except that asserting a listener needs no server while connect() does.

  A read-only pass also has a wall clock (#407, D118). Five minutes, matching the tick, bounds collect, classify, suggest and probe; a pass that outruns it is abandoned under the TIMED_OUT reason and retried on the next tick rather than immediately. A collect or suggest that keeps not fitting is paced instead of abandoned forever (#571, #588, D179, §6.3 Cadence): up to four budgets, every four hours. It frees the worker slot — with WORKER_CONCURRENCY at 1, a pass that cannot finish otherwise keeps the whole fleet queued behind it, which is what it was found doing in production — and, since #454, cancels what the pass had running where the driver registered a way to. engine/inflight.ts carries a per-pass registry in AsyncLocalStorage; the SQL Server connection registers request.cancel around every statement, and runClusterTask abandons the registry on any failure before classifying it. Without that, a budget only stopped the await: the Prod MSSQL collect's Query Store scans ran on for the driver's fifteen-minute request timeout, the next pass found all four pooled sockets busy, and tarn — the pool under node-mssql — gave up after its default 30 s with "operation timed out for an unknown reason", which landed in the nameless ERROR bucket and was retried five times against the same full pool. The acquire timeout is now set to the statement budget on purpose, and that wording maps to PoolExhaustedError, classified with the budget as TIMED_OUT. MongoDB and PostgreSQL statements are not cancelled yet — both need a second connection to do it (killOp, pg_cancel_backend) — and run to their own timeouts. D138. apply and finalize are excluded on purpose: they change indexes and write rollback tokens, and a large build legitimately runs for tens of minutes, so a build is never cut off this way.

  An abandoned pass says WHERE it spent its time (#466, D143). Until 0.19.1 the block, the owner's mail and the log line all carried one sentence — "ran past its 5 minutes budget and was abandoned" — and none of them said which part of the pass was slow. That cost a release: #461 cut the SQL Server collect from ~1,100 statements to ~78 and the tunnelled production cluster still timed out, with nothing to read but the same sentence. engine/phases.ts is a per-pass registry in AsyncLocalStorage on the same terms as inflight.ts, timing the per-database boundaries of collectSnapshots and, on SQL Server, splitting the Query Store catalog read from the plan-XML read — same statement count, nothing like the same cost over a tunnel. The breakdown goes to all three readers, because only one of them gets the mail:

  the collect pass ran past its 5 minutes budget and was abandoned —
  queryStore:planXml 241s/13, indexesByCollection 28s/13, per-collection 9s/362
  

  A phase that has not finished counts, and says so. A budget abandons a pass mid-flight, so the phase running when the clock ran out has not been left — and it is the most useful line in the report. Counting only completed phases left the slow loop out of every report it mattered in.

  Two things the phase report then found, in two days (#470/D144, #478/D146). First, the attribution cache could only be written by a read that ran to completion, and on a tunnelled cluster that read never could: the write happened once after the chunk loop, and an abandoned pass had its statements cancelled before reaching it. Progress is remembered per chunk now. Second — and this is the one that had survived two rounds of looking — the cache was an instance field on a collector owned by a pooled session, and the pool closes a session after five idle minutes. Not a restart: every five idle minutes. So it lives in a module-level store keyed by a fingerprint of the connection string, which is the identity the pool already dooms an entry on when credentials rotate, bounded by an LRU because nothing discards it for us any more.

  Then the bound itself was the problem (#588, D146). It was fifty thousand attributions, about 12 MiB measured rather than the "single-digit megabytes" assumed, and it evicted whole databases: a production store that grew past it lost a database of ~5,000 plans per overflow, re-shipped them on the next pass and went over again, so the probe and the collect both timed out shipping XML they had shipped an hour before. The ceiling is now ten percent of the heap limit (8–256 MiB) in estimated bytes per entry (~30 MiB on the 512 MB all-in-one). Eviction drops individual plans, oldest first, from the least recently written database, into a copy so a pass holding the old map keeps it, and logs at most once a minute when it gives plans up. The same entry now carries what suggest reads off the plan, so either pass's shipping serves the other.

  What it found first (#470, D144). Within minutes of deploying: collect: latencyByCollection 279s/9 (still running), queryStore:planXml 268s/1 (still running), hintedByCollection 6s/8, queryStore:catalog 5s/9. Not round trips — the numeric catalog read is five seconds across nine databases — but bytes, in one database, where eight finished inside the remaining eleven seconds.

  The mechanism is an interaction between two correct changes. planAttributions wrote its result once, after its chunk loop; a pass abandoned mid-loop has every statement it registered cancelled (#454), so the loop throws and the write never runs. The cache that makes this affordable could only be written by a read that completes, and that read never could. Sessions are pooled per cluster, so its lifetime was never the defect — the write was all-or-nothing and the "all" was unreachable. Progress is remembered per chunk now, and PLAN_FETCH_CHUNK drops from 500 to 50 because the chunk is the unit an abandonment destroys and 500 plans of showplan XML is a unit that may not land at all on that link.

  The pass is recorded because the banner used to guess it (#408). Every reason was worded as collection, which is harmless for the five dial failures — they stop every pass alike — and wrong for ERROR, the one reason a pass lands on when the dial worked and the pass did not. A suggest that could not finish was reported to the owner as collection failing. blocked_task follows the sentence rather than the clock: overwritten on every blocked pass, where blocked_since is kept, because it answers what is failing now while the timestamp answers for how long. Text and nullable, so a pass a newer worker names renders as itself and a block predating the column still draws.

  Every pass records how long it took, not only the one that ran out (#571, D178). The phase report reached a reader only when a pass was abandoned, and the block that carried it is deleted by the next pass to get through — so a collect that took four minutes of its five read exactly like one that took four seconds, and no duration was stored anywhere. runClusterTask writes cluster_pass_timings beside the outcome metric, for every outcome but gone: when the run started, how long it took, the budget it ran under, how it ended, and its eight most expensive phases. One row per (cluster_id, task), overwritten, for the grain cluster_blocks has: the question is how the pass behaves now, and a history would grow by a row per pass per cluster per tick to answer one nobody has asked. Best-effort where markBlocked is not, because a block is the only copy of why the pipeline stopped and a timing measures a pass that already did its work. No per-cluster metric either: the job metrics carry no cluster label by design. The overview's Passes panel reads the row, and calls out a pass that landed above 75% of its budget — the next slower hour is the one it does not fit in.

  And a week of the hourly passes, since #639. "Nobody has asked" stopped being true when production's SQL Server finalize ran 26 minutes and failed one day, and 69 minutes two days later (#629). Each was visible only to whoever opened the panel in that hour.

  • What is kept: every run of collect, suggest, classify and finalize goes into cluster_pass_runs as start, duration and outcome, without phases. Rows are pruned at 14 days by the daily sweep, at a few rows a cluster an hour. Probe and apply run every five minutes and finish in seconds, so they keep their last run only.
  • The chart: under the panel, the longest run of each pass in each hour of the last 7 days, read from its own route (getPassHistory). The probe's five-minute refetch of getPasses therefore does not resend a week.
  • Gaps: a stretch with no run for longer than the slowest pace (four hours, plus an hour) breaks the line rather than drawing a level nobody measured.

Restricted to creation only — never an instant drop. "Critical" is defined narrowly: an index that resolves a collection scan on a large, hot collection during an active latency incident. Even then it is gated behind per-cluster opt-in and a maximum-collection-size threshold, and logged prominently. Drops never skip the hide → observe → drop path, regardless of urgency.

How urgent comes from docsExamined — the documents the server actually walked — not from table size:

Workload analysis runs hourly, on its own half-hour offset from the collect pass rather than behind it, because a missing index costs on every execution and most of the old delay was waiting to notice. A critical scan goes from first sighting to built index in minutes rather than the better part of a day.

classify and suggest are re-derivers, not accumulators. Each rewrites its own proposals from scratch on every pass — delete every PROPOSED row carrying that job's source, re-run the engine, insert what comes back — so a finding that stops being true simply fails to reappear. The source column (CLASSIFY, WORKLOAD, RETIRE) exists so each job can identify its own rows rather than guess from the type: a sweep that cleared every PROPOSED row for the cluster would eat the retirement drops finalize files, and nothing re-derives those.

That scheme is only self-consistent while a finding stays in PROPOSED. The moment somebody approves one it leaves the set the sweep clears — and the index it names is still on the cluster, still exactly as redundant or as unused as it was, because the hide → observe → drop path takes days. So the next pass derives the same finding again and inserts it beside the approved one. The dashboard then shows the same index twice, once as APPROVED and once awaiting approval, and approving the second queues a second drop of an index the first is already taking away. Builds have the same shape from the other side: an APPROVED CREATE waits for the change window, and until it is built the index does not exist, so the live index list cannot stand in for the check.

Both jobs therefore skip any candidate whose index already carries a live recommendation of the same family that the sweep will not delete (standingRecommendationKeys in jobs/watched.ts): drops suppress drops, builds suppress builds. Two conditions define "will not delete":

• the row is not in PROPOSED — approved, hidden, scheduled, building; or
• it is PROPOSED but another producer wrote it, so this sweep leaves it alone.

Settled states are deliberately excluded. DROPPED and ACTIVE mean the work happened, and re-deriving is then correct rather than duplicative — an index a graduated build put there is an ordinary index and answers to the same rules as one the customer built. REJECTED and ROLLED_BACK are held off by a cooldown (§7.3.1) rather than by the row's own state, which is what lets a veto expire on its own clock instead of being permanent by accident.

Every one of these guards works by making a derived finding disappear, and until #277 that left no trace anywhere: "nothing to suggest" and "we suggested it and hid it" rendered identically, so a guard that had quietly become too broad was indistinguishable from a quiet cluster. Each pass now counts what it withheld and by which guard into its analysis_notes row (§6.2), and the dashboard reads those counts back under the proposals. Counts rather than the findings themselves, which is enough for what it is for — a guard suppressing more than anyone expects shows up as a number nobody can explain, where before it showed up as nothing at all.

Rows written before the guard existed are cleared by the next sweep like any other stale proposal.

A run of builds is measured against where it started (#282, D101). The post-build watch captures each index's write baseline at that index's own build time, so build #2 is judged against a collection already carrying #1 and #3 against one carrying both — always against the immediately preceding state, never against the original. On graduation, before the baselines are cleared, the collection is also compared against the oldest baseline still live for it; no new column, because every un-graduated build already carries one and graduation clearing them is what ends the chain. The band this adds is 1.3 to 1.5: above 1.5 the oldest row's OWN watch already fires, and it fires by rolling back the oldest index, which is no likelier to be the culprit than the newest. So the collection-level finding reports instead — the owners are told and the collection is parked from unattended builds — and it cannot see an accumulation spread over months, because once every build graduates the chain is empty and the next starts fresh.

The guards are keyed on an index; the cost is paid per collection (#281, D100). Nothing capped how many builds were proposed for one collection, and no proposal knew about the others beside it — so five individually defensible creates on one busy collection were five correct findings that together doubled its write cost. createScore now carries a crowding term: ten points for each index past an ordinary four that the collection would carry once this one is built, counting builds already in flight rather than only what a collect saw, capped at the forty a past regression costs. It is a score and not a cap on purpose — a refusal nobody can see is the invisible-suppression problem above in a new place, whereas a score keeps every finding on screen and lets the owner's own autoApplyScore decide what the engine may do by itself. Net-new only: target_spec.retire exempts an UPDATE widening an index and a MERGE folding several into one, because those leave the collection carrying the same number or fewer. instantCreate gets a hard veto instead of the term, being the one path that never reads the score.

The guards are a suppression and the schema is the net under them (#283). They used to be the only thing holding "one live recommendation per index": the table took duplicates happily, so a fourth producer calling three of the four checks would create them with nothing failing, and classify and suggest run on separate queues, each reading its guard sets before it inserts. A partial unique index over the live states now refuses the duplicate outright, keyed on the claim rather than on the type — DROP_UNUSED and DROP_REDUNDANT both mean this index should go, so a constraint over type would hold one of each, which is precisely the duplicate arriving by two routes. MERGE counts as a build, because its index_name is the compound index it would create. Advisories are excluded, matching the carve-out classify already makes for them by hand: two different things worth telling a human about one index are both worth saying. The producers insert with ON CONFLICT DO NOTHING, so losing the race is a no-op — the other producer said the same thing first, which is the outcome either way. The guards stay, because they do more than deduplicate: they encode why, and that reasoning is what writes the rationale a customer reads.

After each action, measure before/after and persist to roi_metrics:

• RAM / working-set and disk bytes freed (collStats, $indexStats).
• Write-throughput delta (fewer indexes → faster writes).
• Index-count trend.

The dashboard headline is these hard numbers, as four cards (#608):

• Reclaimable: what the open drops would free, in every state from proposed to hidden, with how many wait for review and how many are under way. It is summed by the api (listRecommendations → summary) because the list is capped. It used to be a sum of the PROPOSED rows the page held. On every production cluster those were builds, which free nothing, while the approved drops waited in APPROVED and HIDDEN, so it read 0 KB.
• Reclaimed: bytes freed and indexes dropped, from roi_metrics.
• Built: indexes the engine built and has not rolled back, with the scanning they removed once that has been measured (§8.4).
• Scanning to fix: the documents per week examined by the scanning shapes an open build answers (§8.3), seen in the last week. This is the work the builds waiting for review or for the change window are there to take off the cluster.

Reclaimed is cumulative. It counts what the engine removed, so it only ever climbs, and it cannot answer the question an owner has after a month: is my index footprint smaller than it was, net of everything the application added in the meantime? A cluster where we freed 4 GB while the app added 6 GB has a triumphant ROI card and a bill that went up, and nothing on the dashboard used to show it (#160).

getIndexSizeSeries (GET /clusters/{id}/index-size-series) is that number: total index bytes per day over the same window the latency series uses (LATENCY_SERIES_WINDOW_DAYS). Every collect writes the day's total to cluster_footprint_days (#640, D190): one row per cluster per day, rewritten by each collect, so the day's last collect is what stays. It is pruned with the time series, on the plan's window (§11.2).

The decisions, most of them about being honest with holes:

• A day is what its last collect measured, not what the runs say. The first version (#160) derived every day from index_snapshots and read each run's size_bytes for every day the run overlapped. A run carries its size live: an extend replaces it with the newest reading. So a run spanning a month drew today's size into every day of it, and an idle index growing on a write-heavy collection, the index this product most wants to drop, drew a flat line. Size cannot join the run's identity, because it moves on every insert, and a run keyed on it would fold nothing. So the collect keeps its own number per day. The day is truncated in SQL, in the database's own zone, which is the zone the chart's buckets are cut in.
• A day from before the table is still derived, and nothing is backfilled. The daily sizes were never stored, so there is nothing to backfill from, and those days keep the flat line. The derivation keeps its two rules:
  • Bucketed in SQL, one point per day. The rows behind it are run-length encoded, so their count is a function of how much the cluster changes — a busy 200-index cluster writes roughly four runs per index per day, which is 24,000 rows for a 30-day window and a chart that draws 31 points. Same discipline as §14's bounded reads, applied before the payload exists.
  • A day is joined by OVERLAP, not by an instant. "The run covering midnight" looks equivalent and drops exactly the indexes whose counters move: a busy index's run ends at one collect and the next begins at the following one, so midnight falls in the hole between them. A day counts each index once, at the newest run that overlapped it. At most one run can be picked per index per ordering step, which the exclusion constraint on span (§11.2) guarantees.
• A day nothing measured is null, never zero. Zero is a claim about the cluster ("it had no indexes"); null is a claim about us ("nobody looked"). The chart breaks its line at nulls, so a week of outage renders as a gap rather than a flat line asserting the footprint held steady through it. A collect that saw no index at all writes no row, just as it writes no snapshot, because an empty listing says more about what the credentials could see than about the cluster.
• The summary reads the DRAWABLE ends. firstBytes/latestBytes are the first and last non-null points, not points[0] and points.at(-1). Reading a null end as zero would report a cluster whose collection stopped on Tuesday as having shed its entire index footprint — the most alarming number this product could produce, and entirely our own doing. changeBytes stays null until two different days have been collected, because one point is a measurement and not a trend.

indexCount rides along on each point: a footprint that fell because 40 indexes became 30 is a different event from one that fell because 40 indexes got smaller, and the sum alone cannot tell them apart. A recorded day's count is what its last collect saw. A derived day's is every index with a run overlapping it. A day's total and count always come from the same source.

§8.1 answers "is the footprint going down". It cannot answer the question a reader asks the moment they are shown a proposal: what else is on this collection?

Every index a cluster has was already stored, with its spec, its size and its per-member counters — cluster_indexes holds the identity and the shape, index_snapshots the measurements (§11.2). None of it reached a screen. The only index-level payload the dashboard had was IndexUsage, and D66 keyed that by recommendationId because it was attached to the recommendations table. So an index nobody had proposed anything about had no size, no counters and no key pattern anywhere in the product: the only indexes a customer could see were the ones we already wanted to change, and everything the engine judged fine was invisible along with the judgement.

getClusterIndexes (GET /clusters/{id}/indexes) is that population, and it needed no collector work and no new storage.

One row per index the LATEST collect confirmed. The filter is last_seen_at = (select max(last_seen_at) …), which is the same idiom latestIndexFootprint uses and for the same reason: a collect stamps every run it touches with one now, and an idle index has its existing run extended rather than a new one written, so filtering on captured_at would drop exactly the indexes that have not changed in weeks. It also settles the rebuild case for free — cluster_indexes is keyed by spec digest, so a rebuilt index has two dimension rows, and only the live one's run reaches that timestamp.

The comparison stays in SQL. Postgres keeps microseconds on a timestamptz and a JS Date is milliseconds, so a max(last_seen_at) that has been through the application no longer compares equal to the rows it came from. The endpoint reads it separately anyway, because the page needs an "as of" — but as a display value only, and the repository says so where somebody would otherwise reuse it.

Offset paging, not a cap and no longer a keyset. listRecommendations caps at the top N with an honest total (D47) because proposals are ranked — the top ones are the answer. An inventory is browsed instead: "what else is on orders" has no top, so the sort is (database, collection, index_name), which is a total order because one cluster can hold two indexes of the same name in two collections. The optional database/collection filters are that same order's scoping, and total narrows with them rather than describing rows the reader did not ask for.

It was a keyset cursor until Sep 2026, and the swap is a trade rather than a simplification (D133). Keyset can only step, so "browsed" was being used to justify a control that cannot browse: a Back button, a More button, no page number, and a reader who wants the last of six pages clicking More five times. What offset gives up is what keyset had for free — the set moves under the reader, a collect lands, and a page boundary can then repeat or skip the row that crossed it.

Survivable here, and not in the security trail D67 argued for, which is the whole case rather than an exception:

Two things keep the exposure small and both are deliberate: total is re-counted per request rather than carried in a cursor, so the page count follows a set that changed size mid-browse, and the collect cadence is hours against a reader who pages in seconds.

Past the end it clamps, to the page boundary. A reader who narrows the filter while on page five has asked for an offset that no longer exists. An empty table under a "517 indexes" heading reads as a broken screen rather than as a moved boundary — and a clamp to the raw offset would serve a window straddling two pages, so rows would appear twice. Where it landed comes back as offset/limit beside total, so the control follows the rows instead of insisting on a page that is gone.

The request is the cache key, and the previous page stays on screen while the next is out (D139). Each page, size, order and search is its own entry — ["cluster-indexes", id, page] with the request object whole (§14.2) — so stepping back to a page already seen is a cache hit, and the route's loader warms firstPage() of the same view constant the component reads, or its warm-up would land in an entry nobody reads and the tab would outline on every SSR. A new entry starts empty, which would make every click and every keystroke in the search box swap the table for skeleton rows (and the table disables the box while pending, so a reader could type one letter): the paged reads keep the page before the one asked for as placeholder data (keepPreviousData), the hook says so (PagedRead.placeholder), the footer lights the REQUESTED page until the served one arrives, and the body dims rather than outlines. This all dates from #455, which found that offset paging had never worked in a shipped build: the key was the last of three hand copies of the request, and every suite was green.

The sort and the filter are the api's too, because the paging is (D135). Paging on the server and sorting in the browser cannot both be true at once: the server chooses which hundred rows arrive, so a header that reorders those hundred makes "sort by size descending" mean the biggest of an arbitrary hundred. The rule across the whole dashboard is one line — whoever chooses the rows chooses the order and the filter — and which side that is depends on whether the read is capped or paged:

A cap is what makes the browser's version honest: the rows in hand are the whole set the reader was handed, and "500 of 4,000" says the rest exists.

sort is a closed enum rather than a column name, because the value reaches an ORDER BY — the repository maps each key to an expression in a record that is total over the enum, so ?sort=1;drop table x is a 400 from zod rather than a question about escaping. Each enum value is also the dashboard column's id, so nothing translates between the header and the query.

Some columns lose sorting as a result, and that is the honest half: the key pattern, the flag count and the proposal are computed after the read — out of the spec jsonb, per engine in the dashboard, and from a lookup scoped to the page — so the database cannot order the cluster by them. They say enableSorting: false instead of sorting one page and looking like they sorted the cluster. totalOps is kept at a cost: the per-member split IS the stored value (D66), so ordering by usage is a per-row jsonb aggregate and unindexable by nature.

The filter changed meaning with the move — narrower in scope, wider in reach. It matched any rendered cell of the page; it now matches database.collection and the index name across the whole cluster, and the placeholder says which. % and _ are escaped, since ilike escapes nothing itself and a reader searching for the index id_1 meant id_1.

The per-node split is the recommendations table's, verbatim. usageSplit (apps/web/src/components/app/index-usage.ts) was widened to a structural input so both payloads go through it, rather than the blind-spot rule being written a second time. That rule is the whole reason the module exists: a per-node number that silently omits an unreachable secondary is worse than the total it replaced, so a member the collect did not reach is named in the tooltip and never drawn as a zero. Each member's since — its counter start — rides in the same tooltip, because $indexStats counters are cumulative from that instant and "0 ops" on a member that restarted an hour ago is not idleness (D114).

Flags are drawn only when set, and worded for the engine that set them. The stored spec is one fixed set of booleans because the analysis core reasons about every engine in one vocabulary (§9.1); a screen has the opposite problem. ttl and sparse are MongoDB-only — both relational collectors hardcode them false — hidden is never set on PostgreSQL (D106), include exists only on SQL Server, and isShardKey means a shard key, a primary key and a clustered index on the three engines. A column of "no"s would be a MongoDB vocabulary with two engines' blanks in it, so the page renders badges for what is true and packages/contracts/src/index-flags.ts decides the word, in the same slot and for the same reason as engine-capabilities.ts: the browser cannot import an adapter, and what this file settles is what a sentence says, never what the pipeline does.

hinted sits with those flags. It is collected on every snapshot and had never been displayed, and it is load-bearing: an index named by a hint() cannot be hidden — mongod rejects the hint — so the engine refuses to observe or re-order it, and until now the only evidence of that decision was a recommendation that never appeared.

The last column is whether a live recommendation points at this index. It matches on the recommendation's own index_name and on target_spec.retire, because a REORDER or a narrowing UPDATE names the index it would create while retiring an existing one — and retiring an index is a decision about the index being retired.

§8.2 is the drop side's inventory. This is the create side's, and unlike §8.2 it needed a storage decision: nothing about the workload was persisted at all.

collector.collectWorkload has always returned every scanning query shape with collscan, sortedInMemory, count, docsExamined and observedForHours. jobs/suggest.ts read them once an hour, used them in memory, and persisted only the recommendations that cleared every create-side gate:

Every one of them is right. Each worked by making a finding disappear along with the proposal, and the WORKLOAD note only ever recorded budget — so a query walking 900k documents a week on a small collection was seen, priced, discarded, and never mentioned. That is the same defect D99 (#277) fixed on the drop side, on the other half of the engine.

getClusterWorkload (GET /clusters/{id}/workload) is the surface, and workload_shapes is the table behind it.

index_snapshots is run-length encoded because an idle index reports byte-identical counters collect after collect (D39). That reasoning does not transfer. A shape worth reporting is one that keeps running, so its counters move on every pass, and a run keyed on the counter would collapse nothing in exactly the case worth collapsing.

What the page actually asks is "was this scanning last week too, or did it start on Tuesday" — which is two timestamps. So a row carries first_seen_at, the newest reading, and an observations count, and is upserted in place. Storage is then a function of how many distinct shapes the cluster has, which is the same property run-length encoding buys for the snapshot series by a longer route.

Only scanning shapes are stored. A shape the planner served from an index is not a finding, and storing it would make the table the size of the workload rather than the size of the problem.

The measurement columns are replaced on each pass rather than accumulated: $queryStats counts from the store's own start and the profiler's ring reports what it still holds, so the counters are already cumulative and adding ours on top would double-count (D26). Two columns are computed by the pass and recorded rather than re-derived on read — weekly_docs_examined and severity — because both need the collection's document count as a fallback when the source reports no docsExamined, and that number is not a fact about the shape.

This is the trust decision the feature turned on. constants is the one field on a QueryShape carrying real customer values: only the profiler populates it, since $queryStats shapifies literals away, and the profiler is the workload source below MongoDB 8.0 — which D94 shipped on by default, accepting the trust cost for a read that was transient.

Persisting the same values into our control-plane postgres is a different question with a different answer. The page has no use for a literal — it shows the shape an index would have to cover — so the values go on being used in memory to derive a partial-index candidate — where the engine can build one from them (§9.2, partialIndexFromConstants) — and are never written. storedShapeOf constructs the stored object field by field rather than spreading the shape, which is what makes that structural rather than a filter somebody could forget, and an integration assertion queries on literals and checks they are absent from the stored jsonb.

The page's whole claim is "this is the gate that declined it", and that can only be true if the pipeline says so.

CreateCandidate, SortOrderAdvisory and ReorderCandidate now carry sourceShapes — every shape they answer, including the ones folded in by consolidation and by two shapes wanting the same index — and recommendCreates takes an optional decline sink that names not-recurring, ad-hoc-client, index-exists or no-candidate as each check fires. Both additions are additive and optional, so no existing caller or test changed.

index-exists is any existing index that serves the want, not only one on exactly its fields in its order (#612, D187). Block for block, the index must carry:

• the want's equality fields first, in any order;
• then its sort keys in sequence, each in its direction or all reversed (a backward walk);
• then its range fields as a set, with any trailing keys after them.

It must also cover the same documents. The order-sensitive test let a shape that bound two equality fields the other way round propose the index built for it two hours earlier. Query Store keeps reporting a scan for as long as it remembers it, so for days after a build, the shape it was built for still reads as one. Consolidation reorders the equality block, so the final keys are checked once more.

jobs/suggest.ts then seeds a ShapeLedger with every scanning shape as it enters a collection and lets each producer overwrite with the gate that actually fired. proposed is sticky, and that exception is load-bearing: three producers read the same shapes and disagree by design — a re-order answers an in-memory sort that the create rule declines as already indexed — so a shape one of them acted on must not read as declined because another had nothing to say about it.

The alternative was for the writer to re-apply isRecurring, isWorthIndexing and the already-indexed test to work out why a shape produced nothing. That is a second copy of the rules, true only until it drifted, on the one page whose value is that it reports what the engine actually did.

The document floor and the size ceiling fire before the workload is read: collectWorkload is only asked about the namespaces that clear them. Widening that read means a profiler dial per collection on every cluster below MongoDB 8.0 — collectSlowQueries is the per-namespace fallback — which is a real cost to pay for a row saying a 40-document collection was not analysed. So the pass counts those collections into its own analysis_notes.suppressed map, which is free-form by design and therefore costs no migration, and the page reports two numbers beside the table.

policy.workloadAnalysis === false is the third: suggestForCluster returns before it reads anything, so there is nothing to count either. The payload carries a boolean for it, because an empty table otherwise means both "nothing is scanning" and "nobody looked" — and that is the distinction D19 exists to protect.

Ranked by documents walked per week, descending, because this is a list of problems rather than an inventory: the worst are the answer. The sort key is coalesce(weekly_docs_examined, -1), which does two things — Postgres sorts nulls FIRST under desc, so the one row whose cost nobody measured would otherwise head a page ranked by cost, and making the key total is what offset paging needs (D133): the sort is (coalesce(...) desc, id), and under a partial order one row could appear on two pages of a single browse where the keyset cursor this replaced merely stalled. Same reasoning as orNull in the collections table.

A declinedOnly filter is the page's own question, expressed as outcome <> 'proposed' rather than a list of the declining outcomes, so a gate added later is declined by default — the safe direction, since the alternative is a new gate silently vanishing from the filter that exists to show it.

outcome is a text column, not an enum, for the reason clusters.blocked_reason is: adding a gate should be a constant rather than a migration, and an outcome written by a newer worker than the api reading it must render as itself with no explanation rather than fail the page it is on.

proposed_index names the shape's answer for standing as well as for proposed (#608): the live recommendation it was not proposed twice for. That is how a build finds its shapes once approval has taken it out of PROPOSED, because the ledger is rewritten every pass, and by the change window the shape reads standing.

Each index the engine built is shown with two measurements, both taken from data the collector already writes:

• Reads on its collection, over the week before the build against the week after it, from latency_samples. The comparison sums every interval between consecutive readings that lies wholly inside a window and divides once, so a busy hour outweighs a quiet one. An interval straddling the build belongs to neither side, and one spanning a counter reset is skipped as it is on the charts. A week is used because traffic has a weekly shape. Below MIN_READS_TO_COMPARE (1,000) on either side the dashboard says too few to compare and draws no percentage: a dev cluster measured 157 reads before a build and 29 after, and the averages moved by half. These are the whole collection's reads, so a build that fixed one query of many moves them only by that query's share. On the hosted deployment, userId_1 took one collection's reads from 0.90 ms to 0.31 ms over about 90k reads a week on each side.
• The scanning it was for. At the build, served_shape_digests and baseline_weekly_docs_examined record the shapes whose proposed_index is the index, seen in the last week, and the documents they examined a week. A shape the index serves is no longer a scanning shape, so the ledger stops writing it. The shapes still being written are the ones the build did not fix, and that is the only "after" there is. The answer waits a day after the build (SCANS_SETTLE_MS), because the profiler is a ring and still holds scans from before the build, and until the workload has been read end to end after that day (the WORKLOAD analysis note). Until then it is shown as not yet checked, never as none: not having looked is not having found nothing. Builds made before #608 have no record and show reads only.

The decision core never touches a database driver. Everything engine-specific sits behind two ports plus a session factory (apps/api/src/engine/ports.ts):

• IndexCollector — the read-only statistics surface (usage, sizes, latency, workload shapes, delete patterns). Deliberately contains nothing that can read customer data rows. Every method is per namespace except collectWorkload, which takes them all at once: each engine's workload source is one cluster-wide store you filter per namespace ($queryStats, pg_stat_statements), so a per-collection signature invites reading the whole store once per collection.

• IndexExecutor — the only write surface (hide/unhide/drop/create); implementations must enforce read-only mode structurally.

• EngineSession — a pooled connection exposing both, plus listDatabaseNames() (user databases only — the databases the engine itself owns are excluded BY NAME in each adapter, which is the one rule all three follow; PostgreSQL's postgres is not one of them and is listed only when it holds a user table) and ping() (rotation's verification probe). A cluster whose owner narrowed the observe selection is leased through ObservedSession (src/engine/observe.ts), which filters that one method to the selection. The filter is in the LEASE and not in the pool: connection-pool.ts keys entries by cluster and reuses one session across jobs for up to five minutes, dooming it only when the connection string changes, so a filter built into the pooled session would keep enforcing whichever selection was current when it was dialled. Wrapping one method also means the two callers that walk databases — the collect and the suggest pass — cannot be joined by a third that forgets to filter.

  That filter covers walking, not acting. A pass that already holds a database NAME never calls listDatabaseNames(), so the lease cannot help it: apply and finalize both act on rec.database and each has to check the selection itself. apply refuses to hide outside it; finalize un-hides and returns the row to PROPOSED (#541). Narrowing the selection deletes the open proposals outside it but deliberately NOT the hidden ones — a hidden row is the only record that an index was altered and needs restoring, and deleting it would strand a hidden index with nothing left to un-hide it, which is silent and worse than the drop it was meant to prevent.

• EngineAdapter — connection-string validation (the SSRF guard) + open(), registered per engine in src/engine/registry.ts. diagnose() and provisionScopedUser() both take the observe selection: the first because the per-database privilege requirements are evaluated over the databases in scope (a role covering one database of twelve is a gap against the cluster and a grant against the selection), the second because SQL Server grants per database and a narrowed selection makes the provisioned login narrower than the instance. The two answers to a stale selection differ on purpose and the rule lives in one place (scopeForDiagnosis, src/engine/observe.ts): a diagnosis whose every selected database has been dropped falls back to reporting on the whole cluster, because "your role is missing everything" is the wrong sentence for "the databases you picked are gone", while a collect in the same state walks nothing at all — falling back there would read exactly the databases the selection exists to exclude. The clusters.engine column picks the adapter per cluster; adding an engine is one new adapter directory plus one registry line, with zero pipeline changes.

Four more things are engine-neutral and sit beside the ports rather than inside an adapter: engine/tls.ts (the ALLOW_INSECURE_CLUSTER_TLS escape hatch and InsecureConnectionError), engine/version.ts (the ALLOW_UNTESTED_DATABASE_VERSION reader and UnsupportedServerError), engine/provision.ts (the fixed scoped-user name, ProvisionDeniedError and the already-provisioned wording) and engine/observe.ts. All but the last lived in src/mongo/ until #329, for no better reason than MongoDB being the first adapter written.

The vocabulary those signatures are written in — IndexSpec, QueryShape, ServerHealth and the shapes they are built from — is src/engine/types.ts, next to the ports rather than under src/analysis. That is what lets an adapter be read without reading the layer that consumes it, and it is the direction the dependency has to run: engine/ may not import from analysis/.

The analysis core (src/analysis) is pure and engine-neutral; the jobs, the connection pool, and the API speak only the ports. MongoDB (src/mongo/adapter.ts) is the reference adapter; SQL Server (src/mssql/adapter.ts) is the second, and proved the ports honest — it shipped as one directory plus one registry line, with the only core change being a new counter-reset rule every engine benefits from (§9.3). Vocabulary stays MongoDB-flavored ("database", "collection") — relational adapters map their terms (a SQL Server "collection" is schema.table) rather than the codebase adopting a lowest-common-denominator vocabulary. The engine of a pasted connection string is detected from its scheme (src/engine/registry.ts, detectEngine): mongodb:// vs mssql:// vs ADO Server=… are disjoint, so the connect form needs no engine picker.

Why "once per database" keeps appearing against SQL Server. #454 batched the two Query Store reads because they were server WORK: each was a whole-store scan per table, ~830 a collect. #461 batched the three CATALOG reads — sys.indexes, sys.dm_db_index_usage_stats, sys.dm_db_partition_stats — for a different reason: they are cheap on a local socket and they are the entire budget through a tunnel, because what a tunnel charges for is round trips. The hosted deployment's Prod MSSQL cluster is 13 databases and 362 tables reached over WireGuard; three statements a table is 1,086 a collect, and that collect stopped fitting in its five-minute budget and collected nothing for 19 hours. The MSSQL Dev cluster is larger by index count and fine — it is dialled directly.

Suggest kept one storage statement per table until #614. It was 105–150 s of a 175 s production suggest, at 416 tables, and every second of it kept the control plane's database awake on a plan that bills by the hour awake.

This did not make the tunnelled collect fit (D141 revised, D143). The statement count fell 27× and the pass still timed out; the fleet's own comparison then showed why — the DIRECT MSSQL cluster of nearly the same size (12 databases, 334 tables) collects on every tick where the tunnelled one never does. So the cost is the tunnel and not the round trips, and "a collect costs its round trips" is true of the reads this removed and too narrow as a general claim. Which of the two it is now measurable rather than arguable: see the phase report above.

All five are optional port members, SQL Server's alone: collectSnapshots and the probe use them where they exist and the per-collection reads where they do not. Finalize too, since #629. It had kept the per-table reads, and on production that was 43 whole-store scans of plan XML through the tunnel every hour: one per build under the write watch, one per hidden drop. That made a 70-minute pass, and on 10-08 a request the tunnel reset. Now the write watch reads a collection's live totals only for the cumulative check, through latencyByCollection, once per database. SQL Server's failed-operations read attributes the table's plans from the same cache and sums their Exception executions from one numeric read. The probe reads latency through a bounded form, latencyByCollectionWithin(database, 250) (#588, D146): it answers when at most 250 plans are unseen and otherwise gives the database up to the collect — keeping what it shipped — because a cold store is the collect's to warm and the probe is the five-minute signal. MongoDB implements none, because listIndexes, $indexStats and $collStats are per-collection commands with no database-wide form and each is one cheap round trip anyway. The maps are keyed by schema.table — the string listCollectionNames returns — since an index name is unique per table and not per database, and the live suite holds each batched answer to the per-table answer for every table the fixture database has. D141. | Partial index from a shape's constants | the recommender's {field: literal} map is a partialFilterExpression — proposed and built | filtered index; the executor builds one only from the {definition} T-SQL read off sys.indexes, so partialIndexFromConstants: false and the constants stay keys (#452) | WHERE; the executor builds one only from the {sql} its collector read, and an unrecognised shape yields a full index — so false, and the constants stay keys |

EngineCapabilities marks where engines genuinely differ, so feature gates check a flag instead of assuming MongoDB semantics deep in the pipeline:

• hideIndexes — the safety pipeline's observe stage rides on cheap, reversible invisibility. SQL Server's is DISABLE, with the asymmetry and the guarded index classes in §9.3. PostgreSQL reports false, and the pipeline honours it (#303): apply.ts moves the recommendation into the observe window without touching the index, records no read-latency baseline — the gate asks whether hiding slowed reads, and an index still serving every query cannot answer — and the evidence for the drop becomes the usage counters staying flat, which preflightDrop re-checks. The drop is therefore irreversible at the moment it runs, and undo is a scripted CREATE INDEX CONCURRENTLY from the recorded spec: exact, but minutes rather than instant.
• provisionScopedUsers — admin-string onboarding (§10.1). Each engine's provisioning is inherently engine-specific commands/SQL. True on all three now, but what it provisions is not equivalent: MongoDB and SQL Server get a role that manages indexes and cannot read data, while PostgreSQL's is analysis-only. That is not an unfinished piece — the engine has no grantable index privilege at all, so a role that could apply could also read, and applying takes the owner's own connection string instead. §9.4.
• partialIndexFromConstants — whether the recommender may turn a shape's constant equality predicates into a partial-index candidate (…_partial). The {field: literal} map it derives is MongoDB's partialFilterExpression as it stands, so MongoDB is true. The SQL engines build a partial index only from the predicate their own collector read back ({definition} on SQL Server, {sql} on PostgreSQL) — exact for a restore or a re-order, and nothing they can do with the map — so both are false and the recommender keeps those columns as index keys, the candidate a shape without constants produces anyway. Gates the suggestion, not the build. Read through openClusterSession's canPartial, like canHide. Before the flag existed the SQL Server executor refused the map as an unsupported server version and blocked a production cluster as "wrong major" (#452, D137); a refusal is now IndexBuildRefusedError, which jobs/create.ts records against the row (REJECTED, refused: … in its history) before building the next one. Translating the map into a WHERE is a feature with its own questions — the predicate's types must match the column's for the optimizer to use a filtered index, and a parameterised plan never picks one — and is not done.
• A build the table already has is not built (#612, D187). Before each build, jobs/create.ts reads the table's live index list. A taken name, or an index that already serves every query the build would (isCoveredBy: classify's own isRedundantPrefix, or the same keys on options and columns it covers), closes the row REJECTED, with not built: … in its rationale and its history, and the pass goes on to the next build. The engines each made the collision worse in their own way:
  • SQL Server failed with Msg 1913, which the pass did not handle per row. On the hosted deployment that stopped every approved build on the cluster for days. Msg 1913 is now a refusal of the row, since a statistics object can hold the name and no index list shows it.
  • MongoDB took an identical index as a silent success, and the row was recorded as a build Indexterity had made.
  • PostgreSQL index names are unique per schema, and both build routes say IF NOT EXISTS. On a name another table's index held, the build did nothing, reported success, and a rollback would have dropped the other index. The executor now refuses a name the schema already holds. Since #617 a proposal is named per engine (EngineAdapter.indexName), and a PostgreSQL one carries its table, so two tables wanting the same keys no longer want the same name.

Everything below was measured on live servers — 17.11 and 18.6, plus 14.24 and 16.15 for the version rules — before the adapter was written (#35), and twice the measurement contradicted what the documentation implied.

Hiding is impossible, not merely awkward. hypopg_hide_index() exists and does not do the job: with an index hidden, plain EXPLAIN reports a Seq Scan while EXPLAIN (ANALYZE) and ordinary queries in the same session keep using the index and keep incrementing idx_scan. It is gated on the non-executing EXPLAIN path, which makes it a counterfactual planner oracle — genuinely useful for proving a candidate index before building it — and no kind of hide.

Clearing pg_index.indisvalid is a real reversible hide: the planner ignores the index, real execution avoids it, the invalidation reaches sessions that are already open, and DML keeps maintaining it because indisready stays true, so un-hiding is one catalog UPDATE with no rebuild. It is also superuser-only and undelegatable — GRANT UPDATE ON pg_index does not help — so this product does not use it. Three edges recorded for whoever reconsiders:

• REINDEX INDEX silently sets indisvalid back to true, so a routine reindexdb would end an observe window with nobody told.
• pg_dump -s omits an invalid index entirely, so a dump/restore during the window loses it.
• a hidden index is indistinguishable from a broken one except by indisready: a failed CREATE UNIQUE INDEX CONCURRENTLY leaves indisvalid=f, indisready=f, while the hide would be indisvalid=f, indisready=t.

Only a table's owner may alter its indexes. GRANT ALL PRIVILEGES ON ALL TABLES answers must be owner of table …, and so does pg 17's new MAINTAIN privilege. Since an owner can always SELECT, index DDL implies read access — which is why the provisioned role is analysis-only and diagnose's APPLY tier reports "cannot apply" as the intended shape for a read-only cluster rather than as a fault.

Unless pg_cron is installed, which is the second apply route (#332) and the one that keeps the promise. CREATE INDEX CONCURRENTLY is banned from every PL context, so a plain SECURITY DEFINER function cannot issue it, and dblink escapes the function context only to deadlock against its own caller — CIC waits out every transaction that can see the table, including the one blocked reading the dblink result. A pg_cron job runs its command in a background worker outside any transaction block, where CIC is accepted, and a job scheduled from inside a SECURITY DEFINER function runs as the definer. So the scoped role calls the function, the build runs as the table owner, and no credential is stored anywhere — which is what makes this a better trade than the dblink variant, whose function body has to carry the owner's password in world-readable pg_proc.

Four measured facts shape it, all on 18.6 with pg_cron 1.6:

• One statement per job. A multi-statement pg_cron command is wrapped in a transaction and CIC is refused inside it. So a job cannot unschedule itself, and settling is a later tick's business — which is what makes a build here asynchronous, and why BUILDING finally has a writer.
• The function lives in the cron database. CREATE EXTENSION pg_cron installs schema cron in exactly one database and it exists in no other, so the scoped role dials that one and names its target as an argument. SHOW cron.database_name finds it and needs no privilege.
• The job history alone lies. The scheduled statement carries IF NOT EXISTS, so once a failed build has left its invalid carcass every later run reports succeeded. The catalog alone cannot tell a running build from a broken one either — both read indisvalid=false. Settling reads both.
• shared_preload_libraries means a restart. That is the real cost, and the only one: superuser, once, and nothing after it needs one.

The scoped role gains EXECUTE on three functions and nothing else — it still answers permission denied for schema cron if it tries to schedule anything itself, and still cannot read a table.

Statistics survive a clean restart and not a crash. A crash discards every cumulative counter and sets pg_stat_database.stats_reset to NULL rather than restamping it, so null is ambiguous and pg_postmaster_start_time() is the floor under since. REINDEX does not reset idx_scan, unlike a SQL Server rebuild — one fewer trigger for the ops-went-backwards rule.

One pool per database. SELECT … FROM other.public.t is a parse error ("cross-database references are not implemented"), not a permission one, so unlike SQL Server's single pool with three-part naming, each database needs its own dial.

CREATE INDEX CONCURRENTLY fails dirty. A failed run leaves an invalid index that the planner ignores while writes still maintain it, and nothing cleans it up — so the executor drops the carcass before rethrowing, guarded on indisvalid=false AND indisready=false so it cannot touch a healthy index of the same name. Both CREATE and DROP … CONCURRENTLY refuse to run inside a transaction block.

Everything below was measured on a live SQL Server 2022 (CU24) before the adapter was written — the numbers are from that probe (#36).

Hide is ALTER INDEX … DISABLE, and only for plain nonclustered rowstore indexes. DISABLE is a metadata-only flip (4 ms on a 500k-row index) that deallocates the b-tree: the observe window costs nothing — the disk is already freed and writes stop maintaining the index — where a hidden MongoDB index keeps paying for both. The undo is REBUILD, which costs a full index build (measured at parity with CREATE INDEX on the same data), ONLINE on Enterprise/Developer, Azure SQL and Managed Instance, offline under a schema-mod lock elsewhere. So the hide is instant and the undo is exact but not instant — acceptable because unhide is the rare path (a regression, an owner cancel, an offboard). The executor structurally refuses the classes for which DISABLE is destructive, all of which are isNeverDrop upstream anyway:

• a clustered index — the whole table goes offline (Msg 8655 on every query);
• a primary key — every referencing foreign key is silently disabled with it, and re-enabling is a rebuild plus per-FK WITH CHECK CHECK CONSTRAINT;
• any unique index — uniqueness stops being enforced while disabled (a duplicate insert succeeded in the probe), after which the re-enabling rebuild itself fails on the duplicates.

The counter-reset trap is real on current versions. ALTER INDEX REBUILD zeroes the index's row in sys.dm_db_index_usage_stats — verified on 2022 CU24 — and nothing since-like moves, because the service did not restart. Index-rebuild maintenance jobs are routine in SQL Server shops, so a busy index would read as dead the week after every rebuild. The fix is engine-neutral: countersRestartedDuring (src/analysis/classify.ts) now also distrusts any member whose cumulative ops decreased between two snapshots — a cumulative counter cannot shrink, so one that did restarted. since itself is sqlserver_start_time (restarts wipe the whole DMV; an absent row is zero-since-start), converted to UTC server-side.

Query Store is the durable signal. It survives restarts (verified), which the plan cache (sys.dm_exec_query_stats) does not, so per-table read/write latency comes from Query Store runtime stats: executions and average duration summed over every plan whose XML names the table (Schema="[s]" Table="[t]" — the attribute adjacency is stable and probed), split read/write on the plan's StatementType. Two honest caveats the gates already tolerate: size-based cleanup can retire old intervals, and a hard crash loses the last unflushed interval (15 minutes by default). Where Query Store is off, latency reads as zeros — the regression gates then answer UNOBSERVABLE rather than guessing — and diagnose reports it as the WORKLOAD-tier check with the ALTER DATABASE … SET QUERY_STORE = ON remedy.

Connection strings are accepted in both shapes customers paste: URL (mssql://user:pass@host:1433/db?encrypt=strict, also sqlserver://) and ADO (Server=host,1433;Database=db;User Id=u;Password=p), parsed by the adapter's own parser so the network guard vets exactly the host the driver dials. TLS maps narrower than MongoDB's three checkboxes: insecure ⇒ Encrypt=false, allowInvalidCertificates ⇒ TrustServerCertificate=true, and allowInvalidHostnames alone is refused with the box to tick instead — the driver validates hostnames as part of certificate validation and cannot skip one without the other. Encryption defaults ON when the string says nothing.

Versions: floor 2016 (13.x — Query Store arrives there; older is past end-of-life), tested ceiling 2025 (17.x). Both 2022 (16.0 CU24) and 2025 (17.0.4075) were probed live and behave identically on every load-bearing semantic — including the REBUILD counter reset, which 2025 still performs. 2025's new index types (VECTOR = type 8, JSON = type 9) fall outside the rowstore filter and stay invisible to the pipeline by design. Newer than 2025 refuses with ALLOW_UNTESTED_DATABASE_VERSION=true as the operator escape hatch — one flag for every engine, read in engine/version.ts; CI runs the adapter suite against both supported series so the ceiling stays tested rather than trusted.

Workload shapes (#201) come from Query Store PLANS, not statement text: showplan XML is machine-emitted with a structured grammar — Compare/CompareOp for predicates, SeekPredicateNew/SeekKeys for the equality-prefix/range split, OrderByColumn for in-memory sorts, every ColumnReference naming its database, schema and table — so shapes fall out of a tree walk with no T-SQL parsing. Facts the implementation rests on, all observed live on 2022:

• Query Store groups by TEXT, so literal variants of one shape arrive as separate query_ids; shapes are merged by their extracted key. Trivial plans get auto-parameterized server-side, which erases literals — the constants enrichment is therefore best-effort, exactly like mongo's $queryStats.
• Plans embed the server's own <MissingIndexes> suggestion. It is folded in as an ordinary QueryShape carried by the plan's execution counts, so the workload engine's recurrence and cost gates apply to it unchanged — and the standalone missing-index DMVs (wiped by restarts and by index DDL) are never read at all.
• An index build is recorded as an INTERNAL "insert … select" plan — a full scan feeding a sort, which would read as a phantom missing-index signal for every index the pipeline itself builds. is_internal_query = 0 keeps the server's own work out of the workload.
• Query Store records a query in the database it RAN in, not the one it read; plan parsing filters object references by database so a cross-database plan cannot mis-attribute to a same-named local table. A workload issued entirely from another database's context is visible when THAT database is analyzed.
• New databases default to QUERY_CAPTURE_MODE = AUTO, which skips insignificant one-off statements — aligned with the recurrence gates, and index DDL on a table can additionally remove its auto-parameterized trivial queries from the store (observed). Rare-but-recurring queries appear once they repeat.

Shipped since v1: scoped-login provisioning (#203), the server-health probe (#205) and the purge advisory (#206) — the three follow-ups this section used to list. Indexes the b-tree model does not describe (columnstore, XML, spatial, full-text, hypothetical) remain invisible to the pipeline rather than mis-modelled.

Server health (#205) has no single command behind it — there is no serverStatus here — so it is assembled from sys.dm_os_performance_counters (cumulative work counters) and sys.dm_os_waiting_tasks (the instantaneous queue), both covered by the VIEW SERVER STATE the scoped login already holds. Two of the counters an obvious reading would reach for do not survive a live server: Sort Warnings/sec does not exist on Linux SQL Server, and Range Scans/sec moves for an ordinary singleton seek (500 seeks moved it by exactly 500), so it is not a scanning signal. What works is Page lookups / Index Searches as the docs-walked-per-index-key analogue — measured 3.01 on a seeking workload, which is the b-tree descent and therefore flat as a table grows, against 66.6 when full scans dominated the same table. The spill signal is Workfiles Created/sec. The queue is classified, not counted: the waiting-task DMV holds 33 background threads on an idle container, so COUNT(*) would report a healthy server as permanently critical; filtering to LCK_M_S/LCK_M_IS/PAGEIOLATCH_SH reads 0 idle and 3 under three deliberately blocked readers. Thresholds are therefore per engine — docs-per-key 100/1000 becomes 20/200, sorts 250/2500 becomes 20/500 (a tempdb spill is strictly worse and rarer than an in-memory sort, so the bar goes DOWN), and the scan floor doubles because Full Scans/sec counts a three-row lookup table identically. Two traps: the counters are served from a snapshot that refreshes on its own schedule, so two reads inside one tick return identical values for Index Searches, Page lookups and Range Scans while Full Scans moves eagerly — the probe's five-second gap is now load-bearing rather than a convenience. And the counters are matched on counter_name alone, because a NAMED instance publishes them under MSSQL$<name>:Access Methods and filtering on the object name would read every counter as zero on exactly the deployments most likely to be real.

The purge advisory (#206) keeps mongo's SIGNAL and changes its RECOMMENDATION, because SQL Server has no TTL index to recommend. A recurring age-based DELETE with no index leading with the date predicate scans the table and holds locks for the whole pass, which is the job that makes a table unavailable at night — so the advisory recommends the supporting index, and above ten million rows names the partitioned sliding window as the better answer. It is advisory on both engines for opposite reasons: mongo's because a TTL index deletes data and the engine never builds one, this one because partitioning is a schema change no index tool should make on its own. Extraction needs TWO plan shapes, both captured live: an unindexed purge puts the predicate in <Compare CompareOp="LT">, and an indexed one moves it into <EndRange ScanType="LT"> with RangeColumns/RangeExpressions — reading only the first would make the pattern vanish the moment somebody added the index, which is precisely when the table is big enough for the partitioning half to matter. Retention is an ENRICHMENT: of three purge dialects measured, the parameterised WHERE created_at < @cutoff — what an ORM or a stored procedure sends — carries no cutoff at all, so medianRetentionSeconds is nullable and the advisory says the window is not visible rather than rendering a missing number as "≈ 1 day".

Availability Groups. Every replica the group names is a member, the same way a replica set's are (D81). sys.dm_db_index_usage_stats counts what its own instance served, so a readable secondary's reporting traffic is invisible from the primary — measured on a two-node group: three reads run on the secondary report user_seeks = 3 there and 0 on the primary. Discovery reads sys.availability_replicas and NOT a join to sys.dm_hadr_availability_replica_states, because on a secondary that DMV holds only the local replica's row and the join would discover nothing; each replica reports its own role from its own connection instead. Members are dialled at their read_only_routing_url with ApplicationIntent=ReadOnly — a replica set to ALLOW_CONNECTIONS = READ_ONLY refuses a plain connection with Msg 978 even for a three-part read from another database. A replica that allows no connections, or that has no routing URL (its instance name is not an address), is named in the roster as refused rather than guessed at.

INCLUDE columns are read (sys.index_columns reports them as key_ordinal 0 / is_included_column 1, ordered by index_column_id, which is the order they were declared in) and carried on IndexSpec.include — through the stored snapshot, the undo's rebuildOptions, and the re-order carry-over — so a covering index is never proposed for a drop against one that covers less, and never rebuilt as something narrower than it was. §6.4 has the rule. What is still not modelled: two indexes with the SAME keys where one's includes are a superset of the other's, which is legal, wasteful, and a finding of its own rather than a guard on this one.

The plan parse yields. A suggest pass reads up to 5,000 Query Store plans per database and parses each one's showplan XML. That is ~1.6 s of CPU, and until #230 it was 1.6 s of blocked event loop — 1652 ms measured as the lateness of a timer due every 10 ms, which is what an in-flight HTTP request waits, and the pipeline now runs in the process that serves HTTP. Both plan passes hand the loop back every 100 rows (mssql/chunk.ts), taking the worst stall to 28 ms; chunk 500 still stalled 147 ms and chunk 25 bought only 16 ms more. This is interleaving, not parallelism: the CPU cost is unchanged, it simply stops being a window in which nothing else is served. A unit test parses 5,000 plans and fails if the worst stall passes 250 ms, because losing that yield breaks no other assertion in the suite.

The data plane connects with a custom role granting only index management and stats, excluding find / insert / update / remove on customer collections. It therefore cannot read customer documents — enforced by the server, not by promise.

Automated (admin-string onboarding). POST /clusters/provision accepts an admin connection string, uses it ONCE to create the role + a dedicated user (indexterity) on the customer cluster, verifies the scoped credentials authenticate, and stores only the scoped string (sealed, §10.3). The admin string is never persisted. Re-provisioning refreshes the role's privileges via updateRole, so app updates can evolve the grant. The exact role (apps/api/src/mongo/provision.ts, live-verified against an --auth mongod — every engine command allowed, every document access denied):

db.getSiblingDB("admin").createRole({
  role: "indexterityEngine",
  privileges: [
    // Un-transformed $queryStats needs BOTH actions (verified on mongo 8).
    // serverStatus is the health probe — server-wide, and the one grant here
    // that reads beyond index metadata (the Connecting a cluster page explains what).
    { resource: { cluster: true },
      actions: ["listDatabases", "serverStatus",
                "queryStatsRead", "queryStatsReadTransformed"] },
    { resource: { db: "", collection: "" }, // all non-system collections, all dbs
      actions: [
        "listCollections", "listIndexes",     // discover
        "indexStats", "collStats",            // usage + size + latency → ROI
        "createIndex", "dropIndex", "collMod", // act (collMod = hide before drop)
        "enableProfiler"                      // 0.29.0: watch a hidden index's failures (D183)
      ] },
    // The only find grants are metadata namespaces:
    { resource: { db: "", collection: "system.profile" }, actions: ["find"] },
    { resource: { db: "config", collection: "collections" }, actions: ["find"] }
  ],
  roles: []
})

Where direct role creation is not possible (Atlas manages users through its own UI/API; the endpoint maps that failure to a 422 with guidance), create the same role there manually and connect with the scoped string instead.

The role is versioned (D184, #599). mongo/role.ts holds the base role and ROLE_CHANGES: every privilege a release added since, with its revision, the release that asked for it, and the diagnose check that reports it. ENGINE_PRIVILEGES is derived from the two, so a privilege has no other way in. Each cluster stores two numbers along that list, both raised and never lowered:

• privileges_revision: how far its credentials are known to hold.
• privileges_checked_revision: how far they have been checked. The collect does the check over the session's open connection (EngineSession.checkPrivileges), on any cluster with something not yet held.

Only a change past the first and within the second, known missing, drives a banner, a section on the connection card, one mail per revision, a runnable grantPrivilegesToRole for a role of ours, and an upgrade that takes an admin string once. A change nobody has checked is not shown: broad built-in roles such as Atlas's atlasAdmin already hold most of what a release adds. Two tests keep it honest:

• role.test.ts: every change is a WORKLOAD check;
• registry.test.ts: each engine's CORE and APPLY keys are frozen, and an engine with changes can upgrade.

version:set reads the list off the syntax tree, refuses a change labelled with a later release than the one being cut, and prints what the release notes must name.

• createIndex builds the index under server authority, so no read grant is needed — the role never exposes document contents.
• Confidentiality vs availability. This role prevents data leakage, but dropIndex can still cause an outage and createIndex can spike load. The role does not replace the confirm / hide-first / windowed-build pipeline; both layers are required.
• system.profile is the trust boundary, and workload analysis is now on by default across it (#258). Profiler entries contain query predicates with literal data values; that read powers workload analysis and partial-index detection. It used to be opt-in and is not any more — policy.workloadAnalysis defaults ON, and turning it off is the choice a cluster records. The exposure is version-dependent rather than uniform: from mongo 8.0 the source is $queryStats, whose shapes are redacted, and only BELOW 8.0 does the profiler answer at all. Defaulting on for $queryStats alone and leaving the profiler fallback opt-in was considered and rejected — it would have made "on by default" true on modern servers and false on exactly the older ones with the most to gain from the create side. The cleanup path (drop/merge) is unaffected and still needs index metadata + $indexStats only → zero data exposure.

Onboarding takes a connection string and connects to it. That makes the api a request-forgery primitive unless two things are true.

The target must not be ours. engine/net-guard.ts resolves every host a string would dial — including SRV expansion, because on Atlas the seed domain is not what gets connected — and classifies each resulting address:

• FORBIDDEN, never dialed: link-local (169.254.0.0/16, which is the cloud metadata endpoint and never a database), multicast, reserved, unspecified, documentation and benchmarking ranges, IPv6 link-local.
• PRIVATE, dialed only when ALLOW_PRIVATE_CLUSTER_TARGETS=true: RFC1918, loopback, CGNAT, IPv6 ULA. Self-hosted installs need this; a hosted one must not set it.

IPv4-mapped IPv6 (::ffff:10.0.0.5) is unwrapped before classification — the classic bypass. Every host in a multi-host string is checked, so a public host cannot smuggle a private one alongside it. An unresolvable host is allowed through to fail as "unreachable", which is honest and leaks nothing.

A tunnelled cluster substitutes this guard rather than relaxing it (D110). assertTargetsAllowed resolves on our side, so for a name that exists only inside a customer's VPN it either fails or — worse — answers about a different host than the one about to be dialled. And relaxing it to admit PRIVATE would admit every tenant to our own network, which is the primitive this section closed.

Both routes live in the same file, behind one assertDialTargetsAllowed, so this section describes one place rather than two. What differs is only what a category means — the table of what an address is is shared:

The tunnelled route adds AllowedIPs containment, which the direct one has no equivalent of, and it resolves names through the tunnel — on the customer's own resolver, so onboarding judges the same addresses the dial will use rather than a weaker approximation. An unresolvable name is allowed through on both routes for the same reason: it is undialable anyway, and "unreachable" is a better message than a security error about a host that does not exist.

The tunnel's own gateway is an outbound dial we make from a customer-supplied address, so it goes through the direct route as a PUBLIC target in its own right.

Residual risk: DNS may change between the check and the driver's own resolution (rebinding). Closing it needs the driver to accept pre-resolved addresses; the practical mitigations in place are the per-user dial budget and the fact that a rebound target still has to speak the MongoDB wire protocol and authenticate before anything is stored.

The transport must be encrypted, and validated. Each adapter's client module is the one place that engine's driver client is constructed (mongo/client.ts, postgres/client.ts, mssql/client.ts), and each refuses a string that would not connect over validated TLS unless ALLOW_INSECURE_CLUSTER_TLS=true. The rules are per driver — a sslmode ladder, an Encrypt/TrustServerCertificate pair, tls/tlsInsecure — but the flag and the refusal they all raise are one thing, in engine/tls.ts.

The chokepoint is the design, not a detail. There were four new MongoClient sites — the pooled session, the onboarding diagnosis, and provisioning's admin and probe clients — and none of them said anything about transport, so mongodb+srv:// customers were encrypted by accident of the scheme while a pasted mongodb://host:27017/db sent SCRAM and every collection name in the clear. A guard in guardDial beside the address check would not have been enough on its own: the pipeline never goes through the controller — jobs/ connection-pool.ts opens the STORED string — so onboarding-only enforcement stops new plaintext clusters and lets every existing one keep dialling forever. guardDial does call it, but only so onboarding can refuse with the reason rather than surface the same refusal as a 502 out of diagnose; it runs AFTER the address guard, because a private target is refused for being private whatever its transport.

Three deliberate edges:

• tlsInsecure, tlsAllowInvalidCertificates and tlsAllowInvalidHostnames are the owner's call, per cluster. TLS whose certificate nobody checks is a connection anyone in the path can be — accepting it silently would be enforcing the ceremony rather than the property. But refusing it outright is wrong too: a private CA and an SSH tunnel are real reasons a real cluster cannot connect without one. So each is refused unless the matching checkbox was ticked on the connect form, and the answer is recorded on the cluster row (clusters.tls_overrides).

  Three boxes rather than one "insecure" toggle, because they are not the same concession: a private CA fails certificate validation with a perfectly correct hostname, and an SSH tunnel or a rewritten DNS name fails the hostname check with a genuinely valid certificate. One switch would make everyone give up both.

  The checkboxes are authoritative in both directions — ticking one writes the option into the string so nobody hand-edits a connection string to match a box they just ticked, and clearing one removes the option even if it was pasted in, so the form can never show three empty boxes above a string that disables all three.

  Stored on the row rather than inferred from the sealed string, for two reasons. Every dial is then verified against a decision rather than against whatever the string it was handed happens to contain — MongoConnection, the member fan-out, diagnose and provision all take the overrides, and omitting them gets the strict rule, which is the direction a forgotten argument has to fail in. And it is drawn back: ClusterHeader puts a badge naming exactly which checks are off beside the read-only badge, in the same place a reader already looks to decide whether to believe what is underneath. tlsInsecure reports alone when it is on, because it is a superset of the other two and listing them beside it would read as three problems where there is one broader one. A concession is chosen once, at a moment when the only goal is getting the connection to work — one nobody can see afterwards is one nobody reviews.

  There is no checkbox that turns TLS itself off. That is the deployment switch, and it is not a per-cluster question.

• It refuses rather than adding tls=true. A server with no TLS would then fail its handshake instead, turning a precise refusal into an opaque driver error, and a string that says tls=false is a statement worth contradicting out loud.

• Its own switch, not ALLOW_PRIVATE_CLUSTER_TARGETS. A VPC-peered or PrivateLink Atlas cluster is a private address that must still be forced to TLS, so coupling a transport rule to an addressing rule would quietly weaken the deployments that most look like they need the exception.

A cluster already stored with a plaintext string takes the UnsupportedServer shape in jobs/tasks.ts: skip the pass, warn every tick, mail the owners once a day, never retry — no retry fixes a connection string. Its own insecure outcome label rather than unreachable, because the cluster may be perfectly healthy and we are the ones declining to dial it; reporting "we could not reach you" would send the owner hunting a firewall that is not the problem.

The front door must not be open by default. SIGNUP_MODE (invite by default) gates account creation for both password and OAuth sign-ups via better-auth's user.create.before hook: the first account bootstraps the install, everyone after needs a pending invite for their address. open and closed are the deliberate alternatives. This gates the account only — joining an org still requires being signed in as the invited address, so knowing one buys nothing.

Both are backed by a per-user budget (10 dials/minute, errors/dial-budget.ts) that is consumed before the address check, so aiming at blocked addresses does not buy free attempts.

That budget answers 429 under its own code, DIAL_BUDGET (#162), and its message names the limit, the window and the seconds left. Its own code rather than the bare status because the other 429 this product answers is better-auth's per-address auth rate limit, and the two mean different things: that one says "you are going too fast", this one is a security control on a per-account clock, and an owner who typos a connection string a few times hits the same refusal an address sweep does. The dashboard treats the code as readable whatever list a call site narrowed to (web/src/lib/queries/errors.ts), because all four dialing routes narrow it to their own failures. The window is a rolling 60 seconds — a spent budget always comes back on its own.

MongoDB connection strings and agent tokens are encrypted with a swappable KeyProvider (the master-key custodian). v1 keeps the master key in the runtime environment (Docker secret / host env, never in git); it can later move to self-hosted Vault/OpenBao or a cloud KMS with no call-site changes.

Primitive: @noble/ciphers, XChaCha20-Poly1305 with managedNonce (24-byte random-safe nonce, auto-managed — avoids the GCM nonce-reuse footgun). Pure TS, audited, zero deps, no native build, fully typed.

import { xchacha20poly1305 } from "@noble/ciphers/chacha.js";
import { managedNonce, randomBytes } from "@noble/ciphers/utils.js";

// Master-key custodian. v1 = env; later Vault/KMS — same interface, swap impl.
interface KeyProvider {
  wrap(dek: Uint8Array): Promise<Uint8Array>;   // encrypt data key with master key
  unwrap(wrapped: Uint8Array): Promise<Uint8Array>;
}

// v1: master key = 32 random bytes, injected via env / Docker secret.
function envKeyProvider(masterKey: Uint8Array): KeyProvider {
  const kek = managedNonce(xchacha20poly1305)(masterKey);
  return {
    wrap: async (dek) => kek.encrypt(dek),
    unwrap: async (wrapped) => kek.decrypt(wrapped),
  };
}

interface Sealed { dek: Uint8Array; data: Uint8Array; }

// Envelope: random per-secret DEK encrypts data; KEK wraps the DEK.
async function seal(plaintext: Uint8Array, kp: KeyProvider): Promise<Sealed> {
  const dek = randomBytes(32);
  const cipher = managedNonce(xchacha20poly1305)(dek);
  return { dek: await kp.wrap(dek), data: cipher.encrypt(plaintext) };
}

async function open(s: Sealed, kp: KeyProvider): Promise<Uint8Array> {
  const dek = await kp.unwrap(s.dek);
  return managedNonce(xchacha20poly1305)(dek).decrypt(s.data);
}

Sealed is stored as two bytea columns (Drizzle customType).

Accepted risk (v1). With the master key in the app environment, an attacker who obtains both the runtime env and the database can decrypt everything. A KMS/Vault adds a separation boundary and an audit trail. For early stage with few customers this is acceptable provided the master key is injected at runtime (never committed), host access is tightly controlled, and a rotation path is planned. Moving the custodian to Vault/OpenBao or a KMS is a funded-stage / first-enterprise task, not a v1 blocker.

Password hashing is separate. better-auth owns credential hashing (scrypt/argon2). Never hand-encrypt passwords with the above — it is only for secrets at rest.

clusters.key_version records which key sealed each row, so rotation is a rolling operation rather than a stop-the-world one — old rows keep opening while new ones are written under the new key.

1. Generate a key and add it alongside the current one. Do not remove the old one yet; rows still sealed with it need it to open.

   MASTER_KEY=<old, still set>
   MASTER_KEY_V2=<new>
   MASTER_KEY_VERSION=2
   

   From here every new cluster is sealed at v2, and existing v1 rows still work.
2. Re-seal the backlog:

   node apps/api/dist/rotate-key.js
   

   It decrypts and re-encrypts inside the api process — plaintext never leaves it — one row at a time, each recording its new version. Safe to interrupt and safe to re-run; it reports re-sealed N, failed M and exits non-zero if anything failed.
3. Only once it reports zero remaining (re-run it; it will say "nothing to do") may the old key be removed from the environment.

A row that fails to re-seal almost always means the key that sealed it is gone. The tool names each one rather than stopping, so one orphan does not block the rest. It is also the only way back from an orphan: without its original key the stored connection string is unrecoverable and the cluster must be reconnected.

Organizations → members → connected clusters. Every row is scoped by orgId; all queries are org-filtered. An immutable audit log records every state transition.

The organizations are better-auth's (auth/organization.ts). The plugin's organization / member / invitation models are mapped onto the three tables tenancy already had, keeping their names and columns, so creating, renaming and deleting an org, inviting, accepting, changing a role, removing a member, leaving and switching are its endpoints under /api/auth/organization/* rather than nine hand-written oRPC routes. Two reads stayed on the api because the plugin has no opinion about them: GET /api/org (the plan and how much of it is spent) and GET /api/orgs (the caller's role per org, which organization.list does not return).

The keys stayed uuid. The plugin is handed a crypto.randomUUID() id generator instead of its own, because three tables carry a cascading org_id and the contracts type it z.uuid(). Retyping that key is a migration on a table that will hold live billing rows; a config block is not.

Entitlements did not move. A connected database is not a plugin concept. billing/plans.ts stays the only thing that decides what a plan allows, the plan columns are declared to the plugin as additionalFields with input: false (readable through it, settable by nobody through it), and every limit is enforced in the plugin's own hooks against the same counters TenancyService uses. membershipLimit is set to Infinity on purpose: it counts members, ours counts members plus outstanding invites, and two limits with one name is worse than one.

Three consequences a reader will notice.

• An org is made on purpose. resolveMembership used to insert one called "My Org" behind the first authenticated request, and acceptOrgInvite deleted that shell again if it turned out to be empty. A fresh account now belongs to nothing and lands on a create-org screen; getOrg answers null for it and listClusters answers [], because a create screen is not an error page. Making the next one is on /app/settings/organizations — the create screen only appears to somebody who belongs to nowhere, so on its own it offered exactly one org per account. It sat on the page about the org you were already in for a while, which was the only surface that could offer a second one; making a new org is a fact about the set of them, so it lives with the set. Nothing caps how many: a plan is bought per org, so a limit there would be a limit on how much a customer may buy, and the free tier is held by the cluster cap inside each one.
• The switcher is per session. session.active_organization_id replaced members.is_active, so two browsers can sit in two different orgs. An id naming an org the caller has since been removed from is ignored rather than obeyed, and falls through to their oldest membership.
• An invitation is addressed, not bearer. The credential was a one-time token mailed out and pasted back — anyone holding the string could join. The invitation id is not a secret now; accepting requires being signed in as the invited address, so the api lists your own invitations to you (GET /api/invites) instead of asking you to paste one.
• Three acts need a sign-in from the last hour, not just an owner session (#52, D43). TenancyService.requireFreshOwner gates going live, rotateConnection and deleteCluster on the session row's createdAt being within SESSION_FRESH_AGE_SECONDS (one hour, in @repo/contracts) — the rolling refresh never touches createdAt, so only an actual sign-in clears it. requireOwner asked are you an owner and nothing ever asked are you here right now; a week-old session on a borrowed laptop answers the first. The refusal is its own code, SESSION_NOT_FRESH, which the dashboard answers with a password prompt that re-fires the refused action (ReauthDialog), carrying the active org onto the new session by hand — better-auth mints it with activeOrganizationId null, so a multi-org owner would otherwise come back from the prompt in their oldest org and the retried act would 404. Flipping a cluster back to read-only rides plain requireOwner on purpose: an emergency stop that waits on a password is not one. The session lifetime itself stays better-auth's 7d rolling / 1d refresh, now stated in auth.config.ts as a decision rather than inherited — with the dangerous acts gated on freshness, shortening it would tax every daily user to close a window the fresh tier already closed.
• A second factor, demanded of owners when the deployment says so (#55, D44). better-auth's twoFactor plugin — an authenticator app (TOTP), backup codes, and a six-digit code mailed to the account's address where SMTP is configured (5-minute expiry, 5 attempts, stored hashed). The emailed code is gated on mail actually working: sendMail no-ops without SMTP, so hooks.before refuses /two-factor/send-otp with EMAIL_NOT_CONFIGURED rather than answering 200 for a code nobody receives. Its cost is recorded rather than hidden — the mailbox is where a password reset lands, so an emailed code defends against an attacker holding the password and not against one holding the inbox, and since it is available to every enrolled account it also sets the ceiling on what 2FA is worth here. Making it a per-account choice is a user column and a check on the same hook, and is the shape to reach for if that ceiling starts mattering. REQUIRE_OWNER_2FA is a posture flag like SIGNUP_MODE: off, everything works as before and enrolment is available from the account page; on, every owner-only mutation (TenancyService.requireOwner) and the four org routes that reshape access — invite, remove, role change, org delete, gated by a hooks.before in auth.config.ts because organizationHooks are handed the member and never the caller — refuse with TWO_FACTOR_REQUIRED until a code has verified. Owner of ANY org, since the posture is per account. Credential accounts only: a GitHub sign-in cannot enrol a TOTP and brings its provider's second factor, so the gate reads the account table live and stands aside. Backup codes are the recovery path, shown exactly once with the same only-copy treatment as a provisioned connection string; losing device and codes together is an operator reset, deliberately never self-serve. The cookie cache learns user.twoFactorEnabled the same way it learns org changes — POST /two-factor/* expires the cache cookie in hooks.after, or an owner who just enrolled stays refused for the cache's maxAge.
• The sign-in address can change (#83, D45). Verified accounts approve from the current address (sendChangeEmailConfirmation), then the new address verifies itself; unverified accounts flip at once (updateEmailWithoutVerification) and the new address gets the verification mail — an address that never proved anything approving its own replacement is theatre, and this is also what keeps the flow usable without SMTP. The signup gate applies to the change, refused in a hooks.before on /change-email with the gate's own reason before any token is minted: SIGNUP_MODE decides which addresses may hold an account, and renaming one must not walk around it. The old inbox is told either way, better-auth demands a fresh session on the route (the same hour as the fresh tier above), and a change to a taken address answers success and does nothing — its anti-enumeration behaviour, kept. In-flight invitations to the old address expire unclaimed; they key on the session's email, and re-inviting is one click.
• No request waits on SMTP. Every mail sent from inside a request somebody is sitting in front of — the verification link, the reset link, the emailed second-factor code, the change-of-address notices, an invitation — goes through sendMailDetached, which starts the send and returns. This is not an optimisation; it is the difference between a working sign-up and a broken one. better-auth awaits sendVerificationEmail before it answers, so the reader's sign-up costs whatever the transport costs, and the hosted deploy found what that can be: the platform DROPS outbound SMTP rather than refusing it, nodemailer's default connection timeout is two minutes, and one POST /sign-up/email took 122.5 seconds and then answered 200. Nothing is given up by not waiting — sendMail swallows its failures and returns a boolean no caller has ever read. A job that mails still awaits, because a task must not report success with its mail in flight. Underneath, the transport states all three timeouts (10s connect, 10s greeting, 20s socket) rather than inheriting them, and sets requireTLS so that a non-465 port — which is how you get around a host that blocks the standard submission ports — cannot quietly fall back to sending the SMTP password in the clear. Known limit: a send that fails is still invisible to the reader. EMAIL_NOT_CONFIGURED covers the case where mail is impossible; nothing covers the case where it is configured and broken, so "Send it again" answers 200 either way.
• The verification link lands somewhere that can report the outcome (#324). better-auth builds the link with the caller's callbackURL and falls back to / — the marketing page, which is deliberately static, has no loader and makes no api call, so it could report nothing at all. Success and failure looked identical: confirmed, signed out, on a marketing page, with an expired link redirecting to the same place carrying a ?error= nothing read. The fallback is now rewritten to ${WEB_ORIGIN}/verified in sendVerificationEmail itself rather than at the callers, because there are three of them and one cannot pass it: signIn.email uses the same callbackURL field for where a successful sign-in lands, so the mail that rides along with a refused sign-in has no way to choose. A caller that named a destination keeps it, which leaves change-email on /app/account. The page tells TOKEN_EXPIRED and INVALID_TOKEN apart, offers the resend on both, and refuses to read an unknown code as success. autoSignInAfterVerification stays off, deliberately: it would turn a forwarded email, a shared inbox or a mail archive into a session on an account that holds the connection strings for somebody's databases. The link proves the address; the password proves the person, and the page says so, because a reader who is not told waits to be let in.
• Deciding who is asking costs one round trip, usually zero. requireSession resolves auth.api.getSession once per request and memoizes it on the request object — several endpoints ask twice (the handler for the org, guardDial for the user id), and each ask used to be its own query (#77). Under that, session.cookieCache carries the session in a five-minute HMAC-signed cookie, so the common request decides authn without postgres at all. A session change invalidates the cookie in the same response: set-active re-signs it, and every other organization mutation expires it — the plugin stops at the session row on create, delete and accept-invitation (measured, not assumed: an accepted invitee stayed scoped to their old org until the cookie aged out), so an hooks.after in auth.config.ts expires the cache wherever the response is not already carrying a re-signed one. Ordinary api responses re-arm it (the onSend hook in main.ts forwards what better-auth set while resolving) — and membership and role are still read from members every request, so removal and role changes bite immediately. The trade is revocation: a session torn down behind the browser's back keeps answering for up to the cache window. Sign-out is not that case — it clears both cookies in the same response — and nothing the product does today revokes a session any other way.

Deleting an org is the one verb that reaches past our own tables. Cascades take the clusters, snapshots, recommendations, ROI, policies and cooldowns, and touch nothing on the customer's servers — where an index we hid stays hidden and a least-privilege user we created stays created. So beforeDeleteOrganization runs the same restoration a disconnect does (every observe-window index un-hidden first), and the confirmation dialog makes the owner type the org's name and names each provisioned user with the command to drop it. Dropping those needs admin credentials we deliberately never kept, and after the row is gone there is nothing left to name them from.

There are two rate limiters in front of the api, they are not alternatives, and for a while only one of them was the one that mattered.

The Fastify limiter is per replica, and that is a property of the design. Its counters are in memory, so api.replicas: 2 means RATE_LIMIT_MAX × 2 across the deployment, and a rolling deploy hands every bucket back at zero. The multiplication is the smaller half: memory counters make the limit non-deterministic under load, since which pod a request lands on decides which bucket it spends — so an attacker spreading requests gets the sum, while a legitimate user behind a shared NAT can trip on one pod while another sits empty. Read the documented number as "per replica, per minute, roughly".

better-auth's is the one that decides a sign-in, and it was invisible. Its default is 3 requests per 10s on the credential endpoints — tighter than anything we advertise, so AUTH_RATE_LIMIT_MAX=20 was a documented number that never applied to /sign-in. Measured before the change: 22 of 25 sign-in attempts throttled under NODE_ENV=production, 0 with it unset. Two things fixed that:

• rateLimit.storage: "database" — the counters live in rate_limit (one row per client address and path), so every replica counts against one total and a rollout does not reset it. better-auth prunes rows past the widest window on each write, so the table needs no retention job of ours.
• customRules on /sign-in/*, /sign-up/*, /change-password and /change-email, priced at AUTH_RATE_LIMIT_MAX per minute. The configured number is now the effective one.

Two defaults are deliberately left alone. The global auth limit stays better-auth's 100 per 10s, because the dashboard reads its session through /api/auth on ordinary navigation and pricing the whole surface at the credential number would throttle reading rather than guessing. And the mail-sending endpoints (/request-password-reset, /forget-password, /send-verification-email) stay at better-auth's 3 a minute: each one sends an email to somebody who did not ask for it, which is a different limit with a different reason.

Both limiters need to know who the client is, and only one of them asks. Fastify resolves the address from TRUST_PROXY, which it takes as true or a CIDR list; better-auth is then handed that one answer as a one-entry X-Forwarded-For on the synthetic Request the mount builds for it (auth/http.ts), with whatever arrived under that name overwritten. So there is one resolution and one setting, and the two limiters cannot disagree.

That replaced handing better-auth the CIDR entries as trustedProxies, which was right in the deployments it was written for and unusable in two others. Behind a managed host — Render, Fly, anything behind a CDN — the ranges are not published, so no list can be written; better-auth then believes X-Forwarded-For only when it carries exactly one address, the platform's hop makes two, and every caller resolves to one shared no-trusted-ip bucket. That is a limit which throttles your users and not an attacker, and it announces itself in the log: "Rate limiting could not determine a client IP and is falling back to a single shared per-path bucket." Directly exposed, the reverse: better-auth reads X-Forwarded-For by default and one address is the shape it trusts, so a caller could send a fresh address per attempt and never reach the sign-in limit — the exact hole that TRUST_PROXY being off by default exists to avoid. Overwriting the header closes both.

Which leaves the dialect as the only decision, and true as the one to avoid: it trusts every hop and takes the first entry in X-Forwarded-For, which is the one a caller writes. Where the platform appends rather than replaces — Render does — that lets each attempt pick the bucket it spends. A CIDR list, then, and the address is the one that platform actually saw. A managed host that publishes no ranges still connects from somewhere: with TRUST_PROXY unset, the request log's remoteAddress is that address. A CDN in front is named by the header it sets rather than trusted by its ranges, which its own customers can send from (below). A count of proxies is not a dialect. Fastify trusts no hop at all for a number, because a count never checks the address it is connected to, so anything that reaches the api around its proxy would be believed (D177); the api refuses one at boot rather than serve on with one shared bucket.

Unless the proxy APPENDS — then the client is the one header the CDN sets (#574, D180). Render is such a host: Cloudflare is in front of it and appends, Render's proxy appends again, and a request arrives as <what the caller wrote>, <caller>, <Cloudflare edge>. No TRUST_PROXY value finds the caller in that. true takes the leftmost entry, and Cloudflare's published ranges are no better: a Cloudflare Worker's request arrives from 2a06:98c0:3600::103, which is inside 2a06:98c0::/29 on Cloudflare's own list, so the walk trusts it and reads on into whatever the Worker wrote. CLIENT_IP_HEADER names the header to believe instead — cf-connecting-ip, which Cloudflare sets itself and pins for Worker traffic — and its one address becomes the whole X-Forwarded-For chain, on the http server's request event before Fastify sees anything, so the request log, both rate limiters and the security trail agree. TRUST_PROXY still decides whether the peer is believed, and a header with no trusted peer refuses to boot. On Render's all-in-one image the peer is the dashboard's own loopback passthrough, so the pair is TRUST_PROXY=127.0.0.1 and CLIENT_IP_HEADER=cf-connecting-ip. IndexterityAuthRateLimited firing with a single-bucket configuration looks exactly like a brute-force attempt, which is why the alert names this before it names the limit.

What is still per replica: everything the Fastify limiter counts, which is every oRPC route. Sharing those too would mean Redis (secondaryStorage plus a store for @fastify/rate-limit) — correct, and the first hard dependency this control plane would take on something other than Postgres, so it waits for a deployment that needs it.

actions is an immutable record of every index operation: kind, actor, result, namespace, index name, timestamp. It covers exactly one half of what this system does. The other half — sign-ins, who became an owner, who was removed, whose credentials were rotated — left no row anywhere (#53). session and account are current state, not events: a revoked session is a deleted row.

The asymmetry was the problem. The index trail exists so that when an index disappears you can prove what removed it and why, and that story had a hole in the middle: it could say this drop was approved by user X and not how X came to be an owner, or whether X was the person behind that account at the time. For an install where one org's owners can reach another team's production database, that is the question an incident actually asks.

security_events records the owner-level acts, one row each, never updated:

Four choices in the schema, each one a question the issue asked to settle deliberately:

• A second table, not a column on actions. actions hangs off a recommendation, cascades from a cluster and ages out with the plan's history window. These are per org, several have no cluster at all, and making the recommendation key nullable would leave every existing reader filtering for rows it never wanted.
• Nothing cascades in. Every foreign key is set null, and the actor's email and the target's name are kept beside the ids — so a row still reads after the org, cluster or account it named is gone. A trail that a deleted org takes with it answers nothing about the org that was deleted. CLUSTER_DISCONNECTED goes further and carries the cluster's id in its metadata, because by the time it is written the row it would point at has been deleted. TUNNEL_REMOVED does the same. TUNNEL_TESTED is the odd one in this table because it changes nothing — it is here because it is the only act that reaches OUT, sending datagrams to a customer's gateway on demand, and it records the VERDICT rather than the fact of asking: a run of tests against a gateway that never answered is a story, and "tested" alone is not. It is not marked severe; an owner checking their own gateway is ordinary work. TUNNEL_UPDATED records both sides of a replaced wg0.conf — gateway, AllowedIPs, DNS — because what a peering reached BEFORE is the question, and the only copy of the answer was the config that was overwritten. Never the config itself: it carries a PrivateKey, and this table is read by people who are not meant to be able to bring the tunnel up.
• It does not age out with the plan's window. The incident that needs these rows is usually older than the day it is noticed.
• IP and user agent are stored, which is personal data and is no more of it than session already holds: an incident asks which address and which client, and "an owner, from somewhere, on something" is not an answer. The address is null rather than wrong when it cannot be established — the api's own handlers use Fastify's resolved request.ip, and the better-auth half reads a forwarded header only when TRUST_PROXY says a proxy is in front (§10.5).

Where the actor comes from is the load-bearing part. databaseHooks.session.create would see a sign-in, and the organization plugin's afterUpdateMemberRole would see a promotion — but the plugin's hooks are handed the member, the target user and the org, and never the caller. "Who promoted whom" is the question the table exists for, so it is recorded from an after-middleware, with better-auth's own getSessionFromCtx resolving the actor the same way its sessionMiddleware does.

Two things there were established by measurement rather than by reading the types, both caught by an integration test asserting that no row has a null actor:

• ctx.context.session is empty in an after-hook even on routes that required a session. Every org event landed anonymous until the actor was resolved explicitly.
• A sign-out cannot be recorded there at all — by the time the hook runs it has destroyed the session that would identify who signed out. That one comes from databaseHooks.session.delete, whose payload IS the session that ended. Revocations are not recorded there, because the middleware sees those with the revoker's own session intact and bulk revocation does not delete row by row.

Two deliberate omissions. No credential is copied in: /revoke-session identifies its target by session token, so that row records the scope and not the token, and a cluster's row records the provisioned username and never a connection string — this table is read by people who are not meant to be able to use what it records, and a test asserts it. And previousRole is not recorded: only the plugin hook is given it, so a row says what the member is now and who made them that; the role before is the previous MEMBER_ROLE_CHANGED row for that target, which is what a trail is for.

The table shipped with 23 kinds of act, an index built for exactly the query a reader would make, and no controller that made it (#158). The data was there, already indexed, already exempt from retention — and the only surface it had was psql.

listSecurityEvents (GET /security-events) and Settings → Security are the reader. Four decisions in it:

• Owner-only. Everywhere else in this product a member reads everything in their organization. Not here: every row carries the address and the client a colleague acted from, and who-signed-in-from-where is not team-wide reading. The tab is shown to everyone and the page says Owners only when the api refuses, rather than the tab disappearing — a control that is present for some readers and absent for others is how somebody concludes the feature does not exist.
• Scoped to the active org, always. security_events_actor_time answers the other incident question — "everything this account did", which crosses orgs — and that is an operator's query, not a tenant's, so no input on this route can ask it. What the route does use is security_events_org_time, whose schema comment already said "every read is one org's trail, newest first".
• Paged with a keyset, not capped and not offset. This is the one table that never ages out, so a cap would mean an owner can only ever see the newest hundred acts of a trail whose whole value is that it goes back further than the day anything was noticed. The cursor is (created_at, id) and not a timestamp: accepting an invitation writes rows in the same request, and a cursor that was only a time would skip whichever of them sorted second. Offset would be worse again — the trail grows at the head, so a sign-in between two page loads pushes a row across the boundary and an offset page repeats it.
• total counts what MATCHES the filter. A filtered page reporting the whole trail's size beside eleven rows is a sentence that describes rows the reader did not ask for.

A failed sign-in has no actor, and the screen has to say so. SIGN_IN_FAILED records the address that was typed as the target and leaves actor_user_id null, deliberately, because whoever it was did not prove they were that person. A page that renders every row as "

The list of act names moved to @repo/contracts with this: the screen labels rows and offers the kind filter from the same list the writer writes, and two copies would drift the moment an act is added. The per-act metadata shapes stayed in the api, next to the call sites that fill them in. event stays a string on the contract, not an enum — the column is text so that recording a new kind of act is a constant rather than a migration, and a row written under a name a build does not know still has to render.

Not to be confused with the cluster page's Activity table, which is the index pipeline: every hide, build, drop and rollback. The two were separated on purpose, for the reasons above.

Recording is best-effort and logged. A problem with this table must not stop people signing in, and refusing the sign-in would not un-record it either — the act already happened. Nothing reads the trail through the api yet: it is written, and surfacing it in the dashboard is a separate question about who should be allowed to read it.

• better-auth tables (users, sessions, accounts) — via the better-auth Drizzle adapter. rate_limit joins them when rateLimit.storage: "database" is set, which it is: one row per (client address, auth path), so the auth budget is the deployment's rather than each pod's memory (§10.5). better-auth prunes it on write; nothing of ours has to.
• organizations, members, invites — multi-tenant; everything scoped by orgId, and all three are better-auth's organization plugin's models under our own names (§9.3). organizations carries a unique slug (required by the plugin; nothing routes by it) alongside the plan columns. The org switcher's selection lives on session.active_organization_id, not on the membership — it is per session, and unset falls back to the oldest membership.
• clusters — connection mode, encrypted creds/agent token (Sealed), demoMode (default true).
• agents — registration, last-seen, version (phase 2).
• cluster_indexes — the dimension half of the snapshot series: cluster / db / collection / index, and the spec it was seen in. One row per distinct shape, keyed by a generated spec_digest (sha256(spec::text), which Postgres can compute because jsonb stores its keys sorted), so a rebuilt index gains a row instead of overwriting one and older snapshots stay joined to the shape they were taken under. sha256 rather than md5, and not for secrecy: a collision here silently MERGES two index shapes into one dimension row, so every snapshot of one would be reported under the other's spec — md5's failure is reachable by construction and silent when it lands, which is the wrong trade against 32 extra bytes on a table holding a few hundred rows.
• cluster_namespaces — the dimension half of the latency series (#551): cluster / db / collection, one row each, and an INTEGER identity key where every other table uses a uuid. It is the one key repeated on every row of a table that grows per collect, and in both of that table's indexes, and a uuid is 16 bytes stored and 36 characters on the wire (D145). Readers resolve names once per namespace, never once per sample, and retention removes a namespace older than its window once nothing references it.
• security_events — the owner-level acts: authentication, membership, and what was done to a cluster's access (§10.6). Per org, cluster nullable, every foreign key set null, and no retention window.
• index_snapshots — time series, and only the measurement: index_id, size, per-member ops, hinted, and the run bounds capturedAt / lastSeenAt / observations. See §11.2.
• cluster_footprint_days — the cluster's total index bytes and index count per day, as the day's last collect measured them (#640). One row per cluster per day, upserted by every collect; the footprint chart reads it (§8.1).
• workload_shapes — one row per scanning query shape the create side has seen, with the gate that declined to act on it (§8.3). Not a series and not run-length encoded: the counters of a shape worth reporting move on every pass, so what the row carries is first_seen_at, the newest reading and an observations count, upserted in place. Keyed by a generated shape_digest over the same canonical-jsonb trick cluster_indexes uses. constants — the one field carrying customer values — is dropped on the way in (D128), and retention prunes by last_seen_at.
• analysis_notes — one row per cluster per producer: why a pass had nothing to say (§6.2, D99). Both jsonb maps are keyed by whatever the writing pass called its reasons, which is what let #432 add two collection counts to the WORKLOAD row without a migration.

Two things used to make this table grow with the collect cadence rather than with the cluster, and both were measured on the dev database before being fixed: spec was ~240 bytes a row against the ~99-byte counter it accompanied, and 211 distinct (index, spec) pairs sat across 403 rows for 204 distinct indexes — a per-index constant, rewritten 1,460 times a year per index at 6h. Separately, an idle index's counters are byte-identical collect after collect, so most rows were a fresh timestamp attached to an unchanged number.

The dimension. Everything about an index that does not change between collects moved to cluster_indexes. Every read of the series costs one join; nothing else changed semantically.

latency_samples got the same treatment later, and for bytes rather than correctness (#551, D174). Its namespace, database and collection as text, sat on every row and led both of its indexes, so it moved to cluster_namespaces, and the stored span went with it (below). Measured on synthetic rows shaped like production, a new sample costs 350 bytes on disk instead of 620. Migration 0069 rewrites the table once, which is the only way a dropped column's bytes leave the rows already stored: 57 MB to 25 MB on 96,768 rows. The probe's newest-sample read now joins each namespace laterally to one index probe. That is 8.3 ms against 34.4 ms on 120k rows, and its cost follows the namespace count rather than the history.

The run. A snapshot row now covers the closed interval [capturedAt, lastSeenAt], confirmed observations times. A collect whose counter fingerprint matches the newest row extends its end; anything else inserts. size_bytes is deliberately not part of the fingerprint — an unused index on a write-heavy collection grows on every insert, so keying runs on size would collapse nothing in exactly the case worth collapsing — so it rides along at its newest value. That is all any reader wants but the footprint chart, which keeps its own total per day in cluster_footprint_days (§8.1). hinted is sticky (OR-ed on extend), because one sighting of a hint() anywhere in the retained history is what protects the index and the next quiet collect must not erase it.

Simulated over a year at the 6h cadence for a realistic mix (80% idle indexes): 113 MB → 79 MB with the dimension alone (30%), → 16 MB with runs as well (86%). An index nobody touches costs one row rather than 1,460, which is the property that makes shortening the cadence a load question and not a storage bill.

The cadence is hourly, and the ceiling on it is one table. Storing only what changed made index history nearly free — measured on real data, 76% of looks collapse into an extended run — but latency_samples collapses none of them. $collStats totals move on any operation, so a live collection differs at every look and there is nothing to run-length: that table grows in step with the cadence while index history grows at about a quarter of the rate. Hourly is a usable trend where 6h gave four points a day; going shorter is a question for getLatencySeries becoming bounded and old latency history being rolled up first, not a question of load. A collect takes under a second against a hundred collections, and about a round trip per collection against a remote cluster.

Why omission was not an option. maxGapHours and the usage trust gate read a hole in the series as "we stopped watching, so absence of usage proves nothing" — so simply not writing when nothing changed would have made an idle index indistinguishable from a cluster we lost. A run is the positive form of the same statement: we looked at lastSeenAt, and it was still this. Two invariants keep that honest:

• A run never spans a hole longer than MAX_GAP_HOURS (analysis/classify.ts, shared with the writer in jobs/runs.ts). The trust gate only inspects the gaps between runs, so a run allowed to swallow a fortnight of silence would present an unbroken series. Past that tolerance an identical reading is a coincidence, not a continuation, and it gets its own row.
• last_seen_at has no column default. defaultNow() reads as the obvious choice and is a trap: a caller that sets only capturedAt writes a row claiming a months-old reading was confirmed this instant, in the one field the gate consults. No default makes every insert say what it means.
• A run carries the widest gap inside its own span (max_gap_ms, maintained on extend as greatest(previous, now - last_seen_at)), and the gate checks it alongside the holes between runs. Without it, "no hole inside a run" rested entirely on the collector's MAX_GAP_HOURS ceiling — one half of a safety invariant living in a different module from the half that depends on it, with nothing in the data to confirm it. bigint, because int4 tops out near 24 days of milliseconds and a check that overflows exactly when the thing it guards against happens is not a check. Absent or zero reads as "nothing to declare", so rows written before the column are trusted exactly as they were, and a writer bug that produced a negative fails open here rather than refusing every finding on the cluster — the between-runs check still stands either way.
• The spec digest is sha256, not md5. Not secrecy: a collision silently merges two different index shapes into one dimension row, and every snapshot of one would then be reported under the other's spec. Note for anyone regenerating that migration — cluster_indexes_identity is UNIQUE over the digest, so dropping the column takes the index with it, and re-adding the column does not bring it back. The generated migration omitted the CREATE UNIQUE INDEX; applying it as-issued would have quietly removed the uniqueness the dimension upsert depends on.
• Runs cannot overlap, and the database holds that. A generated span tstzrange(captured_at, last_seen_at, '[]') plus EXCLUDE USING gist (index_id WITH =, span WITH &&) on index_snapshots. latency_samples holds the same guard WITHOUT a stored span (#551): EXCLUDE USING gist (namespace_id WITH =, tstzrange(captured_at, last_seen_at, '[]') WITH &&). An exclusion constraint takes expressions the way an index does, so the stored column existed only to be indexed. Every reader finds holes by differencing previous.last_seen_at → next.captured_at, so an overlap is a negative gap — which reads as no gap at all, the same failure by the back door. Two collects racing, or a clock stepping backwards, can produce one. Bounds are inclusive on purpose: with '[)' a run of one observation is an EMPTY range, and an empty range overlaps nothing, so most rows on a busy cluster would go unprotected. btree_gist is core contrib, so this adds nothing a deployment must install.

How long rows are kept and how much of them a plan may see are one window per plan: entitlementsFor(plan).retentionDays. The daily sweep (jobs/retention.ts) groups clusters by their org's plan and deletes past each group's cutoff:

• the two time series;
• the dimension rows those deletions strand;
• scanning shapes;
• the footprint's days;
• settled recommendations.

Visibility is the same window, applied at every read through historyWindow(db, clusterId) (jobs/plan.ts). That covers the day a row can outlive its window between two sweeps.

The filter is on last_seen_at, never captured_at: a run that began before the cutoff and is still being extended is the current state of a live index, and excluding it would hide the present because the past is out of view — the same mistake that once had retention deleting live rows.

This reverses an earlier split (#549, D172). Deletion used to run one cutoff for the whole deployment, the longest window any plan could claim, with each plan's window applied only on the way out. That made an upgrade return history immediately, and pruning one bulk range. The cost was storage that no reader was entitled to, which on a small managed database is the first limit reached. On Neon Free, 0.5 GB holds about a hundred days of the hosted fleet's history, and a 365-day sweep over FREE and PRO orgs kept half a year nobody could see. Now:

• An upgrade starts from what the old plan kept, and the new window fills as time passes.
• A downgrade deletes at the next sweep, and nothing brings it back. set-plan.js says so when it moves an org down.
• Pruning is still a handful of statements: one pass per plan, not per cluster.

The window applies to the engine's reads as well as the dashboard's. History depth is the entitlement, so a row that is kept but not yet swept has to be invisible to classify.ts too, or the plan is advertised rather than enforced.

Reads that only ever want the newest row — the collector's own run lookup, the five-minute probe, the collection footprint, cluster freshness — skip the window: the newest row is inside every window by definition, and no plan is entitled to less than what is true now.

Every reader was reworked to treat a row as a span rather than an instant — getCollections (now last_seen_at = max(last_seen_at), since the newest collect extended or inserted every index it saw), the trust gate and classifyUsage, activeHours, inferChangeWindow (which takes one series per namespace now, because run ends no longer line up across collections), dynamicObserveDays, summarizeLatency, cluster freshness, and retention (which prunes by when a run ended — pruning on its start would delete the row an idle index is still living in). Every threshold phrased as a count of samples reads observations, not the row count.

• recommendations — type, target, rationale, safety class, usage class, estimated savings, state. estimated_bytes_saved is the index's size times the number of nodes that HOLD a copy — a drop frees it on every data-bearing member, not one. That count comes from cluster_rosters, excluding mongos, and deliberately not from the usage reading: a reading speaks only for the members that answered that pass, which is a property of the pass and not of the index. On the production five-node cluster 18.1% of snapshot rows recorded a partial reading, which used to swing the figure by up to 5x on an index whose size had not moved a byte (#528). On a SHARDED cluster the sizes are already cluster-wide sums, so multiplying still over-approximates by the shard count — an acknowledged ceiling, unchanged.
• actions — audit: what / when / who-or-policy, before/after, rollback token, result.
• roi_metrics — freed bytes, throughput delta, index-count delta, per period. recommendation_id attributes each row to the drop that earned it (undo inserts a negative row against the same id, netting it out of the dashboard's per-index list).
• policies — per-cluster auto-apply rules, maintenance windows, thresholds, observe-window override.
• audit_log — immutable, every state transition.

An index optimizer is an embarrassing place to have nine un-indexed foreign keys, which is what an audit of pg_constraint against pg_index found. An un-indexed foreign key is not a slow query — it is a scan of the whole child table for every parent row deleted. Retention bulk-deletes settled recommendations and roi_metrics references them, so removing ten thousand rows took 8.4 seconds, of which 5ms was finding them. With the index, 594ms.

members is the other one worth naming: it is the tenancy check on every authenticated request, and it was a sequential scan. It is queried three ways — by user, by org, and by both — so the composite leads with user and the org gets its own, which is the same equality-ordering rule the engine applies to everyone else's indexes.

The integration suite now fails on any foreign key without a leading index. The nine are fixed; the point is that a tenth cannot be added quietly — and it caught the tenth: session.active_organization_id, added with the organization plugin, whose ON DELETE SET NULL would have scanned every session row for every org deleted.

• oRPC contracts live in packages/contracts and are shared by api, web, and agent — one source of truth for types, no codegen, no any. Zod schemas validate at the boundary. Split by direction: schemas.ts is what the api returns, inputs.ts is what it accepts. Inputs are named rather than inline because the dashboard's form fields validate against the same objects (§14.6), so a rule has one definition and a route cannot refuse what a field accepted.

• nullable() on the way out, optional() on the way in. schemas.ts holds 61 nullable() and zero optional(): a response says a field is absent by carrying null, so a reader never has to tell "not set" from "not sent". inputs.ts is the opposite, and the asymmetry is load-bearing — an absent field on a PUT means "leave it alone", which is a different sentence from a null that clears it (observedDatabasesInput spells out both).

• A timestamp is instant (z.iso.datetime()), never z.string(). Nineteen fields were the bare string until Sep 2026 (D124), and that is the same unchecked claim as as: the type said string, thirteen places in the dashboard called new Date() on it, and three checked the result. None of the other ten failed — they spread, and every outcome reads as an answer. new Date(bad).getTime() > Date.now() is false, so a pending drop rendered as not pending and its cancel button never drew; the staleness badge that exists to stop old numbers reading as current silently stopped drawing, because NaN > 48 is also false; two screens rendered the literal string "Invalid Date"; and formatTimestamp called .toISOString(), which throws, in the server-render branch. Worth noting that two of those sites already had a Number.isFinite guard and it chose the reassuring answer.

  UTC only, no offset form: toISOString never emits one, so accepting +02:00 would widen what the api may send without widening what any reader was written for. The reader's half is one instantOf (apps/web/src/lib/instant.ts) returning Date | null, so a caller has to say what it draws for null — and where a NaN branch existed it now says unknown rather than says fine.

• Dashboard API: oRPC (@Implement on Nest controllers) over Fastify; consumed by the web app through the oRPC OpenAPILink client.

• Agent channel (phase 2): authenticated, outbound-initiated. The agent posts snapshots over HTTPS and receives commands (create/drop/hide) via a long-lived channel (long-poll or websocket). Agent authenticates with a per-agent token; mTLS is an option for higher tiers.

graphile-worker (Postgres-backed) — no Redis dependency, fits the compose setup. Job kinds:

• collect — snapshot $indexStats / sizes per cluster on a schedule.
• classify — run core over recent snapshots to (re)generate recommendations.
• execute — run the apply pipeline steps (hide, drop, create) with pre-flight.
• verify — measure before/after and write roi_metrics.

In agent mode much of collect and execute runs agent-side on its own timer; the control plane orchestrates and stores. graphile-worker can be swapped for Redis + BullMQ only if scale demands it.

A pass that lands announces itself: every per-cluster task exits through one funnel (runClusterTask), whose ok path sends pg_notify('cluster_events', …) — and the three transitions worth reacting to mid-pass (a drop reaching HIDDEN, a build graduating, a regression firing) emit at the write itself. The api LISTENs and streams them to the dashboard over SSE; §14.9 has the whole path. Emissions are best-effort on purpose: a lost nudge costs freshness, a thrown one would re-run a landed pass.

A pass that fails permanently announces itself too. job:failed fires on every attempt, so what is reported is the last one — the dead-letter transition, one event per task rather than one per retry (§15.4). Every task, not only the per-cluster ones that email their owners: retention, digest and the schedule* dispatchers have nobody to mail, so before that they were the failures with no audience at all.

/app is a layout route, not a page. It owns what every signed-in view needs and nothing about any one view: the auth gate, the "api is unreachable" state, the org switcher, sign out, and the nav rail.

It was one 452-line route that fetched ten things for every navigation, so opening the team page pulled a latency series it never drew. Splitting it is what makes /app/settings cost four calls instead of ten — and splitting it again is what stops a cluster's overview paying for a policy only its settings tab draws.

A cluster is an address, not a selection. It used to be ?cluster=, and "none selected" meant the first — which made a cache key of null mean one thing before a cluster existed and another after, and left the bar and the panels resolving the same question separately. That is what showed as a dashboard drawing the previous org's numbers under the new org's name after a switch. The id is a path parameter now: concrete before anything reads it, bookmarkable, openable in a second tab, linkable from an alert. /app is only the redirect that answers which one did you mean — the first cluster, or /app/clusters/new when there are none.

That redirect is deliberately in both the loader and the component. The loader is what makes a cold navigation land without a flash; the component is what catches the cases no loader re-runs for, because invalidateSession moves the cluster list without re-running one — making your first organization, switching org, disconnecting the cluster you were looking at, being removed from an org. Same rule under /app/clusters/$clusterId: the loader refuses an id this org does not own, and the component leaves if it stops owning it.

The dashboard is a dashboard. No configuration form is on it. A policy is not a measurement and neither is a disconnect button; both were under the ROI cards because that is where there was room. They are the cluster's own settings tab, which is also what lets them be explained at more than badge length.

Connecting a cluster is onboarding, not a card at the bottom of the numbers. It is one link from every page — which is what somebody connecting their second cluster is looking for — and it is where an org with no clusters lands, so the empty state is the thing to do rather than a dashboard of zeroes with a form underneath.

A limit is drawn where it is spent, not where it is described. Splitting the routes once moved 1 / 1 clusters onto the org page along with the rest of the plan, and the only page that could then warn you about the cluster cap was the one without the connect form on it — the e2e test navigated to go and find the number, which is a test agreeing with wherever the number lives rather than defending the behaviour. The clusters half of the meter is beside ConnectClusterForm now and the seats half stays beside the invite form; /app/settings keeps both as a plan summary. Nothing is disabled when the count is full: checking a connection string stores nothing, the api owns the refusal, and a button greyed out by a stale count would be a lie in the other direction.

TanStack Query owns every read and every write, and there is one key per api call. The table below is a representative slice rather than the whole set — listClusterEvents is not in it because it is the stream that invalidates (§14.9) and caches nothing, and the two paged reads carry the WHOLE request in the key — ["cluster-indexes", id, page], where page is offset, limit, sort, direction, filter and namespace scope — so each has a prefix helper (clusterIndexesAll, clusterWorkloadAll) for the invalidations that move every page at once. The request object itself rather than its members spelled out, and the type is the contract's own input minus the cluster (D139): the written-out list stayed on the pre-#445 keyset cursor for three releases, so every page, size, order and search of one cluster hashed to a single entry and the footer snapped back to page one on every click (#455). TanStack Query does not refetch when only the queryFn changes under an unchanged key, and hashKey sorts an object's keys and drops undefined members, so the object is canonical and there is no second list to fall behind:

router.invalidate() appears nowhere in apps/web/src.

The org mutations are not in that table because they are not api calls: create, rename, delete, invite, cancel, accept, reject, role, remove, leave and switch all go to better-auth's organization plugin through authClient (§9.3). The hooks in queries/mutations/org.ts still own the invalidation, and still choose between the three blast radii — one key for something inside the org, org + orgs for a rename, and the whole cache for anything that changes which org answers. What they gained is an unwrap: better-auth resolves to { data, error } rather than throwing, and a promise that never rejects has no onError.

They were grouped that way — three keys, shell, pipeline and telemetry, each one a Promise.all over three endpoints cached as a single entry. It reads well and it was wrong: grouping by what moves a key is a claim about the writes, imposed on the reads. What the reads paid was concrete.

• The org page fetched a cluster list it never draws, because the member list shared an entry with it.
• The latency chart could not fail without blanking the collection table beside it — one rejected call inside the Promise.all emptied all three.
• Flipping a cluster to live invalidated shell, so it refetched the team page's member list as a side effect.

The request count did not change: a blob of three was always three requests, and the Promise.all only decided when they were awaited. What changed is that three answers stopped sharing one cache line.

Two things followed from the split.

Failures became query errors. Each queryFn used to catch and return an empty payload, which put the error where no caller could see it. Now a rejection stays a rejection, isError is available to anything that wants to say more than "empty", and each read hook defaults to a stable empty value — so a dead read still costs its own panel and nothing else on the page. The loaders use Promise.allSettled for the same reason: a rejection there would take out the whole route instead of the one panel it belongs to.

authed / apiDown became a derivation. It was the one thing the composite query was genuinely computing, and no endpoint answers it — it is an inference from how the reads failed. useShell() now derives it over the four org-level queries: a 401 on any of them means signed out and outranks a transport failure, anything else means the api could not be asked. Both are answers the layout knows how to draw, and there is a test per branch because the logic left a try/catch.

plan.clustersUsed is part of the org payload — the api resolves plan usage server-side so the dashboard can show a limit before someone hits it. So the counter that reads "1 / 1 clusters" lives under org while the list it counts lives under clusters. The old shell key held both in one entry and refreshed the counter for free; splitting them made connecting a cluster leave a stale "0 / 1 clusters" on the team page. The e2e suite caught it. Creating or deleting a cluster now invalidates both keys, deliberately and with a test — which is the argument for granular keys rather than against them: the dependency was always there, and it was invisible.

Route loaders remain the SSR entry point, but they write through the router's query client (context.queryClient.ensureQueryData) rather than returning data for the component to re-seed. One cache entry, not a server copy and a browser copy that drift.

All of these are load-bearing:

• The cache must be dehydrated into the SSR payload. @tanstack/react-router-ssr-query does it, wired in getRouter(). The router serializes loader return values, so a loader that fills the cache and returns nothing serializes nothing — the browser then hydrates against an empty cache, every useQuery starts at undefined, and every read the server just did is done again. Covered by the e2e test that blocks every /api call the browser could make before loading /app and expects the dashboard to draw — the SSR render is unaffected, because the web server dials the api off its own request rather than the page's.
• That test cannot tell server output from a client re-render, and there is a second one that can. It runs with JavaScript on, so a dashboard drawn from the hydrated cache after hydration looks exactly like one that arrived drawn. The TanStack DB spike (#42) reverted the whole /app shell to client rendering — server HTML halved, Sign out and every heading gone from it — and that test stayed green the entire time. So a second e2e test reads the raw /app response, which is the only place the distinction exists, and asserts three markers spread across the layout, the route and a section below the tables. A recoverable SSR error takes out the shell rather than one panel, so any one of them going missing is the signal.
• ensureQueryData does not refetch stale data. It resolves with cached data whenever there is any, stale or not, and fetches only when the entry is absent. So re-running a loader is not a refresh: router.invalidate() on the "api is unreachable" card produced zero requests, and the Retry button did nothing until a full page reload. Refetching the key is what retries.
• A key must mean one thing forever. ["policy", null] meant no cluster before one existed and cluster X after, because null resolves to "the first cluster" server-side. One entry, two answers. The dashboard loader resolves the id against the clusters key first, by the same rule the cluster bar uses, so the two cannot disagree either.
• A session change removes entries, it does not mark them stale. Signing in, signing out and switching org replace who is asking, so invalidateSession refetches what is mounted — the reader keeps seeing something — and drops what is not. Marking an unmounted entry stale would not stop the next loader from rendering the previous org's recommendations.
• initialData cannot seed a loader that re-runs — it fills only an empty entry, so after connecting a cluster the cache keeps the pre-connect value.
• The client is constructed exactly once, in getRouter(), carried in route context, and read back out by the provider. A second new QueryClient in the root component gives the loaders one cache and the components another.
• useQuery, not useSuspenseQuery. Sign out and the settings link unmount a cluster page, and a suspending child lets React hold the previous tree — a signed-out user kept looking at the signed-in dashboard.
• A loader WARMS; it does not make the reader wait. The router keeps the previous page mounted until a loader resolves, so an awaited warm is a click with no visible answer for the length of the reads — worst on the settings tab, whose databases read dials the customer's cluster. And it made the per-panel skeletons unreachable: awaiting fills the cache before the component mounts, so pending was never true on arrival and every outline written for that wait was dead code. The browser therefore fires the warms and returns; the server still awaits, because its render is the payload. This is the other half of the useQuery rule above — both exist so that a page arrives and its panels resolve independently, rather than the whole route waiting on its slowest read.
• The pending state does not exist without a pending component. router-core arms its timeout only when a pendingComponent or defaultPendingComponent is set, so defaultPendingMs is inert on its own — for a long time nothing here set either, and a blocking loader showed the page the reader was leaving for as long as it took. getRouter() sets one, and it is only for the loaders that still block on purpose: /app and the cluster layout read to decide where the reader goes, and a page rendered before that decision is one nobody asked for. It runs at 300ms rather than the framework's 1000 — under about that a navigation reads as instant, and defaultPendingMinMs (500) holds the outline once shown whether or not the data beat it.
• staleTime is 30s, and the reason is narrow: it is the only setting that decides whether mounting refetches, and mounting is what happens right after hydration. At 0 every page load re-fetched everything the server had just sent (measured: four browser calls per dashboard load, now zero). Half a minute is for changes someone else made; own mutations invalidate their key and refetch regardless. It stays under gcTime, or an inactive entry is collected while still counted fresh.

All of it is under src/lib/queries, which is the app's cache topology and nothing else:

lib/queries/
  client.ts     createAppQueryClient + invalidateSession
  keys.ts       every key in the app, one per api call
  shell.ts      clusters/org/orgs queries, useShell(), selectCluster()
  pipeline.ts   recommendations/roi/activity queries
  telemetry.ts  latency/latency-series/collections queries
  policy.ts     the policy query
  mutations/
    auth.ts   cluster.ts   org.ts   policy.ts   recommendations.ts

The files still group by what moves a key, because that is a useful thing for a reader to see in one place — but each one now exports a factory and a hook per endpoint, and nothing in it fetches two things at once. One factory per key, shared by the loader and the component that draws it: they were inline in both places, so each key was written twice — and a key written twice is two keys, one of which nothing ever fills.

Every write is a useMutation whose invalidation fires only when the api says something moved — a refused mode change has nothing to refetch, and a refused disconnect must not deselect a cluster that is still connected. isPending replaced the hand-rolled busy flags in the auth, connect and policy forms.

Each hook owns the key it invalidates. Components pass only the local state a mutation cannot know about (a form to close, an error to show), which is why ClusterConnection, OrgSection and PolicySection have no callback props: a member list should not have to know that leaving an org invalidates more than renaming one.

Values arrive with mutate(), not with the hook call. They used to arrive with the hook call, and it worked, because they lived in the component's useState and were therefore current on every render. They live in a form store now (§14.6), which deliberately does not re-render the component when a field changes — so a closure captured at render submits whatever was typed before the last keystroke. The values belong to the submit, which is the moment the form hands them over.

The UI is built from shadcn/ui components (components.json, new-york, generated with the CLI into src/components/ui). Anything interactive comes from there rather than raw elements: Button, Input, Label, Field, Card, Select, Checkbox, Separator, Alert, AlertDialog, Tooltip, Badge, Table, Empty, Skeleton, plus sonner for toasts. The one registry item deliberately not used is chart, which is a recharts wrapper — see §14.8. ~ is aliased to src in both tsconfig and Vite so future shadcn add output drops in unmodified.

Four deliberate deviations. Three are the repo's own standards (§16) applied to generated code. The sonner wrapper is rewritten to drop next-themes and its two as casts (the repo forbids assertions). Field drops the registry's role="group": a group with no accessible name announces nothing — the label names the control, not the wrapper — so it goes rather than being suppressed, and FieldSet, which is a real <fieldset>, is the component for grouping that is actually announced. And card headings stay real <h3>/<dt> elements rather than CardTitle on the landing page, because its heading outline is load-bearing for SEO.

The fourth is a product call rather than a standard: Label drops the registry's select-none. Upstream carries it because a label is a click target for its control, so a double-click that highlights a word instead of focusing the input reads as a misfire. That holds for a checkbox row and for nothing else here, where a label names a namespace, a username, a cluster id, a knob whose exact spelling the reader is about to grep for — and text that refuses to highlight reads as a broken page. It is passed back in at the one call site where the original reasoning survives, CheckboxField in components/form.tsx. SelectItem keeps its copy, which is the same decision rather than an exception to it: Radix commits an option on mouse pointerup, so a drag across one ends by choosing it and unmounting the listbox. The highlight could never last long enough to be copied, and in the org switcher the attempt to copy would have switched orgs on the way. Org names are selectable where they are actually read — the card title and the rename box on the settings page.

Every destructive or cluster-affecting action goes through ConfirmButton (AlertDialog) instead of window.confirm, so the dialog can show the exact consequence — the revoke command for a disconnect, what a drop will observe first, who loses access.

Every per-cluster read hook answers with Read<T> — { data, pending }, in lib/queries/read.ts — rather than the bare payload it used to.

The bare payload was a nicer API and it was the bug. isPending was discarded at the hook boundary, so nothing downstream could tell "there is nothing here" from "we have not been told yet", and both rendered the empty state. Our empty states make claims: "Nothing to review means nothing is obviously wrong" is a safety claim, and it was shown about clusters with forty proposals still in flight. "Nothing collected yet" told a reader their cluster had never been collected. 0 KB read as a measured zero. Then the data landed and the panel swapped — that content-shift is the flicker, and it is the same missing third state.

DataTable takes loading and draws skeleton rows inside its real header and <colgroup>, so the columns are already where the values will land; that is most of what makes a skeleton better than a spinner. Its filter box stays put but disabled, because a filter over rows that do not exist is a control that silently does nothing. The ROI figures, the two latency charts and the policy card each have their own pending branch — the policy skeleton deliberately draws no knob states, since an unticked "Instant create" is a claim about somebody's production cluster.

Two things this had to get right:

• Never on a refetch. Gated on isPending (no data ever), not isFetching. These queries poll and every mutation invalidates them; grey bars on a schedule would be worse than the bug.
• Never on the server. The loaders await every read, so nothing is pending during SSR and a cold navigation must show no skeleton at all. Both SSR e2e tests assert that directly — expect(html).not.toContain('data-slot="skeleton"'), and zero skeletons with every browser api call blocked. Worth noting the older markers had quietly stopped being enough: a skeleton draws the Policy card's heading too, so toContain("Policy") would have passed against a fully client-rendered page.

The empty states themselves are unchanged. This is about not showing them early.

The same argument again, for the state the one above missed. Read<T> is now { data, pending, failed, retry }, because a read that FAILED also fell back to the empty payload — so the reassuring copy above was being shown for a second reason nobody could see. Observed on a database two migrations behind its api: /recommendations 500ed, and the dashboard drew "No recommendations yet — nothing to review means nothing is obviously wrong" over a cluster holding 51 live proposals, with 0 KB and 0 recommendations awaiting review above it.

Two rules for the copy, and they are the whole design. Say whose problem it is — a reader looking at their own database has every reason to read a failure as being about the cluster, and it is not; the cluster was never dialled, and saying so is what stops a healthy server becoming a support ticket. Claim nothing about the data — no counts, no zero, no "none found". A stat card therefore withholds its figure and its caption, drawing an em dash for exactly the reason the pending skeleton beside it already exists: a figure is a measurement, and a failed read took none.

Two deliberate exceptions. failed is not set while a refetch of already-good data fails — react-query keeps the last payload, and blanking a populated table because one poll missed would be the same false claim pointing the other way. And the settings page's cluster-databases dial keeps drawing nothing, because that failure genuinely is about the cluster and is the state the rotation form underneath exists to fix.

/ is the only indexable page: static (no loader, no api calls), so it renders even when the control plane is down, and it carries the full meta set — title/description, canonical, Open Graph and Twitter card, plus SoftwareApplication and FAQPage JSON-LD emitted through the router's head scripts (entries are spread as element props: {type, children}, not {attrs}). The root route defaults to noindex, nofollow and the landing opts back in, so the dashboard and the password-reset page can never be indexed — new private routes inherit the safe default automatically. Canonical and og:url come from a constant in lib/site.ts, overridable at runtime with SITE_URL, so one image serves any domain and a preview host cannot make itself the canonical copy. robots.txt, the OG card and the favicon are static assets under apps/web/public/.

No sitemap.xml: with a single public URL it adds nothing a crawler cannot find from /, and a static one cannot know the deployment's host. Add one (or submit / directly) if the marketing site ever grows more pages.

The browser calls the api itself. One oRPC client, lib/api.ts, used from route loaders during SSR and from the query cache in the browser afterwards — the same query function on both sides, so nothing has to know where it is running. Three pieces differ, and each is a createIsomorphicFn the Start compiler resolves per build environment:

That last one is not fussiness. lib/metrics installs an OpenTelemetry provider at import time (§15.2), so a top-level import would leave the SDK in the client graph for tree-shaking to argue about; importing it inside the server branch means the client build deletes the branch and the import with it. The built bundle is checked for it.

This replaced 28 createServerFn wrappers across 13 call sites — 22 data relays in lib/app-server.ts and 6 auth relays in lib/auth.ts, every one of them a thin proxy from the web server to the api. The relay existed for one reason: the api was on another origin, so a browser calling it directly would have needed CORS with credentials and SameSite=None on the session cookie — which is, by definition, a cookie that rides along on cross-site requests. Putting both behind one origin removes the reason and the code.

What went with it:

• decodeOnce. The api percent-encodes the cookie value and setCookie encoded it again, so a base64 signature reached the browser double-escaped and every request after signing in was a 401 (issue #16, moot rather than done). The cookie the api sets is now the cookie the browser keeps.
• The { ok, message } envelope. A server function had to catch the api's throw and hand back a value; the hook then unwrapped it. The oRPC client throws ORPCError in the same process as the mutation, so onError does that work and lib/queries/errors.ts holds the one rule it applies — which statuses carry a message written for the reader (400/402/403/404/409, plus the ones a given call adds) and which get a generic one, because a 500 must not leak internals.
• One network hop per call, and the second copy of every contract type.

A dead read still renders an empty panel rather than an error page, but the mechanism moved: the queryFn no longer catches, so the failure is a query error and each read hook owns the stable empty value it falls back to (§14.2). The 401 case — signed out, as against "the api could not be asked" — is derived by useShell() across the three org-level queries rather than caught inside one.

Auth is better-auth/react's own client (lib/auth-client.ts) with no baseURL: it resolves one from window.location.origin and appends /api/auth, which is where the api serves it. Writing an origin there would be a second copy of the deployment's address to keep in step.

The browser calling the api needs both on one origin, and that is arranged twice over — the difference between them is a hop, not whether it works:

src/lib/api-passthrough.ts is the second row: intercepted in src/server.ts before the router sees it, since the router owns no /api route and would 404. One transparent forward, streamed rather than buffered so SSE (#22) needs no rewrite, with Set-Cookie passed through byte for byte — same origin in and out, so unlike the relay it replaced there is nothing to re-encode.

This was a deliberate reversal. The first cut made same-origin a deployment requirement and put the rule in four places: the ingress, an nginx service in compose, the vite dev proxy and a node proxy in front of the e2e suite. Moving it into the app deleted three of those four and turned "the operator must get this right" into "the operator may make this faster".

The compose nginx was kept for one round on the argument that it exercised the zero-hop shape. It did not, quite: nginx is not the ingress — different buffering, timeouts and header defaults — so it reproduced a proxy rather than ours, and it hid the passthrough from the stack developers run every day. It also had to grow websocket forwarding for HMR, and would have needed proxy_buffering off before SSE (#22). A second implementation of the routing rule, drifting on its own schedule. Removed.

What genuinely is not covered by any test is the ingress's own /api rule — kind-test.sh installs the chart with the ingress off and curls Services directly. That was true before the nginx container existed and is still true; the place to fix it is that script, not a different proxy in compose. The alternative considered and rejected was supporting genuinely split origins, which costs SameSite=None (the CSRF defence, given up by definition), CORS with credentials, an origin check the oRPC pipeline has never needed, and runtime api config in the browser bundle.

Two rules it enforces, both security-relevant:

• x-forwarded-* and x-real-ip from the client are stripped unless TRUST_PROXY=true. The api rate-limits per IP off those headers, so forwarding one a browser could have written lets a caller pick a fresh address per request and never reach a limit. Same switch and same reasoning as the api's own.
• Origin is never rewritten, because better-auth checks it against trustedOrigins.

It is observable on purpose: indexterity_web_requests_total{kind="api"} counts only what the passthrough answered, so a non-zero rate in a deployment that has an ingress /api rule is that rule not working — a hop being paid on every call that nothing else would report. The e2e suite runs with no proxy at all, which is what keeps this path from rotting: the proxy shape is covered by compose and the chart.

TanStack Form, validated against the api's own input schemas. Five forms — auth, connect-cluster, policy, team, reset-password — and between them they held around twenty useState calls; the sign-in form alone held seven for one form, one of which was a busy flag that had to be cleared on all four exits from the submit handler, including the early returns.

Replacing the state was the smaller half. The larger half is that client-side validation did not exist: the form submitted whatever was typed and relayed the api's 400. The rules had only ever been written on the api, as inline z.object({ ... }) literals inside contract.ts, so the fix was to name them rather than to write them again:

packages/contracts/src/
  schemas.ts    what the api returns
  inputs.ts     what it accepts — named, and composed into route inputs by
                contract.ts, so there is exactly one definition of each rule

A field validates against the object the route validates against:

<form.AppField name="password" validators={{ onChange: signInInput.shape.password }}>
  {(field) => <field.TextField label="Password" type="password" />}
</form.AppField>

Not a second copy of the rule — the same one. Which is also why the messages in inputs.ts are written to be read by a person standing in front of the field: they are now what the api says in a 400 and what appears under the box, and zod's default ("Too small: expected string to have >=1 characters") is wrong for a label even when it is right about the value.

better-auth is included by inversion. Its minPasswordLength happens to default to 8, which is what the sign-up field needs to enforce — but a default that happens to agree is not a shared rule, so auth.config.ts now sets it from PASSWORD_MIN_LENGTH in inputs.ts. Move the constant and both move.

src/components/form.tsx is TanStack Form bound once to shadcn's Field primitives, exporting a useAppForm with pre-bound TextField, NumberField, CheckboxField and SubmitButton. Each field is then one line instead of ten, and there is one place that can forget an aria-invalid.

shadcn's own form component is not usable here — it is react-hook-form's Controller with a wrapper around it. field (Field / FieldLabel / FieldDescription / FieldError) is the form-library-agnostic half of the same design, and is what shadcn's own TanStack Form guide composes against.

• The submit button is never disabled for being invalid. Gating it on canSubmit is the obvious wiring and the wrong one: handleSubmit already refuses an invalid form, and it marks every field touched on the way, which is what makes the errors appear at all. Gate the button and the policy form's paired change-window hours reach a state where the button is grey, nothing is highlighted, and no click will explain why — because filling one hour invalidates the other before the reader has been anywhere near it. A disabled button that cannot say why is worse than a button that refuses out loud.
• A half-set change window is refused, not silently completed. The old code nulled whichever hour had no partner, so a number the owner had just typed vanished out of the box with no reason given.

Errors are shown only once a field isTouched, so an empty untouched form is not a page of complaints; submit touches everything, so one click still lights up whatever is wrong. The one rule the api cannot own is the reset page's "repeat it" — the api is only ever sent a single password — so it is a field validator with onChangeListenTo: ["password"], which re-checks the match when the first box changes and clears a mismatch the reader has already fixed.

TanStack Table for the logic, the same shadcn primitives for the markup. The three tables that were on the dashboard — recommendations, per-collection footprint, and the audit trail — were <table> markup with the rows mapped inline. The index inventory (§8.2) and the scanning workload (§8.3) are the fourth and fifth, and both were written against this from the start. No sorting, no filtering, a header row that had to be kept in step with the body's by hand, and a 379-line route that was mostly cells. A cluster with fifty recommendations arrived in whatever order the api returned them, which is the one order nobody wants.

Headless means this is not a visual rewrite: the same Table / TableRow / TableCell come out the other side. What changes is that a column is one object saying how to read the value, how to draw it and how to sort it, instead of a header cell in one place and a body cell in another.

components/data-table.tsx        the feature set, the column helper, the component
components/app/recommendations-table.tsx   column defs + the per-state action
components/app/collections-table.tsx       column defs + the two-read row merge
components/app/activity-table.tsx          column defs
components/app/index-table.tsx             column defs + the per-engine flag wording
components/app/workload-table.tsx          column defs + the ESR line and the outcome tooltip

9.0.0 is the latest tag and the repo's standing rule is latest stable (§2.6), but the deciding argument is v9's opt-in feature registry: what a table can do is settled statically, in one tableFeatures({ ... }) outside any component, so the grouping and row selection none of these three use are not in the bundle. Registering a feature to call its getter is paying for it twice, which is why rows are walked with getAllCells() rather than getVisibleCells() — the latter belongs to columnVisibilityFeature, and nothing here hides a column.

Pagination joined the registry in Sep 2026 (D133), and it is the one feature that is not additive on its own. rowPaginationFeature defaults PaginationState to { pageIndex: 0, pageSize: 10 }, so registering it and nothing else would have quietly cut the recommendations and footprint tables — which page nothing and pass no pagination props — to ten rows each. Every table therefore passes manualPagination: true unconditionally. Under it the slicing function is never called and every row is returned, so the feature contributes the page arithmetic (getPageCount, getCanNextPage, setPageIndex, setPageSize) and no slicing at all, which is exactly what is wanted when the server owns the page (§8.2).

One more v9 shape worth knowing: onPaginationChange is handed an updater, not a value — both setPageIndex and setPageSize call it with a function of the previous state, and setPageSize's recomputes the page index so the row at the top of the page stays in view. DataTable resolves it against the caller's own state before calling back, so a caller receives the state it is about to request and there is no table-held copy to drift from it.

The costs are real and worth writing down. The integration had to be written from the type definitions rather than from any published example — shadcn's data-table page and this repo's vendored tanstack-table skill both document v8. And v9 types a column against its feature set, so the column helper cannot be created without naming that set (dashboardColumns<TData>() exists to stop every table repeating it), and a heterogeneous column list needs helper.columns([...]), which threads each column's own value type out through a variadic tuple. A plain array widens them all and stops typechecking — the alternative being ColumnDef<TData, any>[], which the repo forbids.

• A default sort per table, chosen from what the table is for. Recommendations by score descending (the engine's confidence is the reason to look), collections by index footprint descending (the table answers "where is the space going"), the trail newest-first. And the toggle cannot cycle back to unsorted: that state is the api's arbitrary order, and no reader asks for it on purpose.
• A missing number is not zero. An unmeasured collection sorts as -1, so it does not rank beside a genuinely idle one — "not collected" and "zero bytes" are different facts and a ranking that merges them is lying.
• Empty is two different facts. Nothing collected yet gets a shadcn Empty saying what will appear and when; a filter that excluded everything gets a row saying so, quoting what was typed. The old markup drew a bare header over both.
• Filtering is global per table, not per column. One input rather than eight, and it matches the questions actually asked: "what happened to this index", "which of these failed", "where did that collection go".

Sorting is announced with aria-sort on the header cell, so the state reaches a screen reader rather than living only in an arrow icon.

Recommendations is collections × indexes; the per-collection footprint is one row per collection. Neither listRecommendations nor getCollections applies a limit, so both are bounded only by how big the customer is. Both take a virtualize prop on DataTable; the audit trail does not, because the api already caps it at fifty.

Three things about how are the decision, not the library.

• Spacer rows, not absolute positioning. The library's own table examples take each <tr> out of the flow and place it with transform. Two things break when you do: a row out of flow stops sharing the table's column widths, so the columns lose alignment the moment you scroll, and it stops being a row to a screen reader, which is most of what a <table> is for. Two aria-hidden spacer rows, one above the window and one below, at the height the skipped rows would have taken, buy the same DOM saving and cost the table nothing. getAllByRole skips aria-hidden nodes, so every existing order and filter test kept working unchanged — the same fact from the test side.
• The height is a maximum. A cluster with four collections renders at its natural height, with no scrollbar, looking exactly as it did. Only the long tail caps and scrolls, and only the long tail was ever the problem. The header goes sticky only when the body scrolls under it.
• initialRect is load-bearing. The server has no element to measure, and without it the virtualizer reports a viewport of zero and renders no rows — which is precisely how a virtualized table silently stops being server-rendered, the failure the TanStack DB spike had just demonstrated (§14.2). Handing it the container's own maximum gives the server a screenful, and the client's first render agrees because it starts from the same number. There is a renderToString test asserting the first row is in the HTML and the last is not; the e2e SSR test cannot see this, because the e2e cluster has no collected rows to draw.

Row heights are measured rather than assumed — a recommendation's score cell grows a second line once a drop is scheduled, and its rationale is prose that wraps — so estimateRowHeight is only the opening guess and measureElement corrects it.

What this does not fix — and what did. Virtualizing is a render-cost fix, so the payload stayed whole; #64 measured it and capped it (D47). On a seeded worst case — 200 collections × 90 days of hourly readings, 4,000 indexes each with a proposal — getLatencySeries shipped 30.9 MB in 4.2 s to draw four collections, listRecommendations 1.86 MB, getCollections 55 KB. So the series takes a 30-day window plus a top-N by evidence, with totalCollections beside them (the before/after summary still reads the full plan window). That top-N is ranked once per metric and unioned, which is the correction #85 needed a third time: every collection is read on the same cadence, so all of them carry the same point count, a single ranking is a total tie, and the slice fell to postgres's row order. On a live cluster that handed the write chart eight collections nobody writes to, and it reported the cut's blind spot as an absence of writes — the very sentence latency-series.ts had been split out to stop it saying, from one layer below where the choice was actually made. Half the cap per metric so the payload does not double, ranked on drawable points rather than readings, tie-broken on namespace so the same cluster charts the same collections every load.

Both latency reads are folded in postgres since #614. The payload the browser received was bounded, but the database read behind it was not. The summary selected every reading of every collection over the plan's whole window, and the series did the same over thirty days to keep eight collections. On the hosted deployment's busiest MongoDB cluster that was 2.8 MB plus 2.0 MB a load, measured with EXPLAIN (ANALYZE, SERIALIZE). It was read again on every collect while the page was open, against a free plan allowing 5 GB of egress a month.

• The summary is one row per collection: the first and the last measurable window of each metric, picked out with lag() exactly as summarizeLatency picks them from the rows (insights/latency-folds.ts). That is 11 kB.
• The series is ranked before any reading is read. chartableNamespaces ranks on counts (drawable read points, drawable write points, all points), which postgres folds in 7 kB. The readings are then read for the chosen collections only.
• The rules stay in analysis/latency.ts. An integration test holds each query to the function it reproduces, as classify's evidence is held (#485). The recommendations read returns the 500 highest-scoring open ones — proposed, or on their way through hide, observe and build — ordered in SQL, because a cap without an order is a random sample, plus a total the table prints as "showing the 500 highest-scoring of 4,000". What is settled (built, dropped, rolled back, turned down) is not in it (D185): a built index used to sit in the same table as a proposal to build one, and settled rows scoring higher than the open ones could push new proposals past the cap. Those are the Activity trail's rows, and Parked's for a turned-down drop; a drop is undone from its row in the trail, which carries an undoable flag for exactly that. The total is load-bearing: the sorting and filtering below it are still client-side (D33), so without it the filter would claim authority over rows never sent. getCollections is left unbounded on the measurement. Server-side sort/filter/cursor params were priced and declined — they make the filter box a debounced round trip and move every default order into the api — and remain the right shape the day a customer needs row 3,000.

One consequence outside the app: jsdom performs no layout and reports offsetHeight/offsetWidth as 0, which are the exact two properties the virtualizer measures — for the container's viewport and for each row. Left at zero, a virtualized table sees a viewport of nothing and renders no rows, and every test about sorting or filtering fails on markup that is correct in a browser. So vitest.setup.ts shims both at a plausible scale, a 600px container over 40px rows, alongside the ResizeObserver and pointer-capture shims already there.

TanStack Charts, pre-1.0, behind a wrapper. The two latency panels are the only charts in the app. They were hand-rolled SVG, then recharts through shadcn's chart wrapper, and are now @tanstack/charts — a grammar-of-graphics library where a chart is a definition (marks, channels, scales) rather than a tree of components.

The alternative was staying on shadcn Charts, which was the safe answer: recharts underneath, stable, already themed to the tokens, already installed. Two things decided against it.

• Size. Removing recharts took the client bundle from 360,985 to 294,441 gzipped bytes — 65 KiB, 18%. Measured by building both branches, not taken from the library's own consumer-size figure; recharts pulls a set of d3 subpackages with it.
• SSR. recharts 3 draws client-side only, so the panels shipped as reserved empty boxes and filled in after hydration. A TanStack Charts scene is SVG computed from a known size, so the server draws the lines. There is a renderToString test for exactly this, because losing it would be a silent regression back to a chart that pops in.

0.6.5 at adoption, and the issue proposing it was written against 0.3.1 — that is the churn rate as a measured fact rather than a hypothetical. So the rule is that nothing about marks, channels or scales leaves one file:

components/latency-chart.tsx    LineChart({ title, unit, series, pending, emptyNote })

Every one of those is about what to say, never about how to draw it — a breaking release is one file to fix, and the callers cannot notice.

pending and emptyNote are the same lesson learned twice. "Not enough samples yet" was once shown while the first fetch was still out, so the panel made a claim about the cluster before anyone had asked it anything; pending split those apart. Then the sentence turned out to cover three different facts — the second collect has not run, the counter never moved, or a mongod restart made the window unmeasurable — and a write chart that had never been able to see writes rendered identically to one on an idle collection. That is what got the empty write panel reported twice against a chart that was working.

So the api ships a readGap / writeGap per collection alongside the points, read off the same conditions the windowed average nulls on, so the two cannot drift. latency-series.ts folds them into one sentence per chart — a restart outranks a quiet counter, which outranks waiting on a second collect — and only for a chart with no line on it, since a chart that drew something has already said its piece. A cluster with no latency history at all still renders nothing rather than two boxes of prose.

• The wide-row pivot. recharts wants one row per timestamp with a column per series; a mark takes tidy rows, one per series per timestamp. toRows is gone.
• Label slugging. It existed only because a db.coll dataKey read as a nested path in recharts. Series are keyed by a z channel now, dots and all.
• The client-only container. components/ui/chart.tsx was shadcn's ResponsiveContainer plus a tooltip; it went with recharts. A future shadcn add chart would bring it back, which is worth knowing before running one.

Nulls still break a line into a gap rather than reading as zero — y accepts number | null directly, so nothing has to be filtered first. The palette is still the validated one (§14.3's dataviz note), passed as an explicit colour domain and range so a series appearing or disappearing between collects cannot repaint the ones beside it.

The dashboard used to change only when the reader reloaded or a mutation invalidated a key — a pipeline pass that produced new recommendations was invisible until then. It follows the pipeline live now, and the whole feature is two small mechanisms glued to what already existed:

One stream per shown cluster. listClusterEvents is a contract procedure like any other — GET /clusters/{id}/events, output eventIterator(clusterEvent) — so oRPC speaks SSE for it end to end: the api handler is an async generator, OpenAPILink hands the browser an async iterator, and the session cookie rides the same first-party GET as every other read (D31: no relay). The api checks ownership once, at connect, then ends every stream after five minutes on purpose — the reconnect re-runs the tenancy check, so a revoked member stops hearing a cluster on the same clock as the session cookie cache (§10). The client treats a clean end as routine and reconnects at the floor delay; only failures back off, and a 401/403/404 ends the subscription for good.

Only while the tab is shown (#548, D171). Each reconnect queries postgres, and the listener below holds a session while any stream is open. So a tab left open in the background, re-subscribing every five minutes, kept a database that suspends after five idle minutes awake for as long as the tab existed. A hidden tab now closes its stream. Showing it again reopens the stream and invalidates every key any event could move. That key list is derived from the contract's event kinds and tasks, so a new one is covered by being added. The refetch is required rather than polite: events sent while the stream was closed are gone, and TanStack Query's own focus refetch reaches only mounted queries past their 30 s staleTime.

Postgres NOTIFY between the replicas. An in-memory emitter would emit into one process and silently reach the subscribers of another, so the transport is the one dependency every replica already shares (D10): the pipeline runs pg_notify('cluster_events', …) right after the state it announces, every api replica holds one dedicated LISTEN connection (a pg.Client, not a pooled session — LISTEN binds to the session), and fans out in memory to its own subscribers. In a single replica the NOTIFY simply arrives on the process that sent it.

That connection is LAZY, and it is the last thing that made an idle api hold a database open (#223). It used to be opened in onApplicationBootstrap and held for the life of the process — defensible while the api was assumed always-on, and the whole remaining problem once the schedule stopped needing a resident process, because it was then measurably the ONLY thing an idle api held. Forty seconds idle and no dashboard open, pg_stat_activity showed exactly one client backend, idle on ClientRead, running listen cluster_events: the request pool and better-auth's pool both release their idle clients, and the cluster-connection sweeper is lazy and unref'd. It now opens on the first subscriber and closes thirty seconds after the last one leaves — same measurement after: zero backends, with the api still answering /api/health.

Thirty seconds rather than zero because a stream is capped at five minutes so ownership is re-checked (above), so every open dashboard re-subscribes on a schedule; tearing the session down on each last-unsubscribe would rebuild the LISTEN every five minutes per dashboard, which is more churn than the eager listener ever caused. What makes disconnecting SAFE is a property the design already had: events sent while the listener is down are lost deliberately, since an event is a nudge to refetch and staleTime covers gaps — so a listener that is down loses exactly what one that is not yet up loses, and both were already correct. The claim is asserted against pg_stat_activity in the integration suite rather than against the service's own view of itself, and the nothing-at-boot case runs first in the file on purpose: any case that subscribes leaves the session up for its grace window (D87).

A transaction-pooled DATABASE_URL breaks all of this silently, so the api refuses to boot on one (#233). PgBouncer, Supavisor and Neon's -pooler endpoint accept LISTEN and then land the listener and the notifier on different backends: the statement succeeds, the notification never arrives, nothing errors. Measured on a real Neon project — same project, same credentials, only the host differing — pooled lost the NOTIFY and direct delivered it, with max_connections reading 901 on both; measured again against PgBouncer 1.25.2 in front of postgres 18, where pool_mode=transaction lost it and pool_mode=session delivered it. The property that breaks is transaction pooling, not a proxy in the path — and the pooler buys nothing in exchange, since this app holds roughly ten connections at defaults. What it costs is this whole section: the panels fall back to staleTime with no polling fallback in the client.

So main.ts runs a NOTIFY round trip before serving anything (db/notify-probe.ts): LISTEN a throwaway channel on one dedicated client, pg_notify from a second, wait two seconds, close both — open-probe-close, because #223 made the listener lazy precisely so an idle api holds zero sessions. Three attempts with a short backoff, so a restarting postgres is a slow boot rather than a crashloop, and the refusal distinguishes "connected, LISTEN accepted, nothing arrived" from "never got a connection", because those call for opposite actions. The hostname (-pooler) is only ever a hint; the round trip is the test. DATABASE_URL is thereby validated twice: as a string by the schema, then PROVEN at boot — a pooled URL is well-formed and still wrong (D89).

The events are signals, not state: { kind, task }, no rows. The client answers one with queryClient.invalidateQueries on the keys that pass actually moves (lib/queries/live.ts — the mapping is a pure function with a test per row, same discipline as the mutations' named invalidations). Four kinds:

The transition events land mid-pass so the row moves when it moves; the pass event closing the same keys behind them dedupes into the refetch already in flight.

What this is not: not a second copy of the dashboard's state (the note in #12 — reacting to an event is one invalidateQueries line), not TanStack DB (D34's neighbour decision — a live collection earns its place when the dashboard grows client-side joins across live collections, which it has not), and not delivery-guaranteed. An emission is a best-effort nudge sent after the write it announces; a lost one costs freshness until staleTime or the next event, never correctness. That is also why the emit helper swallows failures: a pass that landed must not retry — and re-run executor.hide — because a notification could not be sent.

Operational notes, each the residue of a real hazard: oRPC's keepalive frames (5s) keep proxies from timing the stream out and give the integration test a read that always resolves; the SSE route's onResponse duration is excluded from the latency histogram (a stream's elapsedTime is its lifetime, and five healthy minutes would own the p99); and the listener service aborts every open stream in onModuleDestroy — module-destroy hooks run before Fastify's close, which waits on active requests, so without it an open dashboard would hold SIGTERM until the runtime SIGKILLs.

• Separate Dockerfiles — apps/api/Dockerfile and apps/web/Dockerfile (multi-stage), so api and web deploy independently.
• deploy/all-in-one/Dockerfile — the same two builds in one image, both processes under a supervisor (§15.0).
• The images ship no node_modules (D182). Every entrypoint the api's images run — main.js, and the migrate.js, rotate-key.js and set-plan.js CLIs — is one esbuild bundle (apps/api/scripts/bundle.mts), so the runtime stages copy those four files, apps/api/package.json (version.ts reads it) and apps/api/drizzle/ (the migrator does), and nothing else. The all-in-one adds the dashboard's .output, which nitro makes self-contained, the tunnel binary and the supervisor. Before that, the production tree was reinstalled from stripped manifests (D73, D77), because --omit=dev does nothing after turbo prune --docker: the regenerated lockfile carries no dev: true markers. With Sentry 11's build tooling in it, that tree had reached 305 MB.
• What guards it. A package the bundles still need at run time is a MODULE_NOT_FOUND that only an image shows, so the integration suite runs the bundles from a copy staged outside the repository (integration/global-setup.ts) rather than from dist/, where the repository's node_modules would answer for them.
• The base is node:26-alpine, which is most of each image: 181 MB against node:26-slim's 252, so 71 MB off all three. A base swap rather than a musl rebuild, because nothing shipped links against the C library: the bundles are JavaScript, there is no .node file anywhere, and the tunnel binary is static (CGO off). What had to be tested rather than reasoned about is the resolver: this product dials customer clusters by hostname and mongodb+srv:// is an SRV lookup, so all three mongo majors' integration suites were run on musl through a real mongodb+srv:// string (D76). The images are api 216 MB and all-in-one 238 MB (D182), the all-in-one being the api image plus the dashboard's .output and the tunnel binary; web was 193 MB when last measured (D76).
• apps/agent/Dockerfile — the distributable agent (phase 2).
• docker-compose (dev) — postgres + api + web, with hot reload via bind mounts and Turbo watch. PostgreSQL runs in compose. No proxy container: the web server answers /api itself (§14.5), so localhost:3000 is one origin without one.

The api runs the whole job pipeline itself (#231, #232): no resident runner, no LISTEN "jobs:insert", no separate worker process. The chart's topology value only decides how the two workloads are packaged; the code is identical in each.

One tick function, two triggers (#229). The schedule is occurrence arithmetic (BURST_SCHEDULE, duePasses), and a tick is claimDuePasses — enqueue what became due — followed by one runOnce({ pgPool }) drain against the api's own pool. runOnce() picks up the per-cluster jobs those passes fan out inside the same run, to any depth — measured on postgres 17 with graphile-worker 0.17.3 before the design was committed to, because without it a tick would dispatch and exit before executing anything. Two things trigger it:

• An in-process interval, default on (RUN_CRONJOB=true). A plain 30-second setInterval — deliberately not a cron library, because the occurrence maths does the timing and the timer only has to be finer-grained than the tightest pass (FAST_PASS_INTERVAL_MINUTES, five by default). One tick fires at boot, so a fresh install collects at once and a host that slept catches up immediately — owing exactly ONE run per pass, however long the sleep. Replicas are safe: a pass is claimed against its occurrence in worker_watermarks, so concurrent tickers dispatch each occurrence exactly once, which is what retired the chart's old one-replica cap.

• GET /api/internal/tick (RUN_CRONJOB=false), for an external clock. The request runs the same tick — claim, enqueue, drain — because nothing LISTENs any more: if this request did not drain, nothing would. It is bounded at 25 seconds, under the 30-60 s platform proxy floors: past the deadline the response says "drained": false while the drain carries on in-process, and pinging again resumes the remaining queue without duplicating anything — a partially drained queue is a RESUMABLE state, because occurrences are claimed and jobs are keyed. CRON_TRIGGER_SECRET is REQUIRED at boot in this mode: the endpoint is then the only thing that can start a pass, and a deployment without it is a pipeline that is silently dead. The token is compared in constant time over a SHA-256 of both sides, since timingSafeEqual throws on a length mismatch and throwing on the wrong length leaks the length (D88).

  GET, and only GET — POST is not routed and answers 404 (#315). The endpoint changes state, so by the letter of HTTP it wanted POST, and that is what it was; the verb turned out to be the thing standing between an operator and a working schedule. The audience for this flag is the host that sleeps, and what such a host comes with is a URL pinger — an uptime monitor, a platform cron field that takes a URL, a status-page checker — most of which send GET and nothing else, so answering only POST left those operators running a second machine with curl purely to re-spell the request. The change is a method swap rather than a redesign because both halves of safe-method semantics were already satisfied, by decisions made for other reasons: an intermediary may repeat the request, which is free here by construction since occurrence claims mean a prefetch or a double-send cannot dispatch twice; and it may reuse the answer, which is refused outright, because security-headers.ts puts cache-control: no-store, max-age=0 on every response this api sends. That header is the load-bearing one — a cached 200 is the worst failure available here, a scheduler being told the pipeline ran while nothing runs at all, and it is silent — so the integration suite asserts it on this route rather than trusting the global hook. POST is not kept alongside: two verbs for one action doubles what a misconfiguration can look like, and a stale POST scheduler getting a 404 is loud, immediate and names what to change.

A pass is claimed against its most recent occurrence, not against an elapsed interval, and that one choice buys three properties: two triggers firing a minute apart compute the same occurrence so only one claim wins (no lock); a host that slept through eleven buckets owes ONE run per pass rather than eleven; and retention stays on 03:00 with the weekly digest on Monday 09:00, which an interval reading would have drifted (D86).

Hosts that sleep are a supported topology, with the caveats named. On a host that sleeps when idle, or a database that suspends its compute, a resident schedule does not slow down — it stops. The external clock is the answer: any scheduler that can fetch a URL with a bearer header drives the pipeline in request-sized bites, and the work happens inside the request window, which is exactly what a platform that freezes the container after the response needs. The one loss on such a platform is the post-deadline tail of a long drain — a frozen mid-flight job waits out graphile-worker's lock before retrying — so a ping cadence of a few minutes keeps that tail short. Ticked every fifteen minutes, a five-minute pass IS a fifteen-minute pass; that is a latency question, not a safety one, since the observe windows are measured in days.

On-demand work rides the same tick: the collect button and an approval enqueue, and the next drain — at most 30 seconds away with the interval, or the ping cadence without it — picks them up.

Every drain first releases locks a dead worker is still holding (#253). graphile-worker serialises per NAMED QUEUE — collect:<cluster>, probe:<cluster>, one per pass per cluster — and a queue is claimable only while _private_job_queues.is_available, which is a stored generated column reading locked_at IS NULL. There is no time term in it anywhere in the claim path. A worker that dies mid-job (SIGKILL, OOM, a node drain, a hot reload in dev) therefore leaves that one queue locked, and every later job on it is never claimed again: that cluster silently stops being collected, and because the jobs sit at attempts = 0 they never dead-letter, so the owner alert never fires either. Graceful shutdown is unaffected — beforeApplicationShutdown already lets the drain finish, which returns the jobs properly.

The library clears these in resetLockedAt, on a timer owned by the resident run() path. Retiring that path left runOnce — runOnceInternal → _runTaskList(…, continuous: false) — which never reaches the timer, so from #231/#232 until #253 nothing reset anything. jobs/locks.ts is that reset, run by the one thing that still drains, claimed on the five-minute occurrence grid so replicas do not duplicate it and logged at WARN per freed queue — reaching it means something died holding that queue, and the log line is the only trace left of it.

The threshold is the library's four hours for anything unbudgeted, and three budgets — fifteen minutes by default — for the read-only passes that have one (#412, D118). Four hours is a ceiling on how long a wedged queue stays wedged, and the other side of the trade is worse where it applies: a large apply legitimately runs for tens of minutes, and a reset that fires while its worker is still working hands the same job to a second one. A BUDGETED pass cannot legitimately still be running past its own budget, which is what lets the shorter interval be safe.

The repair runs on the DISPATCH path, and ahead of the dispatch (#460, D140). It used to be the first statement of drainOnce, which reads as "before the drain" and is not: drains are serialised through drainChain, and a drain that never returns is the normal state of a busy fleet — runOnce executes work enqueued during its own run to any depth, and the tick enqueues more every five minutes, so the same runOnce keeps going. Measured on the hosted deployment: one pool in flight for five hours, pass:resetLocks frozen for all five while pass:scheduleProbe advanced on every tick, and an orphaned classify queue lock standing the whole time. The repair was starved by a busy queue, which is the only condition it exists for; enqueueDue never was, because it runs on the caller's own path — and that is what made the two watermarks disagree and the diagnosis possible.

Ahead of the dispatch rather than merely beside it, because since #454 the dispatcher reads locked_at to decide whether a pass is already running for a cluster and stands down if it is (D138). Correct while a pass is running; a deadlock when the lock is orphaned, since nothing re-adds the key and so nothing ever notices. Freeing it first means the tick that repairs the queue is the tick that re-dispatches what it was holding. It is still not a scheduled pass — the thing it repairs is the queue, so a pass would be one drain behind at best — and it still claims the fast passes' occurrence (five minutes by default) so replicas do not duplicate it.

With the SSE listener lazy (§14.9, D87) and no interval (RUN_CRONJOB=false), an idle api holds no postgres connection at all and issues no queries: a database which suspends on idle actually suspends, rather than merely being asked to. That holds between pings, and a ping is a query, so a five-minute clock against Neon's five-minute suspend delay left no gap to sleep in. FAST_PASS_INTERVAL_MINUTES (D170, #546) closes that gap from the other side. Raised above five, it spaces the fast passes out, and a ping that provably has nothing to do answers "idle": true from memory. Proving that takes four things:

• a claim round has completed since boot;
• no occurrence has moved since that round;
• the last drain finished;
• nothing reached the queue that the schedule cannot see, meaning a request handler's collect or a failed attempt awaiting its retry.

The skip is gated on the raised cadence rather than always on, because what it cannot see (another replica's collect, a retry that outlives the next ping, a watermark edited underneath it) then waits for the next occurrence rather than the next ping. That trade belongs to an operator who chose it. The alert cooldown lives in worker_watermarks rather than process memory for the same family of reasons — restarts and replicas cannot re-mail a cluster's owners on every tick of a bad day (D86) — and the claim it writes is released back to a five-minute window when the mail it was taken for reached no transport at all (D121).

What makes this cheap is §14.5. The dashboard server already answers /api by forwarding it, so one origin is a property of the app rather than of the deployment — which means merging the tiers needs no routing change. The -api and -web Services keep their names, ports and labels in every topology; they simply select the same pod, and the ingress, the ServiceMonitors and every in-cluster caller are unaffected. deploy/kind-test.sh runs the same assertions against each packaging, which is what holds that claim: they reach both apps through their Services and know nothing about what is behind them.

Two things do not survive the merge unchanged, and both follow from one network namespace:

• The web metrics listener moves to metrics.port + 1. Two listeners cannot bind one port. Both Services still publish 9464, so scrapers read the same number as before; only the containerPort moved.
• Port names are unique across a pod's containers, so the dashboard's are web-http and web-metrics in every topology rather than conditionally.

single-container is the one for a host that has no pods — Fly, Render, Cloud Run, docker run. Its supervisor (deploy/all-in-one/supervisor.ts) exists because one environment has to describe two processes, and because a container is alive or it is not:

• It splits the two variables they would fight over. Both read METRICS_PORT (one namespace) and SENTRY_DSN (two projects), so the supervisor hands the web server WEB_METRICS_PORT and WEB_SENTRY_DSN under the names it already reads. Neither is declared by any schema, which is why homes.test.ts names them explicitly (§15.1).
• It forwards SIGTERM to every child (the api, the dashboard, and the tunnel service when TUNNEL_PORT is set). sh -c "a & b" does not, so the api would never run its shutdown hooks and the runtime would SIGKILL the container ten seconds later. All at once is safe because the tunnel service drains: it keeps the open peerings until the api has closed them (§4, #649).
• It exits non-zero when either process dies, so the host restarts the container instead of leaving a dashboard serving 502s from a passthrough with nothing behind it.

They stay two processes on purpose. Mounting the nitro handler inside Fastify would mean bridging a WHATWG fetch handler onto it, re-implementing the static layer that serves the built assets and their headers, and streaming SSR through the seam — for a container count that is already one.

Sockets are a resource the same way memory is, and both defaults here were the library's rather than a decision.

Postgres. Two pools live in one api process — the request pool (DatabaseService, which the tick's drains share since #231) and better-auth's — and new Pool({ connectionString }) was accepting pg's default of 10 for each. Each backend costs memory on the database server, and a self-hosted postgres ships max_connections=100. max is a required argument to createDatabase, from PG_POOL_MAX (5) for the long-lived pools and 2 for the one-shot commands (migrate, rotate-key, set-plan).

They stay separate, and that is the interesting part. Merging them behind a shared registry was the first instinct and it is wrong: one pool lets a slow report starve a sign-in of a connection. Bulkheads are the reason, not an accident. The drain is capped at min(WORKER_CONCURRENCY, PG_POOL_MAX - 1) for the same reason from the other side — a drain at full fan-out must never be able to starve every request of a connection at once.

Too low is latency rather than failure: pg queues a request until a connection frees, so the symptom of a wrong number here is slow, not broken. There is no idle poll left to price (D72, revised): the pipeline drains on a tick, and between ticks nothing queries the jobs table at any interval.

MongoDB. maxPoolSize was unset, so the driver took its own default of 100 per client — and a session is held per connected cluster (jobs/connection-pool.ts, refcounted, swept after five idle minutes), so the ceiling multiplied by the fleet. MONGO_MAX_POOL_SIZE (10) bounds it, with maxIdleTimeMS at 60s so the pool returns what a busy moment needed instead of holding it for the life of the session. Ten is well clear of what the collectors ask for: their fan-outs are per replica-set member and each member has its own client, so the widest concurrent use of a single connection is the five reads in mongo/snapshots.ts. The stronger argument is not our memory though — those sockets are spent on a database we do not own.

How the database reaches the job pipeline is the wiring question underneath all of this. It used to be a module-level jobDb() singleton that fifteen call sites reached into; it is now an argument — createTaskList(db) at the composition root → each task — so the pool's lifetime belongs to whoever composed the pipeline rather than to whichever module was imported first. createAuth likewise takes a db instead of building one, which is what moves "auth gets its own pool" to a place a reader can see it. ClusterTaskDeps deliberately does not carry the database: nothing reads it there, and that interface's value is being three functions wide.

Measured, not estimated, against the published images.

V8 sizes its heap from the container's limit rather than the host's memory, and grows into that ceiling happily — collecting only as it approaches. So RSS follows the limit, not the workload: the api image idles at 228 MB on a 16 GB machine with no limit and 104 MB under a 512 MiB one, doing the same nothing in both. The practical rule is the opposite of the instinct — raising a limit raises usage, so a limit is raised when something is OOMKilled, not in anticipation.

But that sizing stops scaling below 512 MiB. The default ceiling goes 536 MB at 1 GiB, 268 MB at 512 MiB, and then flattens: 384 MiB, 320 MiB, 256 MiB, 192 MiB and 96 MiB all produce ~262 MB. At 256 MiB and below the process believes it may hold more heap than its cgroup allows, so a heap-heavy pass is killed for memory where it should have been collected. The chart caps each container's old space at 65% of that container's own limit — derived from the limit, so changing one is enough — and leaves the rest for what is not heap.

What is not heap matters here, because two of this product's spikes live outside it:

• Password hashing is memory-hard by design. scrypt allocates roughly 32 MB per hash in flight, outside the V8 heap, bounded by libuv's threadpool (four by default) rather than by AUTH_RATE_LIMIT_MAX — which limits the rate, not the concurrency. It is not a knob to turn down: the cost is the brute-force resistance. The api's limit is the largest of the three for this reason.
• WORKER_CONCURRENCY multiplies rather than shares, since each concurrent job holds a cluster's statistics while it runs. It defaults to 1.

single-container is the one case the chart does not cap, because one cap would be handed to both processes; its supervisor divides the budget at runtime instead (§15.0). single-pod needs no split — Kubernetes gives each container in a pod its own cgroup, verified in-cluster.

Metrics are off by default for the same reason: an exporter costs memory in every process that runs one, and an install with nothing scraping it was paying that for nobody.

Every process validates its whole environment before it serves anything, against a schema rather than against twenty-eight readers that each decided for themselves what a bad value meant. The old env.ts was a set of readers with deliberately different failure modes: requiredEnv threw at import time in auth/index.ts and at request time everywhere else, positiveEnv(name, fallback) read AUTH_RATE_LIMIT_MAX=2O (letter O) as 20, and trustProxySetting() returned false for anything it did not recognise — so a garbled CIDR list disabled per-client rate limiting rather than refusing to boot.

One rule, everywhere: absent is fine, malformed is fatal. An unset optional knob still takes its default, which is the half of the old behaviour worth keeping; a value that is present and wrong is a boot failure that names the variable, says what it expected, and prints what arrived — unless the variable holds a secret, in which case it is named and never printed.

apps/api/src/config/schema.ts layers them, and the layering is not a convenience — it is what the deployment already does.

MASTER_KEY is the one whose placement changed behaviour. A process booted without it used to start cleanly and fail at the first job that opened a cluster — a decrypt error inside jobs/cluster-connection.ts, hours after the deploy that caused it. MASTER_KEY_VERSION is cross-checked against the keys actually present: bumping it to 2 without supplying MASTER_KEY_V2 seals every new row with a key the deployment does not have, which is the rotation's one unrecoverable mistake (§10.4).

Validation happens in a side-effect module imported first by the entrypoint (src/env.api.ts), not in a call inside bootstrap(): every import of an entrypoint is evaluated before the first statement of its body, and auth/index.ts asks for its values while it is being imported. A misconfiguration prints its report and exits 1 rather than unwinding as an unhandled throw — the stack of a config error says only that the config was loaded, and it is what an operator reading a CrashLoopBackOff scrolls past.

• A truthy-looking flag. REQUIRE_OWNER_2FA=1 used to read as OFF, silently disabling a second factor on the accounts that can flip a customer's cluster live. Every producer in the repo writes true or false, so 1 is a mistake.
• A TRUST_PROXY that is not true, false or a CIDR list, or a list with one bad range. Both used to mean "nothing in front", and the deployment served on with one shared rate-limit bucket (§10.5).
• A MASTER_KEY that is not 32 bytes of base64. Buffer.from(x, "base64") never throws — it drops what it cannot read — so a truncated key produced a short one that xchacha20poly1305 rejected much later, with a message about a nonce.
• Half-configured SMTP. A host with no credentials builds no transport, so the deployment sent nothing while every invitation, reset and emailed two-factor code reported success. Sending is still entirely optional: leaving SMTP_HOST unset is how a deployment says so — and indexterity_mail_sends_total {outcome="disabled"} counts what that costs, because "off on purpose" and "off by mistake on an install that meant to send" look the same from every other signal.
• An unrecognised SIGNUP_MODE, DEFAULT_ORG_PLAN or LOG_LEVEL, each of which used to fall back to a default that looked deliberate.

apps/web/src/lib/env.ts uses @t3-oss/env-core rather than a second hand-rolled schema, because the split is the interesting part here. api-origin.ts and api-passthrough.ts read process.env in modules that sit in the SSR graph, and nothing structural stopped one of those values being read from a module the client build also pulls in — a runtime address inlined into the bundle is what broke compose in #2. Every variable is declared server, so reading one in the browser throws by construction; a value the browser genuinely needs has to be declared under client with the VITE_ prefix, which is public by definition. There are none, which is the point. The two build args in apps/web/Dockerfile that survived #2 (VITE_API_URL, VITE_WEB_ORIGIN) read by nothing are gone with it.

One variable changed meaning there. The web server read TRUST_PROXY === "true", but the chart hands this pod the same value it hands the api, and that is usually a CIDR list — so a configured proxy read as "no proxy", and the passthrough stripped x-forwarded-for from requests it forwarded to an api whose rate limits are counted by it. It reads the whole dialect now.

A variable has four homes and nothing checked any of them: .env.example documents it, docker-compose.yml runs the dev stack on it, the integration and e2e suites spawn processes with it, and the Helm chart is what a deployment actually gets. The only thing that ever caught a missing one was a deployment failing.

apps/api/src/config/homes.test.ts walks the schemas and holds all four to them, in both directions:

• every required variable is named by the chart workload that runs that process, by compose, and by .env.example;
• every variable the chart, compose or .env.example sets is known to a schema — which is what catches a typo (REQUIRE_OWNER_2FA misspelt in the chart reads as "off" forever) and a leftover of something the code stopped reading. The exceptions are listed rather than waved through: the dashboard's own variables, the two Sentry DSNs compose maps per container, postgres' own configuration, and the all-in-one supervisor's WEB_METRICS_PORT and WEB_SENTRY_DSN, which no process reads because it splits them into the names each already does (§15.0);
• every declared variable is settable from at least one home, which is how ALLOW_UNTESTED_DATABASE_VERSION and the two rate limits were found to be reachable through nothing but extraEnv.

It reads the chart as text and expands {{ include }} against _helpers.tpl itself, so the unit suite does not need helm installed. A define runs to the last end before the next one, not the first — coreEnv wraps MASTER_KEY_VERSION in an if, and stopping at that end silently credits the chart with none of the variables after it.

OpenTelemetry instruments, exported to Prometheus on a second port — 9464 by default, METRICS_PORT, opt-in via METRICS_ENABLED. Both workloads serve their own /metrics; the chart adds a named port to the api and web Services and an optional ServiceMonitor for each. In a merged topology the dashboard's listener moves to 9465 — one network namespace, one port each — while both Services go on publishing 9464, so nothing that scrapes them has to know (§15.0).

A separate port rather than a route under /api, for the same reason §10 gives for everything else: the ingress publishes the app hosts, the endpoint carries no auth, and cluster counts, queue depth and pipeline state are operator information. It is reachable in-cluster until someone deliberately exposes it, which is also the shape a scrape target is expected to have.

The OpenTelemetry wiring is packages/metrics — one provider, one exporter, one startMetricsServer. It is a factory rather than a module with an import-time provider because the caller has to own the moment the provider is installed: an instrument binds to whichever provider was global when it was created, so an instrument built first is a no-op forever and gives no symptom. Each app has a one-line module that calls the factory, and its instruments import the meter from there, so module initialisation order does the rest. The instruments themselves stay in the apps (apps/api/src/metrics, apps/web/src/lib/metrics), because what to measure is not shared — only how to export it.

The two workloads report disjoint sets, because each can only answer for what it sees:

Four decisions inside that are not obvious:

• The usage-trust gate reports the check that refused, not just that it refused. indexterity.usage_trust.decisions is one point per index the classifier considered, labelled by engine and by outcome — trusted, or which of the gate's eight checks said no, with the counter-reset trigger broken out. A bare pass/fail would say findings are being suppressed and nothing about what to do; the distribution is what tells you whether the reset trigger is stricter than it needs to be, and the answer is expected to differ per engine because MSSQL's routine index rebuilds trip it in a way mongo's do not. This is the operator's answer and not the customer's (#277, D99): "how often does this fire across the fleet" and "why is MY cluster quiet" are different questions, and no aggregation of the first answers the second. The customer half is analysis_notes — one row per cluster per producer, written by each pass and read by the dashboard — carrying how many indexes the gate was asked about, how many it trusted, the count per refusal, and the count per collision guard that withheld a finding. It is a table rather than a second metric because it is state a tenant may read, and a metric is neither per-tenant nor authorized.
• The queue gauges are read from graphile_worker.jobs, not counted at execution. Counters in a process reset when it restarts and report nothing at all when nothing is draining — which is the state most worth alerting on. A job's state is a function of its columns (attempts >= max_attempts is the dead letter, locked_at is running), so it is one grouped query.
• Observable instruments for anything that describes state. A label set that stops existing — the last cluster of an engine disconnected, a task drained — stops being reported, instead of freezing at its final value the way a gauge written as things change would.
• The unreachable-cluster count lives in process memory, per replica. It is the only place the answer exists: an unreachable cluster is a handled condition (§7.4.1), so the tick succeeds and no queue counter ever sees it. With the replica cap lifted (#232) the set is eventually consistent across pods — any replica may drain any cluster's next task, so a stale entry refreshes within a few passes, and the fleet alert is a ratio with a 15 m hold rather than a zero-tolerance count. The dispatcher passes the current fleet on every fan-out, so a cluster offboarded while unreachable is forgotten rather than counted forever.

On the web side, two more:

• One seam, src/server.ts — a server entry vite builds for the SSR environment alone. It starts the listener at boot and counts every response; a listener started lazily on the first request would have Prometheus report the target as down until someone visited the site. Being server-only by construction is the other half of why it is the seam: the browser bundle must not contain the OpenTelemetry SDK, so nothing under apps/web/src/lib/metrics may be imported from a route or a component.
• There is no per-server-function instrument (D29), and since D31 there are no server functions either. The decision outlived its subject: a seam that only restates the instrument next to it is not worth its framework wiring. What D31 did change is the scope of the two that remain — indexterity_web_api_requests_total is SSR reads only now, because a reader's own calls no longer pass through this process, and kind="server_fn" is gone from the request counter for the same reason. A series that is always zero reads as "no traffic" rather than "no such thing".

Route labels are the route pattern, derived from the generated route tree (apps/web/src/lib/metrics/routes.ts) rather than the path that arrived: a counter labelled with arbitrary URLs grows a series per scanner request. The walk reads properties off route objects, so a test asserts it against the real tree — otherwise a framework change would make every request unmatched and nothing would say so.

Instrumentation is the vendor-neutral API, so exporting to an OTLP collector instead of a scrape endpoint is a change to packages/metrics alone. Pod CPU and memory come from the platform; nothing here duplicates them.

The chart ships them (metrics.prometheusRule, off by default like the ServiceMonitor since both need the Operator CRDs), grouped by the question they answer: is the schedule running, is work piling up, can we still reach the clusters, is the safety pipeline meaningful, is the control plane healthy, what are readers seeing, and does anything we say ever leave the building. Thresholds are values; the stale-schedule windows are the crontab in jobs/runner.ts plus room for one missed tick, so the two have to move together.

Four of them are the reason this is in the chart rather than left to whoever deploys it, because the obvious rule is wrong:

• increase() cannot see a dead process. When the pipeline stops, its series go stale — and a stale series matches no == 0 comparison, so the intuitive "no successes in 30 minutes" rule is silent in precisely the worst case. absent_over_time(indexterity_clusters_unreachable[10m]) is what fires, keyed on a gauge only the process running the pipeline ever reports.
• Alert on the schedule* dispatchers, not the per-cluster tasks. scheduleCollect ticks on cron unconditionally; collect only runs if a cluster exists, so a rule watching it pages the moment the last one is offboarded.
• IndexterityControlPlaneDatabaseUnreachable covers a gap in the probes. /api/health returns ok without touching Postgres, so neither liveness nor readiness notices a lost database. A rising indexterity_metrics_scrape_errors_total does, and it fires with no traffic at all — which is when nobody would otherwise find out.
• IndexterityMailFailing is a ratio, not a count of refusals. One rejected recipient is a typo in an invite address, and a rule that pages on it is one an operator learns to ignore; the threshold sits near 1, so it means the transport is down for that channel rather than one address being bad. The deeper limit is that no rule here can see a message the transport ACCEPTED and the recipient's server then rejected — those count as sent, because acceptance is the strongest claim the sending process can make. DMARC is the only thing that reports them, which makes a rua= record at the sending domain part of deploying this rather than an optional extra: a subdomain of a strict-DMARC parent inherits sp=reject, and an inherited reject is silent.

A rule with a typo installs cleanly and then never fires, which helm lint and kubeconform both accept, so CI renders the chart and runs promtool check rules over it.

The metrics say how often something failed; an error says what. Sentry, from all three workloads, opt-in via SENTRY_DSN exactly as the scrape endpoint is opt-in via METRICS_ENABLED — and for a stronger reason. The DSN belongs to whoever installed the chart. A self-hosted deployment reports to its own Sentry or to nothing; no DSN means Sentry.init is never called at all, rather than called with an undefined one.

Sentry rather than a one-pane vendor because §15.2 already exists. The exporter and the alert rules answer the timing questions, so a second measurement pipeline would answer them again somewhere the chart's alerts cannot see. Tracing, profiling, Sentry's own metrics and session replay are therefore all off; what is wired is error reporting and nothing else.

Two projects, split by deployable. The api — pipeline included, one process since #232 — has one; the dashboard shares nothing with it but the contract and has its own release, so it gets its own. Every process reads a plain SENTRY_DSN; which project that is belongs to the deployment (errorReporting.dsn / errorReporting.webDsn, the second falling back to the first).

Plus the SDK's onUncaughtException and onUnhandledRejection handlers in every process, which cover the paths no filter and no job handler can see.

Only faults are reported, and only once. A retry is not an event — the job counters in §15.2 already say how often jobs retry, and reporting each attempt turns one fault into five identical issues. A 429, a 413 and a 404 are the service working. So is an unreachable cluster, which is a handled condition (§7.4.1) the queue records as a success: paging on it would page us about a customer's firewall. That invariant is also why misclassifying our OWN database's failure as a customer's was so expensive (D122): the one signal that the control plane had flapped was suppressed by being mistaken for somebody else's outage. This is why the SDK's @SentryExceptionCaptured() decorator is deliberately not used — its own filter stands down for HttpException and nothing else, so it would report every one of those as an unhandled fault.

The MongoDB driver puts the connection string in the message of what it throws: MongoServerSelectionError names the hosts it could not reach, and an auth failure can carry the whole URI. Redacting request headers therefore does not touch it — it arrives inside exception.value, where no header rule looks.

packages/errors is a pure function, unit-tested without a DSN, a network or an SDK, for the same reason the analysis engine is. It walks the entire event rather than a list of known fields, because a field list is correct until the SDK adds a field and the way it fails is silent. It removes:

• MongoDB and Postgres URIs whole, hosts included — a replica set's member list is the customer's infrastructure and +srv resolves to it, so the hostname is as much theirs to keep as the password is
• the credentials of any other URI, keeping the scheme so the message still says what kind of thing failed
• keys named like secrets, a superset of the Fastify logger's redact list
• the same secrets encoded inside a JSON string, where key matching cannot see them: password in {"email":…,"password":…} is not a key of anything, it is characters in the middle of a scalar
• the values of a failed query — the params: line of Drizzle's Failed query: message, to the end of the line so the stack after it survives (#579). better-auth logs that message on every database error and the console integration makes the log line a breadcrumb, so a sign-in whose database went away was measured sending the user's email in its last two breadcrumbs; a session token or a verification code would have gone the same way

This was learned by measurement rather than by reading, which is why the wiring was exercised against a running stack instead of signed off from unit tests. With it set, a real 500 on /api/auth/sign-in/email — postgres stopped, so better-auth's own handler threw — arrived at a collector carrying request.data: the sign-in body, password included.

In SDK 10.69 the body, stack locals and database payloads are governed by dataCollection, whose defaults collect all three. Each is a route the string scrubber cannot see:

• httpBodies — POST /clusters carries a customer's connection string in its body, and a sign-in carries a password
• stackFrameVariables — the locals of a collect frame hold the decrypted string; sealed at rest (§10.3) is no defence inside a stack frame
• databaseQueryData — every job selects the clusters table, and sealed_dek / sealed_data are columns on it

All are turned off, and the request body is also dropped whole in beforeSend, because setting httpBodies: [] did not stop it arriving — measured twice. The option is kept because it is the documented control and should one day work; the scrub is what actually holds.

Sentry 11, which the api runs since Nest 12 required it (D181), goes the other way: it removed sendDefaultPii and made an omitted dataCollection key mean collect, including the client-address headers the scrubber has no rule for. So the shared defaults now state every category, and a type test fails the build if the SDK offers one they do not name. Measured on the built api, a reported 500 carries what it did on 10.69: the body as [redacted], no cookie, no address header.

Its api layer catches everything a route throws, logs # SERVER_ERROR and returns the 500 itself, so nothing propagates to AppExceptionFilter — which left the most security-sensitive surface in the product reporting nothing. It is reported from the Fastify mount instead, the seam that already reads the status. That costs the stack: better-auth keeps the original error and its onAPIError.onError hook would hand it over, but that option is absent from 1.6.24's type declarations, and a cast a minor upgrade can invalidate silently is the worse trade. Status, method and route separate "the database is gone" from "one route is broken", and the process log still carries the cause beside the same request id.

The SDK instruments modules as they are required, so anything imported before it is invisible to it. main.ts begins with a side-effect import (instrument.api.ts) so the init precedes every other module — the job pipeline's failures are the least-watched in the system, and they run in this same process (#232), so a reordered import would silently uninstrument exactly the workload the feature exists for.

The dashboard initialises from inside its module graph rather than through the documented --import instrument.server.mjs preload. apps/web/Dockerfile ships .output alone, and what lands in .output/server/node_modules is what nitro traced from that graph; a preload file copied in beside it is not part of it, so nothing would have pulled the SDK in for its bare import to resolve against. The in-graph import is what puts it there — an ordering trap worth naming, because the preload would work right up until that import is removed. What it costs is patching before the first require, which buys distributed tracing, and tracing is off.

The dashboard loads nothing without a DSN; the api loads it either way, and that split is a deliberate tradeoff, not an inconsistency (#176). The import was the measured cost, not the init — initErrorReporting already returned early with no DSN in both apps, and the SDK had been evaluated by the time it did: 12.5 MB of heap and 322 ms for @sentry/nestjs alone, 17.3 MB and 340 ms for @sentry/tanstackstart-react.

• The dashboard gates it, and can, cleanly. await import() there is real ESM — fully typed, and ordering is explicitly not load-bearing on that side (tracing is off, and captureException, the global sink and wrapFetchWithSentry need no module patching). Both of its sites are gated: the init in lib/errors/provider.ts and the wrapFetchWithSentry in src/server.ts, which would otherwise have pulled the whole package in on its own. A dynamic import with a literal specifier is still part of the graph nitro traces, so the .output argument above still holds — asserted rather than assumed.
• The api does not gate it, on purpose. A lazy require was built and measured working — ordering intact, cost gone — but require is declared (id: string): any by @types/node, and a real await import() cannot substitute: it is async, and the next require in main.ts would run before it resolved, which is exactly the ordering this section is about protecting. Nothing short of a runtime shape-check could make the lazy version's types trustworthy, and weighed against that, full static typing won — errors/reporting.ts keeps the SDK's real types with nothing narrowed and nothing guarded, at the cost the dashboard avoids. SentryModule.forRoot() in app.module.ts stays conditional on a DSN regardless, since gating which array a real, already-typed reference lands in costs nothing to keep.

Measured: with no DSN the dashboard serves a render with 0 @sentry/* modules loaded (from 538, ~49 MB of RSS), while the api still loads all 548 either way. With a DSN, the api's SDK still loads first — ahead of pg, mongodb and @nestjs/core — with the Nest, Fastify, Postgres, Mongo, Http and both global-sink integrations present, and an error still arrives with its tags intact, exactly as before this section describes a change.

Four things have to say the same version — the tree, the tag, the images and the chart — and nothing derives one from another. So the cut is four deliberate steps and one assertion that ties them together.

npm run version:set X.Y.Z on dev, as a chore: release X.Y.Z commit. It writes the seven package.json files, both of the chart's version fields, and eight places in package-lock.json. A dev → main pull request promotes that, the merge is tagged vX.Y.Z, and pushing the tag is what publishes: release.yml builds the three images and the chart from it, versioned by the tag itself. Nothing publishes from a branch, so what a cluster pulls cannot move because something merged. The GitHub Release is written by hand from the commits the promotion carried.

version:check is what makes that safe, and where it runs is the point. release.yml runs it against the tag before it builds or installs anything, so a tag cannot claim a version the tree does not carry — the failure it replaces surfaced much later, as an npm ci mismatch whose message says nothing about releases.

It earned that job. 0.3.0, 0.4.0 and 0.5.0 each shipped with a stale package-lock.json: npm records a workspace's version there too, set-version.ts wrote the seven manifests and the chart and not that, and the one command whose entire job is reporting version drift printed "agrees" over a tree npm ci refuses (#186). The lockfile is one of the files it writes now, and it is eight fields rather than six — the root's version is stated at the document root and as packages[""], while the node_modules/@repo/* entries are links with a resolved path and no version.

Two file shapes worth recording, both found by trying them:

• The chart is written line-wise, not through a YAML parser. A parser round-trips the document and drops every comment in the file along with appVersion's quotes. set-version.ts matches the two lines and preserves any trailing comment on them instead.
• The lockfile is edited directly, not by shelling out to npm install --package-lock-only. Editing cannot re-resolve a transitive dependency as a side effect of renumbering, and it needs no registry — a release should not depend on the network to renumber itself. Safe only because npm's formatting is JSON.stringify(lock, null, 2) plus a newline, verified byte-identical on the committed 640 KB lockfile.

What this flow does not enforce is the notes. 0.10.0 shipped published with no GitHub Release at all, because writing one is a step outside the tag that nothing checks. The tag-versus-tree assertion cannot see it, and neither can CI.

• No type assertion but as const, and the ban is enforced rather than asked for. scripts/lint-assertions.ts parses every .ts/.tsx/.mts under apps, packages, scripts and deploy and fails npm run lint on any AsExpression that is not a literal narrowing. It parses rather than greps, because text cannot tell a type assertion from select id::text as id or from the word "as" in a sentence, and the honest answer to "how many are left" has to be a number. Generated files (*.gen.ts) are skipped: TanStack rewrites routeTree.gen.ts on every build, and a rule nobody can obey is a rule people learn to disable.

  The allowlist has exactly one entry, and getting there took going to zero first. apps/api/src/test-utils.ts holds stub<T>(partial: Partial<T>): T. It was written, used across ~30 fakes, deleted when the allowlist emptied itself, and then brought back — which is the useful part of the story, because the zero is what proved where it belongs. Partial<T> checks every member a double DOES define, name and type; only the absence of the rest is claimed, and that absence has a cost worth stating: an omitted member is undefined at runtime while the type says it is there.

  Which to reach for is decided by the problem, not by taste. Five ports were probed against stub directly rather than argued about, and four do not compile: a generic query<T> cannot be honestly handed data (the only value assignable to T[] for every T is []), and drizzle's execute/select return builder classes that an object literal is not. The fifth compiles and would be a downgrade — MssqlReplicaCatalog is one method standing in for a dependency on a 22-member class, so collapsing it widens production to shorten a test. So:

  • A dependency wider than its user is a design fix. Name the port. The test then writes a COMPLETE implementation, which is not a fake at all but a second implementation — and the test usually gets better as a side effect (emitClusterEvent took a whole Database to send one pg_notify, so its test could only assert that execute had been CALLED; against a one-method EventNotifier it asserts the event).
  • A vendor type with eighteen members that a test touches one of is a stub. Writing those out is cost with no reader.
  • When the statement is "no member of this is expected", combine them: stub<T>(new Proxy({}, { get: refuse })). new Proxy({}, …) IS a {}, so it satisfies Partial<T> honestly — one line, complete, and it still throws naming whichever member somebody reached for.

  The argument is a result, not a principle. The repo carried 37 double assertions and 30 test fakes; none survived contact with an alternative, and the reason is that a cast you cannot remove is usually a dependency that is too wide. emitClusterEvent took a whole Database to send one pg_notify, so its test faked one and could only assert that execute had been CALLED; against a one-method EventNotifier the test asserts the event itself. That pattern repeated: collectMssqlServerHealth used 1 of a connection's 22 members, the mssql executor 4, the roster's members 3.

  What replaces one, in the order to try: delete it (the types may have caught up, and [] is already assignable to T[]); declare and ASSIGN instead of asserting; widen your own signature; declare global for something the runtime really adds (RequestInit.duplex); declare an OVERLOAD when the body is honestly unknown -> unknown and only the signature knows better (scrub<T>); dispatch through a record keyed by the discriminant, so indexing it with the type parameter proves what a ternary chain could not (RouteScope<L>); fix a port to ONE row type rather than making its method generic; or name a narrow interface so a test's object implements all of it. messageOf, at and present exist for the common three — a caught value, an index, a nullable. For a double that must satisfy a generic method, Promise<never> is a Promise<T[]> for every T: a source whose query rejects is complete and claims nothing.

  Three shapes that are not an as at all, and are banned by the same script:

  • A generic function can declare a return type narrower than its body produces, and it compiles. Found Sep 2026 by auditing this section's own premise, and it had already shipped: keysOf promised (keyof T & string)[] over a body returning Object.keys(record), which is string[]. Inside the generic, keyof T & string reduces to string, so tsc has nothing to object to — and every call site got the literal key union out of a body that had only ever produced string[]. TypeScript declares Object.keys as string[] on purpose, because a value can carry keys its type does not list, so narrowing the result is the claim as K[] makes with a signature for syntax. No as, no overload, no any: invisible to every rule above it. stdlibOffences flags a return Object.keys(…) under a return type mentioning keyof. The replacement is for…in plus Object.hasOwn, where the key type is the compiler's own judgement rather than ours. The rule was tightened during the change that added it, because its first form reported async …(): Promise<string[] | null> — a real shape, honestly declared — and a new rule's first false positive is how people learn to switch it off.

  • An overload signature is not checked against its implementation. Measured: function f<T>(x: T): T[] declared over a body returning a scalar compiles. So an overload whose implementation returns unknown or any is the same unchecked claim in different syntax — it passes the compiler, the linter and every test. scrub<T> was written that way for exactly one commit before this rule existed.

  • JSON.parse returns any, so ANNOTATING its result checks nothing. Four of these were here, two of them in the security gate, where an unchecked parsed.vulnerabilities that npm had renamed would have been undefined and passed every advisory through as "none found". Narrow it or hand it to a schema; unknown is the one honest annotation.

  @total-typescript/ts-reset (recommended set, one reset.d.ts per package) replaces the standard-library types that hand back any, which is what the two rules above can only chase after the fact — the lint script sees const x: T = JSON.parse(s) but not use(JSON.parse(s)). Adding it found two more of the same shape immediately: Array.isArray narrows to any[], so

  if (!Array.isArray(v) || v.some((x) => typeof x !== "string")) throw …;
  return v;   // typechecked as string[] — on a runtime check the compiler never read

  @total-typescript/shoehorn was considered and declined. fromPartial<T> returns its argument, so it checks the members you write and asserts the absence of the rest — which is precisely stub<T>, the helper this repo built, used across ~30 fakes, and then deleted when every remaining call site turned out to have a narrower dependency. A member the double omits is undefined at runtime while the type says it is there. overriding(real, Partial<T>) does the same member-by-member checking and forwards what it does not override to the real object, so an unmocked call fails out loud. Same reasoning rules out vitest-mock-extended's mock<T>().

• A value is T predicate whose body does not prove T is an assertion with is for syntax, and the lint script cannot see it either. typeof v === "object" && v !== null is true of [1, 2, 3], so a predicate written that way narrows an array to a record and every reader downstream treats its indices as field names. Sep 2026 found five copies of it under four names — isRecord, isXmlNode, isNode, isRoute, two of them in the same directory — and four of the five were unsound in exactly that way. One isRecord per app now, and it excludes arrays. The general lesson is the one the overload rule already teaches: the ban catches a syntax, so the shapes worth hunting are the ones that make the same claim without it.

• One narrowing primitive per app, because three names for one function is how a fifth copy gets written. field, fieldOf and errorCode were the same function in the same file: Reflect.get answers undefined for a key that is not there, so fieldOf's missing name in value guard changed no answer, and errorCode was field(error, "code"). The api's live in src/errors/message.ts (messageOf, field, isRecord, keysOf, orpcCode/orpcStatus), the dashboard's in src/lib/narrow.ts.

  at and present are the one justified duplicate, and it is enforced. Both apps need them and neither can import them: every internal package resolves exports["."].types to ./src/index.ts and default to ./dist/index.js, so a typecheck passes with no build and the runtime does not — and the mssql and postgres integration jobs run npm ci and the suite with no build between them. Four went red the day these were shared. What was NOT paid for is drift, and drift had already happened: both copies opened with a comment claiming they held the same thirty lines, while one had grown keysOf and the other field. scripts/lint-twins.ts enforces byte-identity, so anything one app needs alone lives beside its caller. A comment asserting a duplicate is in sync is not a mechanism — which is the same finding as the pgEnum comment in the bullet below.

• at returns T, not NonNullable<T>. The stricter return stripped null out of every element type it was handed, so at(rows, 0) on the (string | null)[] a nullable column reads back as threw on a perfectly good value and blamed the array's length. Absence is what at knows about; a null element is a value, and present is how a caller says it will not accept one.

• satisfies when the wider type exists only to be checked against; annotate when a caller must SEE the wider type. The difference is not stylistic. BURST_SCHEDULE: readonly ScheduledPass[] widened every entry's task back to the interface's type — and the interface said string — so a typo in the schedule compiled and would have enqueued a job nothing handles, which graphile-worker then retries until it gives up. satisfies readonly ScheduledPass[] checks each entry AND keeps it literal, so the seven names travel through BurstResult and addJob and the wiring is typo-proof end to end. The other direction: RESOLVERS needs its mapped-type annotation, because indexing it with the generic level is what proves the scope.

• A type spelled twice needs a check, not a comment. Five types exist once as a zod schema in @repo/contracts and once as an interface in engine/ports.ts, and the duplication is the layering working — the adapters must not depend on the wire format, and the browser cannot import an adapter. What was not working is that nothing held the pairs together. The engine list was spelled four times (contracts' z.enum, db/schema.ts's pgEnum, the port's union, and analysis/purge.ts's PurgeEngine, which had no reason to exist and is now an alias), and the entire mechanism was a comment on the pgEnum reading "must match ClusterEngine in src/engine/ports.ts".

  EngineCapabilities already had this problem and already had the answer — engine/registry.test.ts compares contracts' table against every adapter's own capability, with a comment saying this is exactly the kind of pair that drifts silently — so the best idea in the repo had been applied to one pair out of five. types.test-d.ts now holds ClusterEngine, PrivilegeTier, TlsOverrides, PrivilegeCheck and ConnectionDiagnosis, compared through a local deep-Mutable because the two spellings differ in readonly on purpose (http/mappers.ts copies at the boundary) and the member names are what must not differ. Drift was already caught in ONE direction — add a field to a contract schema and the mapper stops compiling — and the silent direction is the other one: add a field to a port and oRPC's output validation strips it on the way out, so the dashboard simply never sees it.

• Test the types. Several properties here hold at the type level and nowhere else, so no runtime test can fail when one breaks: an owner-level route seeing a resolved membership, a reader port fixing its row, a borrowed connection having no close. Those used to be true because someone had read the code and said so in a comment — the same standard the assertion ban exists to replace. src/types.test-d.ts runs under vitest --typecheck.only as the second half of npm test. Write each assertion, then BREAK the thing it guards and watch it go red; the first draft here was wrong twice and both were caught that way.

• Compiler flags are measured, not collected. On top of strict and noUncheckedIndexedAccess: noImplicitReturns, noUncheckedSideEffectImports and allowUnreachableCode: false, all three free. Three more were measured and turned down, which is the part worth recording:

  flag	cost	verdict
  noPropertyAccessFromIndexSignature	7 errors, 62 rewrites	no — every one was process.env.FOO, where the maybe-ness is already obvious. Found no bugs and made the code noisier.
  exactOptionalPropertyTypes	36 errors	yes, Sep 2026 (D125) — reversed. The earlier verdict blamed vendor types and had the direction backwards: pg declares payload?: string | undefined and better-auth declares its hook members the same way, so our ports were the narrow ones, refusing the exact value they exist to describe. Widening ProbeNotification, AuthHookContext and the dashboard's Answer to the library's own spelling fixed those, and nothing was waiting on a vendor.
  verbatimModuleSyntax	2506 errors	no
  skipLibCheck: false	15 errors	no — all in dependency .d.ts files.

  Measure with npm run typecheck -- --force. Without --force, turbo cache-hits whole packages and reports a number that is far too low — the first reading for exactOptionalPropertyTypes was "2 errors", the figure recorded above it was 18, and the real total across all seven projects was 36.

  Two answers cover all 36, and which one applies is a question about meaning rather than about types. Where absent and undefined say the same thing, the property declares ?: T | undefined — the honest signature, since a caller may omit it OR pass undefined; one such declaration on OpenClusterOptions.tunnels fixed eight job call sites. Where the target is a vendor type, the key is built conditionally, and that is the truer form every time: RequestInit.body is BodyInit and undefined is not one, so a GET carries no body key rather than an empty one. Reach for the spread whenever absence is load-bearing — the connect form sends no observedDatabases key at all, because a missing field is what the api reads as "observe every database", and a key holding undefined only ever worked because JSON.stringify drops it.

  Strict TypeScript everywhere; the contracts package enforces boundary types. No linter-ignore comments.

• Every dependency pinned to latest stable at scaffold time.

• Comments stay short.

• Clean runtime. The api and the web app must run with no errors and no warnings — server logs, build output, and the browser console. A warning is a defect: fix the cause, never silence it. In particular:

  • Nothing may log above info on a healthy request. An expected external condition (a cluster that is down, a customer's credentials we cannot decrypt) is handled — classified, logged once at the right level, and turned into a decision — not thrown as a stack trace and retried.
  • The web app hydrates without mismatches. Anything the server cannot compute the way the reader will see it — their clock, their timezone — is gated behind useMounted() (apps/web/src/lib/hydration.ts) so both sides of hydration render identical markup. A timestamp does not thread that gate by hand: <LocalTime> (apps/web/src/lib/local-time.tsx) owns it, because the failure mode is forgetting the gate rather than formatting the date.
  • Builds emit no warnings; a framework asking for a config value (nitro's compatibilityDate) gets one.
  • A warning only a human can see is not a rule. The canonical-Tailwind-class warning — w-[220px] is w-55, since the v4 spacing scale is 0.25rem — came from the editor's Tailwind integration and nothing else: Biome has no rule for it (useSortedClasses is about class order), and Tailwind v4 ships no lint binary. So it was observable while typing and unenforceable in CI, which is a standard that decays. scripts/lint-tailwind.ts runs inside npm run lint and fails the build on an arbitrary value that has a canonical utility. It is deliberately narrow — the spacing scale, pixel widths, and the fraction percentages, where a value maps to one utility and only one. It says nothing about colours or the named scales (rounded-sm, text-lg), because being sure there would need Tailwind's resolved theme and a check that guesses is worse than none: the first false positive teaches people to skip it. Vendored components/ui is exempt, which is #9's decision rather than a new one — rewriting registry code forks it, and the next shadcn add puts it back. The script verifies its own cases on every run, because the way a lint script actually fails is by matching nothing and passing forever.

• No raw control characters in source. A literal NUL byte used as a map-key separator (`${a}\0${b}` written as the byte rather than \u0000) makes git classify the file as binary: no line diff in a pull request, no useful blame, and a reviewer sees Bin 4175 -> 6050 bytes. Two files had one and nobody noticed, because the code works perfectly — it is only the tooling that goes quiet. git ls-files -z | xargs -0 grep -laP '\x00' finds them.

• graphify — once the monorepo exists, build the knowledge graph and treat architecture / flow / dependency questions as graph queries first. Re-run when a large change may have made the graph stale.

The original phasing is complete through the engine, the apply pipeline, ROI, multi-tenancy, billing, the Helm chart and v0.1.0.

Planned work now lives on the project board. It is not duplicated here — two roadmaps in one repo is one roadmap and one lie. This page records what was decided and why (see docs/decisions.md); the board records what is next.