noetl/ai-meta#455 · spec repos/ehdb/design/north-star-distributed-registry.md · wiki North Star

The vision. EHDB should be "like etcd but truly distributed", with extensions NoETL needs: an append log, catalog relations, a topology / service registry with discovery (including ephemeral runtime units), secret references (references, never material), and vectors for AI.

This is all internal EHDB — self-sufficient, no external datastore. It is cleanly separate from the business catalog, which is external by design (travel → Firestore).

Internal (this umbrella)	NoETL's own objects: the append log, topology, the internal catalog. EHDB and only EHDB.
Business	Domain data. Playbooks + an external DB/API. Never EHDB.

The internal catalog holds the playbook that reads the business data. It never holds the business data.

Stated up front because the phase order depends on it: a lease without fencing is not safe under partition, and single-writer durability for the unsealed tail is a real gap rather than a configuration choice. Those two are P7 and P8, and they are the hard parts — not the registry, and not vectors.

election.rs and fencing.rs exist in ehdb-reference, which the server does not link. Existence is not reachability.

phase	work	risk	status
P1	wall-clock TTL liveness on D8 (list_live_at)	low	✅ ehdb#384
P2	generic (kind, id) registration + discover	low–med	✅ ehdb#385
P3	watch(after_seq) poll API + stated latency	low	✅ ehdb#385
P4	ephemeral execution registration with TTL	low	✅ ehdb#385
P5	secret_ref type + refuse-material validation	low	✅ ehdb#385
P6	vectors on catalog objects + recall@k	med	✅ ehdb#390 — and it reframed the deliverable
P7	unsealed-tail replication	high	⬜ specced, deliberately not started
P8	writer election + fencing on the production path	high	⬜ specced, deliberately not started

P7/P8 change durability or safety semantics, so each wants its own spec and prod sign-off. They are flagged rather than started.

ehdb v0.5.0	✅ tagged + released 2026-10-08, FORMAT_VERSION still 1
server adoption	✅ v3.125.0 live in prod (build_info{version="3.125.0"}); v3.124.0 adopted the library, v3.125.0 exposed its metrics
P1–P4 capabilities	✅ REACHED in prod (server v3.126.0). The server self-registers as RuntimeKind::Server, renews its lease (heartbeat=4, age reset 59s→6s), and serves GET /api/runtime/topology + /watch. watch?after=<cursor> is quiet, which is what makes it a watch rather than a scan
P5 / remaining P-work	⚠ workers, gateways and EHDB instances do not self-register yet; executions unregistered; SecretRef unused
state gauges	✅ LIVE and verified in prod on server v3.125.0 — 34 ehdb_l0_* series, manifest_parts=311 (a real reading, not decoration), parts_under_replicated=0
#391 compaction	⚠ inapplicable to the server by construction — it uses only D1EventLog (which deliberately does not opt in) and has 0 VectorStore references. This is why "existing behaviour unchanged" passes, and it is not a win to claim

Adopting a library is not reaching its capability. Deploy log: noetl/ai-meta#456.

date	what	where
2026-10-08	P2–P5: RuntimeKind, discover(kind, now, ttl), watch_since, ephemeral units, SecretRef	ehdb#385
2026-10-08	B6/B7: memory is O(parts) not O(records); concurrency measured for the first time	ehdb#386
2026-10-08	B8/C6/D5: drain benches (530x), state gauges, benches running in CI	ehdb#387
2026-10-08	P6: recall is 1.0 by construction; query cost is O(ops) not O(live points); compaction cannot help	ehdb#390
2026-10-08	ehdb#391: Dataset::supersede_key — compaction takes 4,608 records / 65.02 ms to 500 / 7.13 ms	ehdb#392
2026-10-08	release automation — ehdb had none; now gated off behind EHDB_RELEASE_ENABLED, verified skipped, tag still v0.4.5	ehdb#393
2026-10-08	findings filed: ehdb#388 lazy-ack O(n²), ehdb#389 drain attribution + the #298 cap tradeoff	—

⚠⚠ A shared AtomicU64 sequencer plus a Mutex is not sufficient. Minting the sort key before taking the append lock reorders 34.2% of appends while losing and duplicating exactly nothing. That is #362's prod mechanism — a snowflake id minted before insert, so commit order is not id order — reproduced in 20 ms in a unit test. My own first draft of that test asserted the sequencer was safe; the ascending canary caught it.

⚠⚠ A standing replication deficit was invisible. The uploader records only the successful replica writes, so a part holding 1 of 2 copies is is_durable(), on_upload_done fires, and the age-based durability window reports it done. replica_writes is cumulative and climbs while the deficit stands. New parts_under_replicated gauge closes it.

⚠⚠ The batch cap that fixed #298 makes a drain above the cap quadratic (exponent 2.06). Both the bound and the quadratic are real; the tradeoff was never measured.

⚠ inflight is not derivable from lag. Measured: lag 10 with 0 in flight (stalled) and lag 10 with 4 in flight (busy) — identical lag, opposite operational conditions.

⭐ Memory is O(parts), not O(records): ~16.6 KB fixed + ~1.8 KB per part, so 1 M records at seal_max_records=1024 holds ~1.7 MiB. Part count is the memory signal and merge is the memory control.

⭐⭐ P6 reframed its own deliverable. top_k is an exact brute-force scan, so recall@k is 1.0 by construction — a reported 1.0 proves only that an exhaustive scan was exhaustive, and recall becomes falsifiable when an ANN index exists and not before. The number that actually bounds the design is cost, and it tracks op-log depth rather than live points: 500 points re-embedded 10x (5,000 ops) costs 64.9 ms, within 2% of 5,000 written once (66.1 ms) — so re-embedding a collection is as expensive as growing it, 378x over the same 500 written once.

⚠⚠ And compaction cannot fix it. A merge made the 10x case 9% worse, because live_points reads every op ever written and folds latest-wins, VectorDataset has no dedupe_key, and the Dataset trait exposes no supersede hook at all — merge combines parts, not keys. The prerequisite for vectors at scale is a key-level compaction primitive, and it is broader than vectors: RuntimeStore's D8 folds and the catalog's attribute/relation folds share the pattern. ehdb#391.

⭐⭐ ehdb#391 is closed, and the eligibility answer was narrower than expected. The compacting merge works — 4,608 stored records / 65.02 ms → 500 / 7.13 ms, converging to exactly the live set, with per-record cost identical across every sealed row. But of the three consumers the issue named, only one may opt in: RuntimeDataset (D8) is disqualified by watch_since, the op-log reader added in P2, because compacting it would delete the history a watcher resumes from. One history reader disqualifies a dataset. D1EventLog is excluded by platform rule. The catalog folds remain a candidate but that is the catalog's semantic call.

✅ ehdb is releasable. It had no release automation at all — every v0.4.x tag was pushed by hand. semantic-release.yml + release-ehdb now exist and avoid both known traps by construction: .releaserc.json omits @semantic-release/git so GH006 cannot occur, and the tag workflow is dispatched explicitly because a GITHUB_TOKEN-pushed tag does not trigger workflows. ⚠⚠ Gated off behind EHDB_RELEASE_ENABLED and verified by merging it: the job ran on main and was skipped, tag still v0.4.5.

what	where	state
ehdb v0.5.0 cut; both server pins moved + re-vendored	server v3.124.0	live
L0 state gauges on /metrics	server v3.125.0	live
server self-registration + GET /api/runtime/topology	server v3.126.0	live, Server: 1
worker-pool registrations mirrored into D8	server v3.128.0	live, Worker: 5, TTL expiry observed 8→5
heartbeat throttle + P4 ephemeral executions	server v3.129.0	live, one execution observed live then expired
Ehdb hook on the events publish	server v3.130.0	⚠⚠ shipped and unreachable — see below
Ehdb hook on the command publish (Landed only) + per-address throttle	server#506 → v3.130.1	deployed, awaiting first dispatch
P2 Gateway self-registration	gateway#55 → v3.13.0	✅ reached — Gateway: 1, id = the exact pod
P5 catalog refuses secret material	server#507 → v3.131.0	released, not yet deployed

v3.130.0's Ehdb registration was correct in every respect except where it was called from. It fired only on a successful publish, never from configuration, used the address actually routed to — and measured 0 on prod, because should_publish skips system executions and those are all of prod's regular traffic (publish_skipped{reason="system_execution"} = 22, event_ingest_total absent, command_publish_total{pool="system"} = 4).

Two structural tests covering it passed throughout. Both assert the call site is correctly gated; neither asserts anything reaches it. This is #326 recurring, in the same change whose commit message cited #326 as the reason a different hook was placed correctly.

Two sub-lessons:

• Deferred is not liveness. It means the on-path budget elapsed and a background task is still retrying; the writer may be down. The hook sits on Landed only.
• One cap for two addresses. EHDB_LAST_SEEN was a single global timestamp, so whichever bus published first throttled the other out for a whole floor interval, leaving a live tier unregistered while every counter read healthy — the same shape as one cap for two directions (worker#314).

The guard refuses registrations, so its false-positive rate matters more than its recall. Measured over 26,081 string values in 494 real playbooks before shipping: it produced one finding and the finding was wrong — vertex-ai.yaml embeds Python that reads a service-account credential, so credential-handling code names both of its keys while holding no material. Requiring a PEM body took it to 0 in 26,081.

⭐ And a run against prod's 1527 catalog entries reported 0 findings with scalars_scanned=0, because /api/catalog/list omits content. Without the published denominator, that was a clean bill of health computed over zero bytes.

1. Observe discover(Ehdb). v3.130.1 is deployed; the fresh pod's command_publish_total is absent (pruned, not zero) and the dispatch period is tens of minutes. Volume is not duration — a sized window, not faster polling. Report it either way rather than inferring.
2. Deploy v3.131.0 (P5) and prove it end-to-end: a planted material-shaped value is refused, an alias-referencing entry still registers.
3. P6 vectors on catalog objects. ⚠ Reframed: top_k is an exact brute-force scan so recall@k is 1.0 by construction, and the number that bounds the design is cost, which tracks op-log depth, not live points.
4. A5 — fold determinism has no engine-side enforcement point. Measured and proven order- and completeness-sensitive by test, but nothing in the engine refuses or counts a non-deterministic fold at runtime.
5. ehdb#388 / #389 — the two drain findings.
6. ⚠ Not observed: a Gateway row expiring. TTL expiry is proven live for Worker and Execution via the same list_live_at path; restarting prod's user-facing gateway to watch a timer already demonstrated on two other kinds was not worth it. Recorded as not-yet-observed for that kind.
7. Owner-gated: the shadow→enforce auth flip, which now gates write endpoints, execution data and POST /api/runtime/register — an enforce flip needs a service credential on the gateway or registration starts failing fail-soft. Plus #457 (disk headroom) and #458 (26 stale project references). P7 (tail replication) / P8 (election + fencing) stay specced.

• Umbrella: EHDB primary serve — the serve path.
• Umbrella: Catalog model on EHDB — the internal catalog that P6's vectors attach to.
• agents/rules/self-sufficiency.md — no external datastore does not mean no dependencies; proven libraries are preferred over hand-rolling subtle algorithms.