Umbrella: EHDB primary serve path — a writer-fronted tier service

Recent activity

2026-10-10 (evening) — B3+D2 LIVE on prod

Server v3.143.0 / ehdb v0.9.0. NOETL_EHDB_TAIL_REPLICATION=true, TICK_SECS=60, cleanup default on. Rolled by digest sha256:a5adb800 (rollback sha256:2fbbc8be), gen/obs=80/80, 0 restarts, env name-set identical.

⭐⭐ The whole chain, measured on prod over 32 minutes:

t	appends	seals	batches	records	reclaimed	retained	bucket tail/
1458 s	188	0	2	174	0	4	3
1555 s	188	0	3	188	0	6	3
1845 s	188	1	3	188	6	0	0

188 unsealed records off-box, then all 6 objects reclaimed once a durable part covered them. tail_failed=0. Bucket counted externally with gcloud.

⚠ Window: 60 s instead of 900 s — bounded, not zero. No remote ack gates the local append.

⚠ The first ~400 s read appends=0 — the window, not the code: prod had no execution traffic (only runtime_heartbeat). Not worth triggering an execution to move a metric.

Next concrete steps

1. ⛔ The worker prod roll is blocked on a release that produces no image. Two failures (v6.2.2, v6.2.3) from a floating noetl-tools range that --locked could not fix, because cargo chef cook writes its own lock. Exact pin merged (worker#351); waiting on v6.2.4 reaching AR, then roll with the #460 baseline/rollback.
2. #466 — no alert watches any PVC utilisation (0 of 19 policies).
3. C2+C3 stay HELD. Election must be live and minting epochs, verified in shadow, before fencing enforces.

2026-10-10 (later) — B3: the unsealed tail now leaves the node

ehdb v0.8.0 + v0.8.1, server v3.142.0. Flag off; not enabled on prod.

A2/A3 covered sealed history. The remaining exposure was the unsealed tail, and it was live: SEAL_MAX_AGE=900 with oldest_unsealed_age=719 and unreplicated_oldest_age=0 — nothing waiting to upload, ~12 minutes of records on a single disk.

	worst-case loss window
before B2	unbounded in time
after B2	the seal interval — 900 s
after B3	the replicator tick

⭐ Bounded, not zero. No remote ack gates the local append.

Proof: the RED control first (flag off → 16 of 21 recovered, so the loss is real and the greens are falsifiable), then all records recovered from the replica alone with the local root destroyed — 21 of 21 in ehdb, 13 of 13 through a real GCS emulator, tail objects counted from the bucket under tail/ and absent from parts/.

⚠⚠ A defect I introduced in v0.8.0 and caught before enabling: the driver held the engine lock across the remote put, and that lock is taken by shadow_append on the live write path — p50 77 ms / p99 234 ms injected into any colliding append, once per tick. v0.8.1 splits prepare / upload / commit so only the middle phase does I/O and it holds nothing. The rule was already written in ehdb's uploader; I had to be reminded by reading my own code against it.

Next concrete steps

1. D1 — the tick. Prod measures 314 appends/hour, so 15 s carries ~1.3 records/object (5,760 objects/day) while 60 s gives a 60 s window at a quarter of the objects. ⚠ I recommend 60 s and believe the 15 s default I shipped is mis-set; the one-line change is ready on request.
2. D2 — cleanup. Nothing deletes tail objects today (~1,440/day at 60 s). Cost is negligible, but it is a data-deletion policy on the event log, so it is an owner call. Recommendation: delete gated on the covering part being confirmed durable, with its own counter so "cleanup is not running" is visible rather than inferred from bucket size.
3. Then roll v3.142.0 by digest and set the two env vars — exact commands on #460.
4. C2+C3 stay HELD. Election must be live and minting epochs, verified in shadow, before fencing enforces.

2026-10-10 — A2 armed on prod, and the acceptance proof found the defect

A2 is live against a genuine second failure domain. NOETL_EHDB_REPLICA_GCS_BUCKET=shastaratech-noetl-prod-eventlog-replica on noetl-server-rust-embedded-0 (env 73→74, image unchanged, 1/1, 0 restarts). The bucket and the roles/storage.objectAdmin binding for noetl-result-tier@… were created in this session with no permission denials.

domains=[local-device:66320, remote-gcs]
replica_set_size        1 → 2
survives_node_loss      0 → 1
replica_single_point_of_failure  1 → 0

New seals replicate — measured. A 1853s sampler; the denominator is stated because a window shorter than the ~900s seal period would have proved nothing:

t	seals	unsealed_age	bucket objs	bucket .eslog
857s	0	869	1	0
934s	1	0	4	1

Object counts taken externally with gcloud, not from the engine's own view.

⚠⚠ The acceptance proof then refused to pass, which is the entire reason for having one. Read back from the remote's own durable manifest:

parts listed in the remote manifest	40
replica_count == 1	39
replica_count == 2	1

Uploads are enqueued at exactly one site — register_and_upload, on seal — so attaching a substrate to an existing store replicates only what is sealed afterwards. Nothing retries, because nothing considers it outstanding. And every aggregate gauge above read healthy the whole time, because each is computed from self.replicas, the declared replica set, not from the parts: survives_node_loss is a statement about configuration that was being read as a statement about data. The only part-level signal was parts_under_replicated sitting at a non-zero constant, which is indistinguishable from a backlog draining slowly.

⚠⚠⚠ The state is worse than an empty remote. write_manifest_to_all ships the manifest on every upload, so the bucket holds a manifest naming all 40 parts and the bytes of one. A cold load from it alone resolves 39 parts to a replica it does not have — so the bucket reads as a complete store and is not a recoverable one. That makes the backfill a correctness prerequisite, and it means cold-load from the remote alone is not yet claimable on prod.

Fixed rather than redefined. ehdb#400 → v0.6.0 backfill_under_replicated, which enqueues every under-replicated part onto the existing background uploader (so startup is not blocked). Its own trap sat one level down: Manifest::durable_view stores local_path: None and the durable view is what seeds the in-RAM manifest at open, so after any restart a repair path that only knew fs::read(local_path) would have compiled, run, reported success and copied nothing — the same defect repeated. Hence UploadSource::Replica.

Backfills carry their own counters (backfill_uploads, backfill_upload_bytes). A part sealed in an earlier process has no seal→durable interval; folding it into upload_lag_micros_total would divide the real total by a larger denominator and deflate a mean that a capacity decision reads.

A second defect fell out of the worker pin bump. v0.3.2 → v0.5.2 had exactly one compile break across two minors — the new ShardLag.inflight. Defaulting it to 0 was refused: render_snapshot publishes the field and high lag with zero in-flight is specifically the stalled consumer reading, so a hardcoded zero makes a healthy bus publish an outage signal on every scrape. ehdb#402 → v0.7.0 adds ClaimCoordinator::inflight().

⚠ Three of my own mutants did not compile before one did. The first two "RED controls" on the server wiring guard proved nothing; only removing the call block in a form that built actually killed the test.

Still open / next: server#529 carries the v0.7.0 pins plus NOETL_EHDB_REPLICA_BACKFILL (default off) — merge, release, roll by digest, then run the backfill and only then attempt the cold-load-from-remote-alone proof. The worker v0.7.0 bump needs its kind validation. C2+C3 stay HELD; election-before-enforce is absolute.

Recent activity

2026-10-09 (evening) — the hot volume becomes a window, and durability gets honest gauges

Archival is live and draining on prod. ARCHIVE_ENABLED=true on server v3.137.2: 1,766 of 6,253 archivable executions archived so far, every one also verified_durable and indexed, 0 failures, 5,320 real GCS puts.

⚠⚠ No disk reclaim is claimed. The prune floor refuses while any archivable execution is unarchived — correctly — so enabling prune today reclaims nothing. Reclaim is gated on the drain, not on code, and the number that closes #457 does not exist yet.

The floor is wired (server#516). compute_prune_floor had shipped with zero production callers. Three refusals make the wiring safe: an unarchived backlog, an incomplete candidate listing (the paging cap was silent, and a floor from a partial population can sit above an execution never classified), and an absent footprint. The footprint comes from the manifest, not from reading records — the obvious implementation reads tens of MB under the append lock every pass on the one process hosting both buses, and bloom false positives only lower the floor, so the error direction is safe.

⚠ Two defects found by controls rather than reasoning: the all-archived case returned NoFootprint, which is exactly the steady state the drain produces, so the whole prune path would have reclaimed zero while every call returned Ok; and the put metric's "pin" was observe(0.0) on a histogram — a fabricated sample, and in prod the only reading below 50 ms was that invented zero.

⭐ The measurement that closed a decision. In-region GCS put p50 77 ms, p99 234 ms (n=300) — ~19x a ~4 ms local fsync. #460 decision (i), synchronous per-record remote append, had been rejected on an estimate of "tens of ms"; the measurement closes it harder.

A2 — off-box replication (server#518, #519). GcsSubstrate declares Remote{provider:"gcs"}, which is what flips survives_node_loss. Proved by moving bytes, not metadata: the log cold-loads from GCS alone, confirmed outside the test as 6 sealed part objects spanning sequences 1-48 plus the manifest chain, control bucket at 0. ⚠ Two GCS buckets are one domain and do survive node loss, so domains_distinct — not single_point_of_failure — is the gauge that catches a two-bucket plan.

B2 — age-based sealing (server#520) existed in ehdb and was unreachable from the server: no config, no caller outside ehdb's own tests. Both halves now ship together, guarded including the spawn in main.rs.

⚠⚠ And my own B2 gauge understated its exposure (#521): it stops at sealing, but replication is asynchronous so a sealed part still has a local-only window. unreplicated_snapshot() has computed the whole window all along with zero server callers.

The durability statement, unchanged. Sealed parts survive node loss. The unsealed tail does not. Replication is asynchronous. Recovery is a bounded-loss consistent prefix — B2 bounds it, B3 (not built) would remove it. Not "no data loss".

Next concrete steps

1. Let the drain finish, then deploy v3.140.1+ and enable PRUNE_ENABLED=true. Measure the actual PVC reclaim — that is the number for #457/#459.
2. A3 needs a replica bucket + IAM grant. Owner-gated; exact commands on #460.
3. B3 (streaming tail replica) removes the remaining RF=1 window. Sequenced after A2, with B2 shrinking the exposure in between.
4. ⚠⚠⚠ C2+C3 (election + fencing) stay HELD until the A/B replication story is verified in prod. Election must be live and minting epochs, verified in shadow, before fencing flips to enforce — enforcing against an unminted epoch is an immediate outage.

2026-08-29 — the fold is measurable: recovery reads the durable tier (server v3.92.0)

#307 option 2 shipped (server#369); prod is live in verify — the fold is measured and digests compared, and nothing is served.

noetl_ehdb_recovery_source_info{mode="verify"}                    1
noetl_ehdb_recovery_fold_total{source="tier",outcome="folded"}    4   <-- was structurally impossible
noetl_ehdb_projection_read_total{outcome="served_tier"}           0   <-- verify serves nothing
equivalence sweep: agreed=5  disagreed=0  equivalent=true
                   tier_refusals={payload_too_old: 10}            <-- pre-v2 records, fills in

The defect was never a failure. Recovery folded only from the worker's in-memory spine, which holds in-flight executions and evicts them on completion, while the comparator by its nature asks about completed ones. Coverage was ~0 by construction, and the only thing saying so was a zero.

The blocker that made this more than a source swap: mirror_payload never wrote context, and WorkflowState::apply_event reads it. A tier-sourced fold was guaranteed to digest-differ, so repointing recovery on its own would have reported divergence on every execution. Pre-fix records now refuse as payload_too_old instead of folding short and being scored as diverged.

⚠ What verify does not buy. The in-path verdict machinery is, by explicit design, a tier-internal consistency check — it never consults Postgres. And wal_projection_state materialises then refolds, so it re-materialises from a divergent tier and agrees with itself. Divergence against Postgres is caught by two other layers, both demonstrated in the kind gate: the continuous cross-store parity comparator, and the equivalence sweep. That is the reason the flip to tier serving is a separate decision rather than a continuation of this one.

2026-08-28 — READ_SOURCE=wal is live and serving nothing (noetl/ai-meta#307)

Measured on prod, read-only:

noetl_ehdb_projection_read_total{outcome="served_tier"}          0
noetl_ehdb_projection_read_total{outcome="spine_refused"}        4
noetl_ehdb_projection_refold_total{verdict="match"}              0
noetl_ehdb_crossstore_events_compared_total{tier="projection"}   0     (eventlog: 8476)

Every refold refused; nothing ever served from the tier; zero events compared. The refusal reason is spine_incomplete, which reads as transient — and is not: executions nine hours old refuse identically to ones seconds old.

A positive control found the mechanism:

t+4s   complete=True   count=6    indexed_executions=1
t+8s   complete=False  count=0    indexed_executions=0

The spine walk works. The window is a few seconds wide and closes when the execution finishes. state_spine_handler reads the worker's in-memory index and never calls the bounded rehydrate path the drive uses — although NOETL_STATE_INDEX_REHYDRATE_ON_MISS=true and the retained WAL holds 38,611 events. The ratio confirms the paths are separate: 2,281 incomplete builds against 7 rehydrate attempts.

So the comparator, which by nature runs against completed executions, queries a structure that by nature holds only in-flight ones. Coverage is ~0 by construction, not by failure — nothing is broken; two components have incompatible lifetimes.

Anyone reading "READ_SOURCE=wal is live" as "the WAL is serving reads" would be wrong, and no counter says so except by being zero.

The mirror side is healthy, for contrast: 8 mirrored, lag well inside the 30s window, no divergence on any kind.

Next concrete steps

Three options, all owner decisions (noetl/ai-meta#307):

1. Have the spine route rehydrate on miss — smallest change, reuses the bounded path already gated by a flag that is already true. Costs up to deadline_ms on a route that currently fails fast.
2. Point the comparator at the WAL/tier directly — removes the lifetime mismatch at its source rather than papering over it.
3. Accept and document that recovery covers in-flight executions only — and say so where READ_SOURCE=wal is described.

Landed meanwhile: server#364 adds noetl_ehdb_projection_refold_refusal_total{reason} so spine_refused stops laundering five causes (no_events / spine_incomplete / source_unavailable / unparseable / fold_failed) into one label. The six-verdict fail-closed vocabulary is unchanged.

Issue: noetl/ai-meta#257 · Status: 🟢 LIVE — the event-log tier is primary and serving in prod (2026-08-13) · Primary repo: noetl/worker

🟢 The event-log tier is primary and SERVING in prod as of 2026-08-13 on worker v5.118.1 / server v3.81.0 — the first EHDB tier ever promoted. served_primary=232 with every demote outcome 0; serve_state: served_primary in the endpoint reply. The writer-fronted tier service is bound at :9110 on the eventbus RWO PVC, and reads resolve tier_query_source=service from every replica.

⚠ Superseded for the projection tier, 2026-08-14. projection now HAS a runtime serve path — SERVE_WIRED_TIERS = ["eventlog", "projection"] — via #265 (worker#277 + server#350, both open and inert, kind-gated 9 arms / 0 failures, not deployed and not flipped). Note the narrower meaning there: the write path consults decide and reports the verdict; no reader resolves projections from EHDB, so orch_snapshot::load_latest still answers from noetl.projection_snapshot. Read-serve is #265 phase B1.

⚠ kv, object and vector remain shadow and remain unflippable — they have no runtime serve path, so setting them primary is a no-op with a warning. Nothing below changes that.

⚠ served_primary is per-pod. The server's mirror relay targets the noetl-worker-rust-metrics Service, so appends land on whichever app=noetl-worker-rust pod it reaches and the system pools receive none. Sum across replicas when alerting; one pod reading 0 is not a demotion.

What this umbrella is for

A tier in primary mode claims to serve authoritatively. For that claim to be honest it needs a single durable store to serve from — a writer-fronted tier service — rather than the pod-local fragment each worker happens to hold. This umbrella builds that service and the policy that decides when a tier may use it.

The safety property throughout: divergence demotes; it never serves a wrong answer and never fails the caller. The incumbent is the store with history, so when EHDB and the incumbent disagree the incumbent answers and the tier is marked degraded. primary only ever appends, so demoting loses nothing.

The serve decision needs three conditions, not one

Serving requires the tier is primary, a durable service is actually reachable, and parity held on this operation. Collapsing that into the flag alone is how worker#251 shipped a primary that turned verification off — the flag was treated as sufficient when it was only necessary.

The middle condition is where the interesting bugs have lived.

Recent activity

Date	What	Artefacts
2026-08-13 (later)	🔴→🟢 The serving tier was missing every system-pool claim. 11 of 13 events per hourly system/scheduled_cleanup, with unmirrored_by_design=0 claiming completeness — on a tier that is primary. Not an event-type problem: test/simple_loop carries 5 command.claimed and mirrors 29/29. claim_command forks on should_publish; the false branch writes noetl.event inside the claim transaction and never reaches emit_events, and should_publish is false for every system-pool execution by construction (is_system_execution), making the bypass exactly coextensive with the system pool. server#348's "the one chokepoint every authoritative event passes through" was a doc comment with nothing enforcing it; the guard now counts INSERT sites against mirror sites. Kind gate on prod config: before 11/13 → after 13/13, mutation back to 11/13, user pool 29/29 in every arm, controls_ok=true throughout. Prod: fresh scheduled_cleanup 13/13 outcome=match, served_primary 470→523, append errors 0. Closed out on the unprompted hourly cron (13 events, both command.claimed, never flagged; divergent flat at 29 while match climbed 25→30). ⚠ the fix cannot backfill — the 17 pre-fix executions keep their short copy permanently and merely aged out of the sampler's 1h lookback; backfill is an open owner decision. ⚠⚠ found while validating: the parity endpoint writes the counter its own alert reads (#264).	server#349 → v3.81.1 (deployed), #263 closed, ai-meta bd26b0aa
2026-08-13	🟢🟢 GO-LIVE — the event-log tier is primary and serving. Pre-flip gate on a fresh store, 20 executions: 0 divergence, 20/20 full-set match (auth=29 ehdb=29 matched=29, unmirrored_by_design=0), tier-service append errors 0 (was 302), store parses 580/580, all five replicas returning the same 29 records, comparator self-test 8/8. Flip confirmed three ways: served_primary=232 with every demote outcome 0, serve_state: served_primary, and the IS SERVING log line. Post-flip 8/8 COMPLETED, append{ok}=812 / errors 0, log parses 812/812. Got here through three corrections: six of nine merged PRs had targeted a parent feature branch not main (v5.117.1 / v3.80.0 shipped green without the P0); the first soak found a real defect and stopped the flip (#261 torn concurrent appends); and the serve config needed three env vars the plan omitted, without which the soak reads green while downgrading to local.	worker#274 + server#348 (consolidation), worker#276 → v5.118.1, server#348 → v3.81.0, #261 closed, #262 new
2026-08-11	Go-live prep — four artefacts, zero flips. (a) The flip-time warning claimed no serve path on a pod serving 48 ops as primary; reproduced on the released image, replaced with three outcomes (primary_not_wired / primary_no_tier_service / primary_armed), and the wired-tier list moved into primary_serve::SERVE_WIRED_TIERS with a mutation-checked test that re-derives it from the tier sources. (b) The RFC's cutover order is not runnable — it names four unflippable tiers first and the only wired one last; corrected, OQ1 resolved by construction. (c) PR 6 ops wiring staged: port unconditional, BIND default-off, two-sided render gate — not applied, and applying it rolls the single writer. (d) OQ2 answered: accept replicas: 1 (the tier path is the only writer dependency with a fallback; HA blocked on RWO storage) — decision with the user. ⚠ New flip gate: the tier service records no metrics at all.	worker#263, ops#255, ops#256, #260, ai-meta 4e703259
2026-08-11	Re-promotion becomes evidence-based. is_reachable() read the cached-negative deadline as its verdict, so a lapsed cooldown restored belief with nothing having contacted the service. Belief moved to VERIFIED_REACHABLE, set only by a real successful op and cleared by any transport failure; is_reachable() now reads no clock. Bounded startup-probe retry with backoff. New noetl_ehdb_dataplane_ops_total{operation="reachability"}. Five-arm kind gate PASS + mutation-checked twice.	worker#262 → v5.115.3, ai-meta 245bda68
2026-08-10	Traced audit closes all wiring gaps. GAP 1 — probe_at_startup was spawned only inside spawn_event_writer_host, gated on NOETL_EVENT_BUS_HOST which workers do not set, so no worker ever contacted the service and primary could never promote. GAP 2 — no append on the live path called the client, so a service that died after startup never demoted.	worker#261 → v5.115.2
2026-08-10	Reachability is measured by the append, not inferred from config. The policy was fed TierClientConfig::from_env().is_some() — which measures configured, not reachable — so a black-hole address counted as a durable service.	worker#260
2026-08-10	Permanent guard for the arm-D reachability defect.	worker#259
2026-08-10	Primary-serve policy wired into the event-log append path; tier client accepts DNS names, not only IP literals.	worker#257, worker#258

The recurring failure mode

Four separate rounds of this work shipped correct logic fed by a lying input, and each was caught only by the kind gate:

1. config.is_some() answered "is it configured", not "is it reachable".
2. The probe that would have answered it was never spawned on the worker.
3. Nothing on the append path ever re-measured, so death-after-startup was invisible.
4. The cooldown deadline was read as a verdict, so time re-promoted a dead service.

Each is an instance of the same shape: a representation of reachability that nothing forced to agree with reality. See representation-drift — and note that #4 was unreachable until #2 was fixed, which is why it survived three prior rounds.

The gate that catches them

scratchpad/gate_pr6.sh — five arms, every assertion read from the worker's own /metrics outcome labels (behaviour, not source), parsed with awk over exact label matches:

arm	asserts
A	flag-off shadow mirrors and never reports served_primary
B	primary + reachable + parity serves (makes C/D/E non-vacuous)
C	primary with no service configured demotes, and records the demotion
D	primary with a black-hole address demotes — never-contacted ≠ healthy
E1	reachable at boot then killed: demotes within a bounded number of events, and stays demoted across a cooldown lapse
E2	service restored: re-promotes, and only alongside a successful append

⚠ Arm E1's bound is the subtle part. Asserting delta == 0 across the kill half is wrong — the accepted one-operation lag means a bounded number of events legitimately serve before the first failed append demotes. The assertion that carries the fix is E1's second wave, run after the cached-negative window has provably lapsed with the service still dead.

Next concrete steps

0z. The tier serve-flip — awaiting an owner decision. Prod is on verify (measured, serves nothing). Before flipping NOETL_EHDB_RECOVERY_SOURCE=tier: let the equivalence sweep accumulate coverage past the payload_too_old backlog (it can only attest to executions completed after the v3.92.0 deploy), confirm equivalent=true on a meaningful denominator, and decide knowingly that the in-path verdict cannot detect tier-vs-Postgres divergence — the cross-store parity comparator and the sweep are the layers that can. Rollback is one env var: unset it, back to pre-#307 behaviour.

0. ✅ DONE — the flip shipped 2026-08-13. Both parked decisions resolved by the go-live: replicas: 1 accepted, and the staged changes applied (the writer tier-service face was wired directly rather than via ops#255, which remains open and now describes the live shape — it should be merged or closed as superseded). #260 was satisfied first: the tier service now records 115 series.

0a. Open follow-ups from the go-live:

• #262 — should one torn record fail an entire scan, or be skipped and reported? Larger stakes now that the tier serves.
• ops#255 still open while the cluster already carries the change — an IaC/reality divergence to close.
• A serve/demote alert so a silent fallback to the incumbent is caught immediately (demote outcomes going nonzero while eventlog is primary).

0b. ⚠ Only the event-log tier can be flipped. kv, object, projection and vector have a serve_primary_cycle reachable only from bin/ehdb-selfcheck, so setting them primary is a no-op with a warning. Wiring them is unstarted work. §3.4's suggested order describes intent, not a runnable sequence. 0c. ⚠ The tier service records no metrics (#260) — four tracing lines, three on failure paths. The client half is instrumented; the process that would hold the authoritative store is invisible. Should land before any prod flip.

1. Decide whether any tier goes primary in a non-prod environment first. Nothing is primary anywhere today; the slices above make it safe to try, they do not constitute a decision to try.
2. Durability of the tier store. The kind gate mounts /data/eventbus from a PVC so the store survives a writer restart; any real deployment must do the same — the durable dir is otherwise pod-local.
3. Reachability is per-pod and process-global (GAP 3, named not fixed). Each worker's verdict reflects its own client operations, which is correct — a worker that cannot reach the writer must demote regardless of its peers.
4. The query path also records the verdict (GAP 4, named not fixed), so a read can promote the append path. Both speak to the same service, so this is sound, but it is an implicit coupling.

Related

• Umbrella: Event WAL Storage
• Data Access Boundary
• Deployment Validation
• Sessions Log — 2026-08-11 entry