Chronological log of agent sessions that touched the NoETL ecosystem. Each entry links to the durable artefacts (PRs, commits, wiki updates, ai-task issues) so future sessions can pick up where the last one ended.

cli#98 and cli#99 merged; semantic-release cut CLI 5.3.1 / executor 0.13.1 on every channel.

• render_template no longer re-scans substituted values.
• The subscribe spool test no longer depends on a 10 ms wall-clock window.

Wikis now describe the template gap as fixed (agent-playbooks Security and Architecture; cli agent-steps and executor-crate-architecture). The follow-up pointer bump closes #474.

Left for later: the PyPI wrapper still bundles CLI 5.3.0, without the template fix; defang_template_markers can go.

Opened #474 (Umbrella).

What shipped:

• noetl-tools 4.1.0 (#112) and 4.2.0 (#113).
• CLI 5.1.0 (#95), 5.2.0 (#96) and 5.3.0 (#97), with noetl-executor 0.11 → 0.13.
• PyPI wrapper 5.2.0 (noetl#710) and 5.3.0 (noetl#711).

Design.

• AgentSession is turn-by-turn because the CLI runs each agent tool as a sub-playbook, and its runner enters tokio itself.
• framework: claude bills an API key, and NoETL sees every tool call.
• framework: claude-code runs headless claude -p on a Claude subscription, with the API-key env stripped so it can't fall back to API billing.
• No Python anywhere, per the user's standing rule.

Credentials. Alias X resolves from NOETL_CREDENTIAL_X or a mode-600 file. http gains bearer / api_key / google_oauth. noetl credential google runs the installed-app OAuth flow with PKCE and stores a refresh token; a refresh token can never be sent as a bearer value.

Live. Four real Gmail inboxes, read-only, triaged through the published pip install noetl==5.3.0.

Cost. $0.35 → ~2–3¢ per run. Haiku, thinking: disabled, the Primary tab, compacted tool output and prompt caching. Tokens: 87,715 in / 6,148 out → 8 in + 8,158 cache-write + 2,126 cache-read / 2,667 out.

Defects the work found in itself:

• ⚠ Haiku spent a summary turn on a thinking block and the run was reported as successful with empty text. Now an error that names the block types (tools 4.2.0).
• ⚠ render_template re-expanded substituted values in HashMap order. Found by the agent e2e test, stopgapped by defanging model input; fix in cli#98.
• ⚠ The member-version guard failed cli#95 on an unmoved executor version. That's correct behaviour; bumped to 0.11.0.

Open: cli#98, cli#99 (flaky spool test). Owner: publish the Google OAuth app, or re-authorize weekly.

Wikis: tools: AgentTool, cli: kind: agent, cli: Credentials, agent-playbooks.

ehdb v0.9.0, server v3.143.0, worker v6.2.4. Closed #457; opened #466.

D2 cleanup, owner-approved and shipped. Delete-after-sealed. The design is the refusal, not the deletion: contiguous_durable_watermarks stops at the first non-durable part, because max-over-durable skips a local-only part in the middle and would authorise deleting a tail object whose records exist nowhere off-box. Both mutants — the naive watermark and delete-regardless — are caught by that one safety test. The consumer's flag defaults ON, the only one here that does, because omitting cleanup grows a bucket forever while the deletion is safe by construction.

⭐⭐ B3+D2 proven on prod with real data. Tick 60 s (not the shipped 15 s: at prod's measured 314 appends/hour, 15 s carries ~1.3 records/object and writes 5,760/day). A 32-minute sampler caught the whole chain — 188 unsealed records out in 3 batches, objects under tail/ counted externally with gcloud, the age seal at ~900 s, then all 6 objects reclaimed (3 per replica × 2), retained 6→0, bucket 3→0, tail_failed=0.

⚠ The first ~400 s read appends=0. That was the window, not the code: measured independently, prod had no execution traffic at all — only runtime_heartbeat writes. I did not trigger an execution to move a metric.

✅ #457 closed on measurement — /data 31% → 9%, local 1.2 G → 11.9 M, substrate 1.2 G → 75.5 M. Its "is growth linear or decelerating" question is moot rather than answered: retention changed the shape deliberately. ⚠ One goal item was genuinely not done — 0 of 19 alert policies reference any volume metric — so it was carved out as #466 rather than disappearing with the close.

⛔⛔ The prod worker roll is blocked by a broken release, and my first fix was wrong. release-worker failed for v6.2.2 and v6.2.3, each ~45 min in, producing no image — on commits whose cargo test --locked was green. noetl-tools = "~4.0" floated to 4.0.6, which needs async-nats 0.47 against this crate's 0.38 ⇒ two incompatible Context types in one graph. It surfaced now only because 4.0.6 was published after the last good release (2026-10-02), so the break was waiting for whoever released next.

⚠ Adding --locked did not fix it. cargo chef cook works from recipe.json, a skeleton reconstruction, and writes its own Cargo.lock over the one copied in — so the cook stage resolved 4.0.6 with --locked passed and the real lock in place. The durable fix is an exact pin (=4.0.1), which removes the resolver from the decision; verified by building the real image (1,008 crates, 0 errors, one async-nats).

⚠ Also corrected: the pools already run v6.2.1, not v6.0.0. I had read build_info off the writer — the only pod with a reachable :9090 — and generalised it to the pools. The skew was one patch, not two minors.

C2+C3 remain HELD. Election-before-enforce is absolute.

ehdb v0.8.0 (#404) + v0.8.1 (#405), server v3.142.0 (#531). Design PR ehdb#394 closed as superseded by the wiki page rather than merged — a design doc duplicating a maintained page is a second representation of one thing.

The gap, measured not argued. A2/A3 covered sealed history. A part is local-only until it seals, so the window was the seal interval, which B2 bounds and cannot close. On prod: SEAL_MAX_AGE=900, oldest_unsealed_age=719, unreplicated_oldest_age=0 — nothing waiting to upload, ~12 minutes of records on one disk.

replicate_tail writes the pending records as write-once per-batch objects under tail/ (never parts/, or plan_retention could drop them) and cold_load_replicated replays them, deduping by sort key against the manifest's max_sequence — the overlap is expected, since a part sealing after its tail objects makes them redundant by design. Per-batch rather than re-uploading the active part, which would be quadratic over a part's life. And no new read path: read_partition_after_limited already reads the active writer, which is the discovery that shaped the whole design.

⭐ Window: 900 s → the tick. Bounded, not zero.

⚠ The RED control carries the file. With replication off, 16 of 21 records come back — the loss is real, so every green result is falsifiable. With it on, destroying the local root and cold-loading from the replica alone returns 21 of 21, and 13 of 13 against a real GCS emulator. Four compiling mutants with distinct kills, including one that kills only the read-side tests and one that kills only the overlap test.

⚠⚠ A defect I introduced and caught before enabling anything. My v0.8.0 driver performed the remote put while holding the engine lock — and that lock is taken by shadow_append inside emit_events on the live write path. Every append colliding with a tick would have waited out the put: p50 77 ms, p99 234 ms, once per tick, indefinitely. A durability feature becoming a write-path latency regression. The measurement was already on #460 from the A2 work, and ehdb's uploader thread had documented the rule the entire time. v0.8.1 splits prepare (no I/O) → upload (free function) → commit (no I/O), with a source guard asserting the driver cannot regress and a RED control that restores the defect.

⚠⚠⚠ And a correction to my own shipped default. Prod's measured append rate is 314/hour, so a 15 s tick carries ~1.3 records per object and writes 5,760 objects/day — the object count is driven by the tick, not the data. 60 s is the better choice: still 15× better than 900 s, at a quarter of the objects. I chose 15 s before measuring the rate.

Stopped at emulator-proven-and-ready; not enabled on prod. Enable needs D1 (the tick) and D2 (tail-object cleanup — a data-deletion policy on the event log), both owner calls, now with measured numbers and exact commands on #460. D3 resolved as recommended: a failing tail put fails nothing, the watermark stays put so the next tick re-drives, and the failure gets its own counter because a failing replicator and an idle one both leave tail_batches flat.

C2+C3 remain HELD. Election-before-enforce is absolute.

Landed. server#528 (vendor/ removed, closing #463), ehdb#401 → v0.6.0, ehdb#403 → v0.7.0, server#529 (pins + backfill wiring). Prod: the replica bucket and IAM binding were created and A2 was armed (NOETL_EHDB_REPLICA_GCS_BUCKET, env 73→74, image unchanged, 1/1, 0 restarts).

A2 reached a real second domain — domains=[local-device:66320, remote-gcs], replica_set_size 1→2, survives_node_loss 0→1, single_point_of_failure 1→0 — and new seals replicate: over a 1853s window (the denominator is stated because a window shorter than the ~900s seal period would have proved nothing) the age seal fired at t=934s and the part, the manifest LATEST and a versioned snapshot reached the bucket, counted externally with gcloud rather than from the engine's own view.

⚠⚠ Then the acceptance proof refused to pass. Read back from the remote's own manifest: 40 parts listed, 39 at replica_count 1, one at 2. Uploads are enqueued at exactly one site — on seal — so arming a replica replicates nothing that already exists, and nothing retries because nothing considers it outstanding. Every durability gauge read healthy throughout, because they are computed from the declared replica set, not from the parts: survives_node_loss is a claim about configuration and was being read as a claim about data. ⚠⚠⚠ The resulting state is worse than an empty remote — write_manifest_to_all ships the manifest on every upload, so the bucket holds a manifest naming all 40 parts and the bytes of one. It reads as a complete store and is not a recoverable one, which makes the backfill a correctness prerequisite rather than an optimization, and means the cold-load-from-remote-alone headline is not yet claimable on prod.

Fixed, not redefined. backfill_under_replicated enqueues every under-replicated part onto the existing background uploader. Its own trap was one level down: durable_view stores local_path: None and the durable view is what seeds the in-RAM manifest at open, so after any restart a repair path that only knew fs::read(local_path) would have compiled, run, reported success and copied nothing. Hence UploadSource::Replica. Backfills carry separate counters because a part sealed in an earlier process has no seal→durable interval, and folding it into upload_lag_micros_total would divide the real total by a larger denominator and deflate a mean a capacity decision reads.

The worker bump produced the second fix. v0.3.2 → v0.5.2 had exactly one compile break across two minor versions: the new ShardLag.inflight. Defaulting it to 0 was refused on purpose — render_snapshot publishes that field and high lag with zero in-flight is specifically the stalled consumer reading, so a hardcoded zero makes a healthy bus publish an outage signal on every scrape. ClaimCoordinator::inflight() now exists.

⚠ Three of my own mutants did not compile before one did, and a mutant that does not compile is not a caught mutant — the first two "RED controls" on the server wiring proved nothing until the third removed the call block in a form that built.

⚠⚠ Rule 0a drift, found on the dashboard itself: Home's Active-umbrellas section was missing #457, #459, #460, #462 and #463 — every umbrella opened in the previous two sessions, including the one this session was about. Fixed in the same change set.

C2+C3 (election + fencing) remain HELD. Election-before-enforce is absolute. Framing kept honest: a backfilled sealed part survives node loss; the unsealed tail stays RF=1 and replication is async, so this is a bounded-loss consistent prefix of sealed history, not "no data loss".

Headline: the retention/archival tier is live and draining on prod — 1,472 of 6,251 archivable executions archived, every one also verified_durable and indexed, 0 failures, 4,431 real GCS puts. ⚠⚠ The disk-reclaim number is NOT measured and is not claimed: the prune floor correctly refuses while any archivable execution is unarchived, so enabling prune today reclaims nothing. Reclaim is gated on the drain finishing (~4 h), not on code.

Deployed: server v3.137.2, by digest a3bb8087 (rollback c46a032a recorded first). 1/1, gen==observed=69, revisions match, env 71 = baseline, 0 restarts, PRUNE absent.

The measurement #460 was decided without. Real GCS-put latency, n=300 in-region: p50 77 ms, p90 98 ms, p95 139 ms, p99 234 ms. Roughly 19x a ~4 ms local fsync, against a provisional estimate of "tens of ms" — so decision (i), synchronous per-record remote append, is closed harder than the guess closed it.

Two fixes proved themselves on the first pass after the roll:

1. The prune floor would have reclaimed ZERO. compute_prune_floor returned NoFootprint when every archivable execution is archived — which is exactly the steady state this drain produces. Every call would have returned Ok. Caught only by a positive control asserting that a fully-archived log does reclaim; without it, every negative test is satisfied by a prune that never prunes. This is also why HotExecution::max_sequence exists — added for this case, then never read.
2. collect_candidates could silently truncate. Capped at 200x100 and returned Ok on a short list. A floor computed from a partial population can sit above an execution never classified, and its records are then deleted. Completeness now travels with the data.
3. The put metric's "pin" fabricated samples. observe(0.0) on a histogram records a measurement never taken. inc_by(0) on a counter is harmless; this is not. In prod the only sample below 50 ms was that invented zero — on the very number the #460 decision turns on. Replaced with a pinned counter.
4. My own B2 gauge understated its own exposure. It stops at sealing, but replication is asynchronous, so a sealed part still has a window where the only copy is local. L0Engine::unreplicated_snapshot() has computed the whole durability window all along and had zero server callers.

server#518 / #519. GcsSubstrate implements ehdb's DurableSubstrate and declares Remote{provider:"gcs"}. It lives in the server, not ehdb-l0: the storage core has no HTTP client and a deliberately lean dependency set, and DurableSubstrate is the extension point for exactly this. ⚠ ehdb-core becomes a third pin that must move with ehdb-l0 — a different tag puts two EhdbError types in the graph.

⭐ The unit tests establish the domain algebra, which a substrate that declared Remote while silently dropping every upload would also pass. So the proof is a cold-load of the log from GCS alone, confirmed by listing the bucket outside the test: 6 sealed part objects spanning sequences 1-48, the FORMAT_VERSION guard and the manifest chain — 14 objects / 26,159 bytes, control bucket 0.

⚠ A correction: two GCS buckets are one failure domain, but they do survive node loss (both are off-node, and is_single_point_of_failure() is !survives_node_loss). The gauge that would catch a two-bucket plan is domains_distinct, not the single-point-of-failure one. Reading redundancy off the wrong gauge is how a correlated replica set reads as safe.

server#520. seal_max_age + seal_aged_parts() existed; the server configured neither and called the sealer from nowhere. ehdb's own doc names the trap: should_seal is consulted only on append, so a configured age with no timer is "inert on exactly the shard it was added for." Both halves now ship together, guarded including the spawn in main.rs. The exposure gauges publish unconditionally — including with the trigger off, because the tail is an RF=1 exposure either way. manifest_parts is the deliberate counterweight: smaller parts is the trade B2 makes, and that is the shape that filled the cmdbus PVC.

Sealed parts survive node loss. The unsealed tail does not. Replication is asynchronous. Recovery is a bounded-loss consistent prefix; B2 bounds it, B3 (not built) would remove it. Not "no data loss".

server #514–#521; releases v3.137.2 (DEPLOYED) through v3.140.1 (not deployed). Wiki: noetl-server-wiki — the floor's three refusals, the put-instrument lessons, and a new off-box-replication-and-sealing page.

⚠ Owner-gated: A3 needs a replica bucket + IAM grant (exact commands on #460); and PR #355 carries 38 commits stranded on a design branch, including pointer bumps for several prod rolls, now CONFLICTING — all four gitlink conflicts measured as "this branch is strictly ahead", STATUS.md needs a human call.

⚠⚠⚠ C2+C3 (election + fencing) remain HELD; election must be live and minting epochs, verified in shadow, before fencing flips to enforce.

Umbrella: #455 · #456 · #459 · #460 Prod: noetl_server_build_info{version="3.135.0"} · digest sha256:231e8eeb… · rollback sha256:94bf4efb… (v3.135.0 ← v3.131.0)

Closed a four-release gap in one roll — v3.132.0 (retention config, #459 P1–P4), v3.133.0 (retention metrics, P6), v3.134.0 (archive retrieval, P5) and v3.135.0 (the #460 A1/B1 instruments) had all been built and sitting in AR while prod ran v3.131.0.

1/1 ready, 0 restarts, gen 64/64 → 65/65 with currentRev==updateRev, 67 env vars identical, all three chain gates still armed, divergence 0 before and after. The 2 ERROR lines were my own probe requests — tower_http logging the two correct 503s from the archive endpoints; zero in the previous container.

noetl_ehdb_replica_set_size                1
noetl_ehdb_survives_node_loss              0
noetl_ehdb_replica_single_point_of_failure 1
noetl_ehdb_replica_domains_distinct        1   (vacuous at RF=1, labelled so)

And on the real pod:

WARN embedded event-log store is a SINGLE POINT OF FAILURE: no replica is in an off-node
failure domain … reality=replica_set_size=1 survives_node_loss=false
domains_distinct=true (vacuous at RF=1) domains=[local-device:66320]

The redundancy did not change. Its visibility did. What read as a green ehdb_l0_replica_domain_violations 0 now reads single_point_of_failure 1 — because that 0 was an unevaluated pin (the engine's check is gated on replicas.len() >= 2 and L0Engine::open makes a set of one), and this 1 is a verdict.

0 archive/prune/retention env vars on the StatefulSet, and all 14 retention series present and zero — archived=0 pruned=0 prune_bytes_reclaimed_total=0. Which is exactly what the unconditional pinning was for: a 0 that means checked and nothing ran rather than absent.

The retrieval endpoints answer 503 with the variable to set, and 400 on a malformed date — "not configured" and "nothing archived" stay different facts, verified on prod.

noetl_object_store_put_seconds_count{backend="gcs",outcome="ok"} 1 with sum 0 is my init seed, not traffic. observe(0.0) makes a histogram label set present — and for a counter inc_by(0) is harmless, but for a histogram it is a fabricated data point, so a naive mean is skewed toward zero by a sample I injected.

⭐ And the pre-seed is unnecessary for a histogram: build_info{version} already tells a reader whether this build has the metric. The pins are right for counters and gauges and wrong here. Not hot-fixed — the seed is subtractable, the first real number will be reported with it subtracted and said so, and the next server change drops the pre-seed.

Umbrella: #455 · #460 Landed: noetl/worker#348 (d135e97d) — a structural guard, zero src changes.

build_durable_stack wraps the shared backend when NOETL_EHDB_FENCING is set, and a decorator cannot be skipped by a caller who forgot. ⚠⚠ But it can be bypassed by constructing the store somewhere else — invisible to every runtime test, because a second site works perfectly and simply is not fenced. That is the M5 shape one level up.

Four guards, the two substantive ones RED-proven: a second FilesystemSharedBackend::open fails; a new unregistered append* in the durable-write modules fails. It prints its denominator (6 scanned) and asserts its own test-stripping slice is plausible before asserting anything about it.

"Build the stack from plain" and "read the setting after the wrap" are both compiler-guaranteed — plain is moved in both match arms, and the wrap is match setting {…}, a data dependency. My mutants failed to compile or were self-defeating, and a mutant that does not compile is not a caught mutant. Shipping those assertions would have been checks that cannot fire for the reason they claim. The finding is recorded in the test file: within build_durable_stack the bypass is impossible by construction.

⚠ And an over-constraint caught by its own control: my first ordering assertion failed a mutant that merely hoisted a read earlier — harmless. An over-constrained guard teaches people to delete the test rather than read it.

FORMAT_VERSION is 1 at both pins; ehdb-service/src is unchanged between v0.3.2 and v0.5.0; ehdb-feed/src differs only in the KEDA scaler.rs; ehdb-l0/src is 4161 insertions against 2 deletions. And the two do not share a volume. ⭐ The real consequence is capability, not correctness — the worker cannot use the D8 registry, supersede_key or the state gauges until its pins move, which is why C2 has a pin bump and a worker release in front of it.

Prod runs v3.131.0; the latest tag is v3.135.0. Four releases are built and sitting in AR undeployed — the retention tier (P1–P4), its metrics (P6), archive retrieval (P5), and the A1/B1 instruments. ⭐ One deploy delivers all four. Until then option (i)'s rejection still rests on an estimate, and it stays labelled as one.

P7/P8 is now blocked on user decisions (#460) plus the IAM grant for archival — there is no further safe autonomous work. What remains all changes durability or safety semantics: C2 (election + a dependency decision + a pin bump), C3 (⚠⚠⚠ an immediate outage with no election live, so C2 strictly precedes it), B2 (a new feature on a primary-serving store), B3 (a replication protocol), A2/A3 (the injection decision, then RF>1).

Not building any of them speculatively, and not inventing adjacent work to avoid idling.

Umbrella: #455 · #460 Landed: server#511 (1753084b) · spec 5ff3511 Observability + audit only. Nothing armed, no semantics changed.

Four always-computed gauges, seeded pessimistically (survives_node_loss=0, single_point_of_failure=1) because until something evaluates the replica set, "we do not know" must not read as "safe". ⭐ The alert keys on survives_node_loss, not on "RF < 2" — two replicas on one device also die with the node, and an RF<2 alert would have missed prod entirely.

A real bug the tests caught: distinctness first used the derived Eq on FailureDomain, which includes root, so prod's two copies on device 66320 compared as distinct. Now uses ehdb's own label(). ⭐ Which collapses Remote to its provider — so two GCS buckets are one domain, and any "spread across two buckets" plan must reckon with that.

I had said the tail fix was to wire with_seal_max_age into the tier. True that tier_store.rs never calls it; misleading, because the tier is a JSONL append file with segment sealing by rename, and with_seal_max_age is an L0Config knob. The equivalent — age-based segment sealing — does not exist, and would be a new feature on a primary-serving store. So it stays specced and flagged rather than built, and must default off per tier_seal_max_bytes's own fail-safe-off rule.

/data/eventbus  19.5G total, 12.7G used (65%)   ehdb-tier 11.4G
13 sealed eventlog segments, all 256.0–256.8 MiB   ← byte sealing works as configured
ACTIVE: eventlog 62.0 MiB · projection 129.0 MiB · catalog 7.1 MiB, UNSEALED ~7 HOURS

⭐ catalog.jsonl is the age exposure made concrete: a byte-only trigger bounds the high-rate tiers and leaves the quiet ones arbitrarily exposed — the opposite of the intuition that low traffic means low risk.

⚠ And one of my own errors: I read df /data, got the overlay at /, and briefly concluded the tier was on ephemeral storage. /data is not a mount; the PVCs are at /data/{cmdbus,eventbus,eventkv}. Checking the mount table rather than publishing the first reading caught it.

I had said the server links neither election nor fencing. True, and the wrong binary: the writer does link ehdb-reference, and the fence is already a selectable decorator (NOETL_EHDB_FENCING ∈ Off/Shadow/Enforce). What is missing is the election that mints tokens, not the fence that checks them.

⭐⭐ And the writer's own metric HELP text already states it: "0 means single-writer rests on StatefulSet replicas:1 alone, and fencing enforce would be an outage." Prod has the var unset, so the decorator is Off.

⚠⚠⚠ Which sharpens C3: with no election live, enforcing the fence is an immediate outage, not merely reduced availability — the writer would fence itself against an epoch nobody mints. C2 strictly precedes C3.

tier_store::{append, append_with_seal, append_batch}, tier_shadow::append, dataplane::append_domain_record, eventlog_backend::append_selected — consistent with ehdb_fencing_active 0. Remaining work is a structural guard so a path added later cannot bypass the fence silently, which is the M5 shape. ⭐ A decorator is the right answer to M5 because it cannot be skipped by a caller that forgot.

⚠ Worker pins ehdb at v0.3.2; server is at v0.5.0.

A P6 test went red in CI while passing locally. The signature named the cause — left: 0, right: 4242, and only a concurrent init_…_series() writes 0 there. My first hypothesis (a collision with the negative-age test, which writes 5) was wrong, and the 0 ruled it out. The race would not reproduce naturally: 0 failures in 82 attempts across three control configurations. Forcing the interleaving reproduced it in 47 of 20,000 reads — a ~0.24% window, which is why 82 single-read attempts missed it and one CI run caught it. Fixed with a file-scoped mutex; my new test file had the identical latent race and happened to pass.

Umbrella: #455 · new tracking issue #460 Landed: noetl/ehdb design/p7-p8-distribution-frontier.md (ece9ac7) Design only — nothing built, no prod changes, no durability-semantics changes.

Both of prod's "replicas" are on device 66320 (/dev/nvme0n5), and every durability signal reads healthy: replica_domain_violations=0, parts_under_replicated=0, parts_local_only=0.

That 0 was never evaluated. L0Engine::open wraps the substrate in a replica set of exactly one, and the failure-domain check is gated on if replicas.len() >= 2 — "one replica makes no spreading claim". The server calls open(), so the check short-circuits and the healthy path pins the counter at 0.

The code names the rung above: "an RF of N over one domain is an RF of 1 wearing a larger number." Prod is one rung below, where the check that would say so is unreachable.

⭐ Same shape again: FailureDomain::Remote exists in the enum and is constructed in tests only; survives_node_loss requires it, so that predicate is false by construction for every deployment, with no production caller either.

Verified in code rather than recalled: with_seal_max_age is called at exactly three sites (command_bus.rs:225, event_bus.rs:262, eventlog_backend.rs:1379), and ehdb/tier_store.rs calls it zero times. So the two buses and the eventlog backend all get the 5 s bound, and the tier — the store at risk — gets TIER_SEAL_MAX_BYTES = 256 MiB and no age bound at all: up to 256 MiB of acked events on one PVC for as long as it takes to accumulate.

• ⭐ EHDB is not etcd-with-sharding; it is a WAL with a lease. etcd has consensus and no shards; EHDB has shards and no consensus.
• FLP ⇒ a failover window can be bounded, never eliminated.
• CAP ⇒ P8 reduces write availability: a partitioned writer must stop accepting, which is the opposite of today's behaviour. The owner's trade.
• CALM ⇒ appends are monotonic so the append path stays coordination-free (a real result) — but there is no coordination-free linearizable cross-shard read, ever. A theorem, not a gap. The coordination-free offer is a monotonic consistent-prefix read, which is what replay, projection and audit need.
• Two claims to avoid afterwards: P8 gives mutual exclusion, not consensus, and a second copy is not a quorum.

Terminal+expired executions only, lagging by the retention window, and downstream of sealing — so it can never hold what the hot store lost. ⭐ What it does is retire the dependency blocker: DurableSubstrate is a sync trait and the substrate write runs on the uploader's dedicated std::thread, not the append path, so GcsSubstrate can be implemented in the server crate (where a GCS client is already live in prod) and injected — ehdb gains nothing. That is the option second-substrate-choice.md parked on and did not consider.

The state machine is already proven. The 2026-09 M5 incident is the proof of where the risk is: fencing that was "ACTIVE, ENFORCING" still served a superseded writer, because the guard sat on a path the publish skips. So the acceptance test is enumerating every durable-state mutation path — the bullet skipped last time.

A1 / B1 / C1 are pure measurement and unblocked — all three replace an inferred belief with a number, and two address a signal that is green for the wrong reason. B3 (streaming tail replica) and C3 (enforcing the fence) change semantics and are owner-gated, with three decisions listed on #460.

⚠ One claim deliberately left marked as an estimate: the GCS-put latency that option (i) was provisionally rejected on is unmeasured, so B1 owes a real number from the writer pod before the rejection is treated as settled.

Umbrella: #459 (new) · closes #457 Landed: server#508 · #509 · #510 · spec noetl/ehdb design/retention-archival-tier.md · noetl-server-wiki 35fe938 Also filed: #458 1440 tests, clippy clean. Flags default-off — nothing in prod deletes anything.

Four things already existed and were not rebuilt: the retention planner (plan_retention — drops whole parts below a sort-key floor and never splits a part), its executor (apply_retention), a GCS client already live in prod (ObjectBackend/GcsBackend, ADC via Workload Identity, 21,569 objects written), and the archive bucket itself. So: no new object client, no new auth, no credential handling.

⚠⚠ Four did not exist: no cloud DurableSubstrate (only LocalFs + InMemory, so nothing goes off-box and prod's "second replica" is the same PVC); apply_retention had zero callers in the server — built-and-dormant, the #332 pattern, verified with a control grep; no time- or completion-based floor; no per-execution layout, date index or config.

partition = shard_for_execution(execution_id, 1) and index_key = execution_id, so a part holds many executions interleaved by append sequence. An execution owns no part, execution_bloom cannot enumerate, and ⚠⚠ dropping an old part would destroy the early events of a still-running execution — data loss, since noetl.event is append-only with replay as the source of truth.

Resolution: archive per execution (exact), prune per part (coarse), with a floor as the bridge — min(global_sequence) over everything not yet safe to prune.

BEFORE: parts=9 bytes=10736
pinned floor: keep_from=2, pinned by execution 1002 (2 of 3 safe)
  dropped=0                        ← nothing pruned while a live execution is unarchived
  live events: 20 of 20 survive    ← the data-loss case, proven impossible
open floor: keep_from=101 (all archived)
  dropped=9  parts 9→0  bytes 10736→0  reclaimed=10736

Against fake-gcs-server through the production client: 120 records round-tripped set-equal; date=… listing keys_seen=12 ids=12 unrecognised=0; and with the hot store pruned, served from archive 6 of 6 and fetched 6 of 6 discovered ids. Verified independently with curl so my own reader was not the only witness.

The all-archived floor branch used max(min_sequence)+1 — the highest first sequence, far below the log tip — so a fully-archived log reclaimed nothing while every call returned Ok. Caught only because the end-to-end test asserts on reclaimed bytes rather than on the call succeeding.

• The store this prunes is a SHADOW. Verified with a control grep: its only readers are the shadow writer and the parity verifier, and GET /api/executions/{id} reads Postgres noetl.event, which is append-only and never purged. So pruning it cannot make API history unreadable, and wiring a retrieval fall-through into that endpoint would have been an instrument that cannot see its own target. The archive routes are for EHDB-format history, which is a real need — just a narrower one than the brief assumed.
• The parity verifier already has ReadVerdict::OutOfCoverage, so a prune will not make it report a false divergence. Nothing was needed there.

• ⭐⭐ Two independent flags for archive and prune, both default-off, so a state exists where the archive is provably complete and nothing has been deleted. Prune-with-archive-off is refused.
• ⭐⭐ One date-index object per execution, not an appended per-day manifest: GCS has no append, so that would be read-modify-write and lose entries. 60 concurrent archives on one date all survive.
• Order is the guarantee: upload → read-back verify → record durably → then prune.
• ⚠⚠ The pinned-floor hazard is a metric, not an assumption: one stuck execution blocks all reclamation while every counter reads healthy. The alert pairs two series — archiving succeeding while reclaimed bytes stay 0.
• Three read states, never two: an empty hot result is a miss that falls through, and a broken archive is an error, never a clean 404.

The server SA has no IAM binding on the archive bucket, so archiving fails closed. The exact gcloud command is on #459. Then: deploy → archive-only → check → prune → measure.

Umbrella: #455 · Umbrella: EHDB north-star registry Landed: server#506 → v3.130.1 · server#507 → v3.131.0 · gateway#55 → v3.13.0 · noetl-server-wiki 30810c6 · noetl-gateway-wiki aafbbe4 Filed: #458 Prod: server v3.130.1 (sha256:86283cb5…, rollback sha256:4fc8053c…) · gateway v3.13.0 (sha256:8d91c62f…, rollback sha256:02756b57…)

The gateway identified itself to the server in no way at all before this — zero references to any registration endpoint, no component_type, no background task in main.rs. So the registry could describe every component except the one users reach.

GET /api/runtime/topology on prod now returns Server: 1 · Gateway: 1 · Worker: 5, with the Gateway id equal to the exact new pod (gateway-6c585b4784-tzlmw), so it is the registration and not a stale row.

⭐ The heartbeat interval is read from the server's ttl_secs response rather than configured on the gateway. A TTL duplicated on both sides is a representation that drifts the moment one changes, and an interval derived from a stale copy would expire the lease it exists to renew. A missing ttl_secs is an error, not a default.

v3.130.0's Ehdb registration measured 0 on prod across 18 minutes. The hook sat on the events publish, and should_publish skips system executions — which is all of prod's regular traffic:

noetl_event_ingest_publish_skipped_total{reason="system_execution"} 22
noetl_event_ingest_total                                            (absent — family pruned)
noetl_command_publish_total{pool="system",route="legacy"}            4

Two structural tests passed the whole time. Both assert the call site is correctly gated; neither asserts anything reaches the gated path. That is #326 — existence is not reachability — committed in the same change whose message quoted #326 as the reason a different hook was placed correctly. The guard I trusted was measuring the wrong half.

Fixed on the command publish's Landed arm only. Deferred means the on-path budget elapsed and a background task is still retrying, so the writer may be down — registering there would claim liveness from a timeout, the same misclassification as tier-load reporting ok: true for an explicit refusal.

Adding the second call site exposed a real defect: EHDB_LAST_SEEN was a single global timestamp, so whichever bus published first throttled the other out for a full floor interval, leaving a live tier unregistered while every counter read healthy. One cap serving two addresses is the same defect as one cap serving two directions (worker#314). Now keyed per address and extracted as tier_seen_due so the decision is assertable instead of buried in a static. RED proven at runtime first, not as a missing symbol.

Registration is the last point at which the answer can still be "no": the catalog is EHDB-backed and noetl.event is append-only and never purged, so an inline credential cannot be removed later. ⭐ The scan runs before the write — the log keeps it either way, so the order is the guarantee (proven by mutation). ⚠⚠ The error carries the YAML path and marker class only, never the value, not even a 12-char prefix: a validator that quotes the secret into its error writes it to the log.

⚠⚠ Measured against 26,081 string values in 494 real playbooks before shipping a hard refusal — it produced one finding and the finding was wrong. vertex-ai.yaml embeds Python that reads a service-account credential, so credential-handling code names both keys while holding no material at all; the naive rule would have broken a legitimate registration. Requiring a PEM body took it to 0 findings in 26,081 values, and the false positive is kept as a regression test. ⭐ That fix then made the class unreachable behind the bare-PEM test — caught only because the test asserts the class is not PemBlock rather than merely is_some().

The fresh pod's noetl_command_publish_total is absent (family pruned — absent, not zero), and the prior pod had accumulated only 4 publishes, so the dispatch period is tens of minutes. Volume is not duration: the fix is a window sized to the mechanism's period, not faster polling. Reported as not-reached rather than inferred.

⚠ My first poll conflated absent with zero by summing matching lines; corrected to count series explicitly.

• A topology parser looking for a flat members reported live members: 0 against a response that actually carried 6 — the shape is kinds[]. The data was right.
• An unauthenticated GHCR probe 404'd on a tag that demonstrably exists; with auth it still 404'd on all four tags including a deliberate fake, so the probe had zero discriminating power. The digest came from the build log instead.
• A grep --include=*.rs unquoted in zsh returned "no matches found" for a file that exists.
• A Python str.replace silently matched nothing (a line-continuation that had been collapsed on write) and reported success; the test kept reading the old value.
• A lock-count assertion read 4 where the answer was 2, because the doc comments mention try_lock. Comments standing in for callers, again.
• One of my own new tests was decorative: it re-implemented the enable predicate inline instead of calling it, so it would have passed against any change to the real arms. Split out as disabled_by and confirmed by mutation.

#458 — 26 ops files still name the retired noetl-demo-19700101 (vs 19 naming the live project, as a control), including automation/agents/mcp/firestore.yaml with a hardcoded fallback at line 879. A wrong GCP project returns an empty collection, not an error, and the planner's 404-tolerance disguises it. ⚠ Whether the registered version matches the repo file is deliberately left open — drift here runs in both directions (#295).

The shadow→enforce auth flip — it now gates write endpoints, execution data, and POST /api/runtime/register, so an enforce flip needs a service credential on the gateway or registration starts failing (fail-soft: it keeps serving, and goes missing from discovery). #457 disk headroom · #458 · P7/P8 stay specced.

Umbrella: #455 · Umbrella: EHDB north-star registry Landed: ehdb #385, #386, #387 · ehdb-wiki 916e947 Filed: ehdb#388, ehdb#389 1,188 tests passing, clippy -D warnings clean.

RuntimeKind {Server, Gateway, Ehdb, Worker, Playbook, Execution} with discover(kind, now, ttl) turns D8 from a worker table into a service registry — without discovery by kind, "where are the gateways" means inferring kind from the shape of an id, which is a naming convention masquerading as a schema. watch_since gives a resumable cursor whose load-bearing property is that resuming returns nothing when nothing changed (a watch that re-delivers the whole log every poll is a scan, and the two look identical in any test that only polls from 0). An Execution registers ephemerally and vanishes by not being renewed — no tombstone, no reaper. SecretRef is provider://path[#version] and cannot render material, because the type never holds any.

⚠⚠ A real bug the tests caught: heartbeat reset the kind to Worker, so a renewed Execution silently stopped being discoverable one heartbeat after registration — a registration that disappears while being actively renewed.

⚠ An inert guard in my own new code: disabling MATERIAL_MARKERS entirely left every test green, because each material case was refused by a different rule. Fixed by adding the three cases only it can catch — material smuggled inside an otherwise valid reference.

⭐ Memory is O(parts), not O(records) — exponent 0.52, fitting ~16.6 KB fixed + ~1.8 KB per part, so 1 M records at seal_max_records=1024 holds ~1.7 MiB. Part count is the memory signal and merge is the memory control. The control ran first, because a counting allocator that is not installed reports a confident 0 bytes of growth — which is also the ideal result.

⚠⚠ No ehdb-l0 test had ever spawned a thread. The engine is a single writer by construction (append_record(&mut self)), and throughput falls 10% from 1 to 8 threads. And a shared AtomicU64 sequencer plus a Mutex is not sufficient: minting the sort key before taking the append lock reorders 547 of 1,600 appends (34.2%) while losing and duplicating exactly nothing — ids stay unique and dense, only their order is wrong, which is why it is invisible in production. That is #362's mechanism, the one that rolled back the v0.4.4 chain-store ramp, reproduced in 20 ms. My own first draft asserted the sequencer was safe.

⚠⚠ A standing replication deficit was invisible. The uploader assigns p.replicas = locations — the successful writes only — so a part holding 1 of 2 copies is is_durable(), on_upload_done fires, and the age-based durability window reports it done. replica_writes cannot fill the gap: cumulative, so it climbs while the deficit stands. New parts_under_replicated gauge, proven with a write-refusing substrate.

⚠⚠ Batching is worth 530x on the drain path (1.353 s vs 2.55 ms for 4,000 records) — what #344's never-batching relay cost. And the batch cap that fixed #298's unbounded memory makes a drain above the cap quadratic (exponent 2.06). Both real; the tradeoff had never been measured.

⚠ poll_assign without acking is 2.86x slower than with — the expiry scan walks the whole in-flight BTreeMap (keyed by sort key, not deadline) on every poll, so lazy acking is O(n²) in the consumer. Doing less work costs more.

⚠ inflight is not derivable from lag: measured lag 10 with 0 in flight (stalled) and lag 10 with 4 in flight (busy) — identical lag, opposite conditions. The accessor had existed all along with no render_* emitting it.

CI ran cargo bench --no-run — it compiled every bench and executed none. Running them is most of the value, because each case asserts in its timed region that it got the records it claims; that is what caught ChangeFeed::poll returning 2,000 of a 10,000 backlog. ci/bench_ratios.py gates the denominator (expected=13 parsed=13) and monotonicity, and reports the headline ratios — gating those would fail CI the day somebody improves the drain path.

⭐ Cross-machine, first run: batch ordering 530.5x local vs 528.3x on the runner (machine-independent); per-poll cost 32.0x vs 16.4x (factor of two). A "per-poll ≥ 25x" threshold would have failed on the runner, not the code.

A part count that matched the parts directory (1 part for 16,000 records, wrong by ~60x and plausible-looking); a println! whose labels did not match its arguments, printing under_replicated=200 on a single healthy replica; a SERIES row count of 10 against a real 31, because only 10 rows are short enough for rustfmt to keep the string on the opening-paren line — a line-oriented grep over Rust source counts formatting, not code; a gauge called inert by a test dataset that defines no dedupe_key; a "not measured yet" list that understated work already shipped; and an awk summary printing failed=0 with a FAILED line on screen, because ok. and FAILED. shift the field positions.

noetl/server pins both ehdb-feed and ehdb-l0 at v0.4.5, and the two must move together (different tags put two ehdb-l0 versions in the graph). ehdb has no semantic-release.yml, so its tag is manual. None of P1–P5 or the new gauges reach the server until a tag is cut and both pins move. P7 (unsealed-tail replication) and P8 (writer election + fencing) remain specced and deliberately not started.

ehdb#392 + ehdb#393.

RED first: 2,400 records before, 2,400 after — the merge ran and collapsed nothing. Dataset::supersede_key + a compacting merge_once: 4,608 stored records / 65.02 ms → 500 / 7.13 ms, converging to exactly the live set, per-record cost identical across every sealed row.

⚠⚠ Eligibility was narrower than the issue assumed. Only VectorDataset may opt in. RuntimeDataset (D8) is disqualified by watch_since — the op-log reader I added in P2 — because compacting it would delete the history a watcher resumes from. One history reader disqualifies a dataset.

ehdb is releasable. It had no release automation; every v0.4.x tag was hand-pushed. GH006 and the bot-tag-doesn't-trigger trap are both avoided by construction. ⚠⚠ Gated off behind EHDB_RELEASE_ENABLED and verified by merging: the job skipped, tag still v0.4.5. Next version computes as v0.5.0 — cutting it and moving the server's two pins together is an owner decision, and nothing reaches prod until it happens.

⭐ The dry-run guard caught a real defect in its own first two runs: semantic-release reads its branch from env-ci (GITHUB_REF), not git HEAD, so a pull_request event reported refs/pull/N/merge, matched no --branches, analysed nothing — and exited 0. Checking out the head ref changed nothing, which located the mechanism.

⚠ #393's first push produced zero workflow runs at all; an empty commit dispatched it. Same symptom as catalog#28's "zero CI runs ever dispatched" — several probes went into hunting a YAML defect that did not exist.

All three items done and verified. The headline is not the completions — it is that travel/playbooks/catalog/calendar/list carried six separate defects, every one producing a clean COMPLETED, 0 failed steps, and an empty result.

⚠⚠ #3 fires BEFORE tool-kind validation, so it masked #1. My earlier report that fixing the tool kind made the playbook runnable was wrong, and only executing it showed that.

⚠⚠ #2 is why none of it was noticed. A wrong GCP project returns an empty collection, not a 404, so one symptom covered five causes and never looked like a bug.

⚠ #6 was invisible to a 29-check guard that execs each step's embedded code:. The step bodies were correct; only the declaration was wrong. Three of the new checks are therefore structural over the YAML, which no fixture can replace.

Proof it reads the live project — same thread, same registered playbook, only gcp_project differing:

Firestore holds exactly 12 documents at that path.

Baseline 0. Now 6, all READY, and set-equality exact against firestore.indexes.json. firebase CLI is absent, so gcloud firestore indexes composite create was used; the account carries roles/owner, so no permission blocker.

53/53 by set-equality, all at v1, 0 rows removed. Audited before registering: all 53 parse, 53 distinct paths, only kind: postgres, all have a start step, and no GCP project pins — they target external Postgres by keychain alias (adiona_actor ×49, adiona_migrator ×4), so the Firestore project does not apply to them. Catalog went 1395 → 1453; the +58 reconciles exactly as 53 adiona + the 5 calendar/list versions.

• Prod server is v3.123.2, not the v3.122.0 my index claimed.
• I read registered=53, failed=0 and simultaneously printed "non-success: 53" — my filter looked for "success" when the status word is "registered".
• Two probes returned a confident empty from a mistyped scratchpad path and from cargo missing on the PATH I exported.
• mk() {…} and $K get pods both died on zsh traps already in the index.

The port-forward carried every prod reading, and the negative control confirmed it: killing the forward killed the probe.

travel#135–#139 · ops#324 · wikis 84b8cc5 + 537f3d6

The business-catalog half of the two-catalog split, built where it belongs: Firestore, reached from playbook steps, never EHDB.

There is no firestore tool kind and none was needed. The 25 ToolKind variants are Http Postgres Duckdb Ducklake Python Workbook Playbook Playbooks Secrets Iterator Container Script Snowflake Transfer SnowflakeTransfer Gcs Gateway Nats Shell Artifact Noop TaskSequence Rhai Subscription Wasm — Gcs is Cloud Storage, not Firestore.

Four connector limits, read off the registered implementation rather than its docstrings, and documented rather than worked around (#451): AND-only where over = < <= > >= array-contains; the query built without allDescendants, so no collection-group queries; limit capped at 500 with no cursor; and delete_doc that does not recurse, so no cascade.

I reasoned my way to a read/write project split-brain — the inline read defaulting to shastaratech-noetl-prod while the MCP write path defaulted to noetl-demo-19700101 — and went to check instead of publishing it. The Firestore data refuted it: slot_state is live in shastaratech-noetl-prod to 2026-09-30 and frozen in noetl-demo-19700101 since 2026-08-09. Both paths use the live project.

But the check found the real shape. The registered connector says shastaratech-noetl-prod; the git copy said noetl-demo-19700101. Re-registering from git would have done two things, neither of which errors:

1. Repointed the entire travel business catalog at a project frozen two months ago. Both projects hold a (default) database and a chat_threads collection, so the wrong project returns an empty collection, not a 404 — and the planner treats an empty slot_state read as "a brand-new thread… not an error". Every conversation would have looked new.
2. Dropped half of every turn's writes. batch_write's set_docs_extra / append_events_extra keys — which persist_all_atomically depends on — are absent from the git copy, and absent keys are ignored silently, not rejected.

Third: travel's playbooks/catalog/calendar/list, registered in prod as v1, declared kind: agent — which could never have run — and pinned the dead project.

And the travel repo had no index declaration file at all; firestore.rules governs client access and says nothing about indexes. Every query in the new model needs one. firestore.indexes.json now declares the six — a declaration, not a deployment.

Borrowed from a relational catalog structurally, not schematically: a recursive CTE becomes a materialised ancestor array queried with one array-contains; localization becomes items/{id}/content/{lang_code}, where the document id is the language; attrs is a map so Firestore auto-indexes attrs.<key> and the EAV rows arrive with the parent.

⚠⚠ Firestore enforces no referential integrity, so nullable and cardinality are advisory and on_delete: cascade must be written as playbook steps — an orphaned subcollection is invisible to every query under its deleted parent while still billed and readable by path. Soft delete preferred.

scripts/business_catalog_items_test.py: 29 checks, runs the playbook's own embedded code, no network, no credentials. It carries a RED control feeding query_collection's documents key to a batch_get_docs reply — the exact wrong shape my first draft used — because that bug produces an empty join, indistinguishable from "no item has a translation". 4 of 4 injected mutants caught, green baseline after restore. And verified running, not merely present: guards discovered: 6 … guards run: 6 failed: 0.

⚠ Three of my own measurements were wrong first: the ops checkout was 28 commits behind, the travel checkout was on a feature branch without adiona/ at all, and two Firestore probes returned a confident empty because the scratchpad path was wrong and because cargo/curl were not on the PATH I exported.

noetl/travel#135 · noetl/ops#324 · travel#134 closed · wikis a28d21e + 537f3d6 · ai-meta@6ea4025.

Architectural clarification: "catalog" names two different concerns in NoETL, and conflating them is the one mistake worth designing against.

The internal catalog holds the playbook that reads the business data. It never holds the business data.

Two files modelled business entities as internal objects, and I wrote both:

• foreign_key_identity.rs — table_row with categories/10, category_types/1, trip_category/1, trips/100: adiona's business rows. Re-fixtured onto real internal references, playbook → mcp and playbook → credential.
• localization.rs — resource_type: "category" with category_name in en/de/ka: adiona's categories and its translations, business data twice over. Re-fixtured onto memory, a real internal type.

A fixture teaches what belongs in a store even though the store cannot tell the difference. ⚠ My first re-audit had a bad denominator and I published it. It scanned one idiom, resource_type: "x", found 15 literals and reported the types as playbook, subscription, memory. There are three idioms; with the helper constructors and ResourceType:: literals the population is 46 and the set is six — playbook, subscription, memory, credential, mcp, agent. The narrow scan missed three of six, including both halves of the FK fixture it was meant to be checking. All six are internal types, so the conclusion held and the measurement did not.

"No business entity name appears anywhere" is also false as written. By data vs prose: categories 4/4, category_type 3/3, trip_category 2/2, table_row 1/1, item_content 1/1 — all prose, zero as data. The 11 mentions are doc-comments recording where the structural ideas came from, provenance worth keeping. Two more hits were substring artifacts of my own greps: 9 of 14 trips are round_trips, and every flights hit is the playbook name muno/playbooks/flights-details.

It was added here citing adiona's 24 _translate / _content tables — which are business tables. adiona.item_content carries lang_code in external Postgres, read by adiona/playbooks/catalog_list.yaml through kind: postgres + auth: adiona_actor. So no internal object type has a demonstrated localization need; the lang dimension is kept only because it is already built, tested and inert, and is not built on further.

From real evidence in noetl/travel: 53 adiona/playbooks/*.yaml reach the external adiona.* schema via kind: postgres; flights/hotels/places/documents go through mcp/duffel, mcp/hotelbeds, mcp/google-places, mcp/firestore. That is exactly what execution-model.md already mandates — a data touch inside a playbook step under its policy block, credential by keychain alias. The business catalog is that pattern, not a component to build.

Because adiona is a business catalog, and its relational model already lives on the business side. What the internal catalog took was structural — one polymorphic identity instead of a table per entity type, the EAV collapse, the self-referencing taxonomy, the typed value union. Its tables were never the target, which is why there is no DDL parser and why acceptance is set-equality over noetl's own objects.

catalog#25. travel#72's registry note corrected too: the model as an object is plausibly internal, the weights are neither (object store, with a catalog holding the reference), and training data drawn from the domain is business data that must not enter EHDB.

Set out to build the graceful-shutdown mirror drain. It is already built, already called, and measurably working — and the premise for needing it was my own error.

enqueued             11330
drained              11330
shutdown_abandoned       0      <- the loss counter
mirrored on 1st try  11323
recovered on retry       7
mirrored + recovered 11330      == enqueued ✅
parity match          1332 / divergent 7   = 99.48% agreement

Every enqueued event reached the tier. flush_on_shutdown() (ehdb_eventlog_mirror_queue.rs:449) is called from main.rs:1464 after graceful_shutdown on SIGTERM, and its own comment already states the stake: "Dropping them on SIGTERM would produce a permanent missing_event."

1. attempt_total{outcome="unavailable"} is not a loss counter. It is AttemptOutcome::Retryable — a failed attempt that gets retried. The counter to read it against is mirror_total{outcome="recovered"}, which says all 7 affected batches succeeded. I built a causal story ("one lost batch per roll") on a retry counter.
2. I attributed unavailable=2 to rolls that predate the pod. The pod started 19:23:33Z; a counter cannot record anything before that.
3. ⚠ The discriminator added for this very issue already answered the sub-question — send_error_total{kind="timeout"} 1, every other kind 0 — and was sitting unread while I hypothesised. It exists because a prior session could not tell a timeout from a refused connection.

divergent = 7 equals recovered = 7. A batch that fails and recovers on retry arrives late, so a comparator sampling in that window finds the tier short and records a kind="count" divergence — a timing artifact of the retry path, the same "two parity oracles disagree by construction" family as #325. ⚠ Not claimed as proven identity: counters cannot establish that, and proving it needs execution ids the metrics do not carry.

There is no drain to add and no loss to stop. Inventing a fix here would mean changing working code on a story the data refutes. The residual — whether the comparator should tolerate an in-flight retry window — re-tunes a paging alert and is owner-gated, like #325.

Measure-first throughout; every claim below was read from the live system or the repo, not from notes. Two of my own probes were wrong and are recorded as such.

v3.123.2, digest 49eb47fc, 1/1, restarts 0, env 67. Chain gates armed and working — all three present in the process env, chain_populate{extended}=229, and bad states (diverged / fork / multiple_roots / stale_log) at 0. #410 applied and closed.

⚠ Eventlog divergence is 7, kind=count only, with mirror_attempt{unavailable} still pinned at 2 — the two env-only rolls of 2026-10-06 and nothing since. The pair is the reading: flat unavailable with climbing count means a bounded deficit being re-counted, because the comparator counts comparisons rather than distinct diverged executions. #343 updated.

⚠ A Rule 0a stale version assertion killed: the Ecosystem-map noetl/server cell read "v3.123.1 released · v3.123.0 in prod, digest 091ff47f". Prod is v3.123.2 / 49eb47fc. The cell now says so and tells the reader to take the version from the pod digest, never from the table.

⚠ The board had drifted from the tracker, and the issue's own state wins: 8 closed issues showed In progress (#251, #297, #339, #341, #342, #378, #402, #415), and #241 was OPEN while the board showed Done — the dangerous direction, a ticked box over work that is not finished. All nine flipped.

⚠ ai-meta has 104 open issues, not 100 — --limit 100 is a cap, not a page size, a trap already recorded for gh issue list and gh project item-list.

1. The CI-gate table — wrong twice, and the published figure was also wrong. The first pass reported catalog as || true because the grep matched || true inside comments that say "NO || true" (comments counting as code). I corrected that and published "3 of 8 gate; 5 cannot fail" — which was also wrong, for a different reason: I measured the local submodule checkouts, which are 7–23 commits behind origin/main, so I read the pre-#374 state. The truth: all 8 Rust repos gate clippy with -D warnings, all pin 1.99.0, only cli carries 2 named allows. #374 and #378 both closed correctly; there is no clippy-gate debt. ⚠ This happened inside the sweep whose purpose was killing stale assertions, in the same edit that told readers to "read the version from the pod digest, never from this table". Measure a repo claim against origin/<default>, never a local checkout. ⚠ And a third defect in the same three-line probe: head -1 of the clippy lines picks cargo clippy --version, so catalog would have misread as ungated even on a fresh tree.
2. gh issue create --json is not supported and returned empty, which I nearly read as a failure. Verified by listing before retrying — an empty return is not evidence either way.

Two untracked findings from the catalog object-type enumeration are now filed:

• #446 — subscription is a first-class catalog type in the server with no noetl.resource row for noetl.catalog.kind's foreign key to reference. Deliberately does not diagnose whether the FK is enforced on prod; related evidence is that 'Playbook' (875 rows) has no row either.
• #447 — a resource type is not a tool kind: two namespaces, 5(+1) vs 25, with agent/mcp valid in one and rejected in the other. Conflating them is what the catalog's first ingest allowlist did in spirit.

Updated with re-measured state: #343 (above), #422 (reader DONE via server#495; gauge still absent — 0 series in a 66,688-byte scrape, so a real absence rather than an empty scrape; replay still absent), #406 (both upstream waits still stand; closing it would lose the reason the exposure is accepted).

catalog#23 — README rewritten to what is built (5 crates, the full API surface, the four-dataset invariant, the three reverse indexes, adiona as inspiration only, and a table of what was consciously dropped), a CURRENT SCOPE banner at the head of the spec because earlier sections still read as though adiona were the target, and the staged wiki page marked STAGED NOT PUBLISHED with the exact three publish steps.

⚠ The noetl/catalog wiki repo still does not exist — catalog.wiki.git returns Repository not found because GitHub creates it only after a first page is saved through the web UI. One manual step, nobody can do it from a clone, and nothing claims otherwise.

catalog#19–#22. 142 tests, fmt + clippy -D warnings clean, AC3 still exactly 4 Dataset impls.

Generic catalog for noetl's internal object types. No SQL surface. API-only. adiona is cited as inspiration for the relational/EAV patterns only — its schema is not mapped in, there is no DDL parser, the acceptance proof is not an adiona slice. Dropped: the catalog-schema DDL crate, the adiona round-trip test, further localization work.

Five seeded rows, and noetl.catalog.kind has a FK to it. Their executable/catalog flags map onto ResourceType's two fields exactly — which is why those fields existed. playbook·t/t · credential·f/t · mcp·f/t · agent·t/t · memory·f/t, plus subscription as the sixth, de facto.

⚠ Two findings for noetl/server: subscription is not seeded in noetl.resource while the server treats kind: Subscription as first-class (nothing for the FK to reference); and a resource type is not a tool kind — ToolKind has 25 variants, and agent/mcp are rejected tool kinds while being perfectly good resource types.

matches!(type_name, "playbook" | "subscription") rejected four of noetl's own six types. Every kind now registers, with reporting (new_types) replacing the gate. SkipReason::UnknownKind and its metric label were deleted — unreachable, and an unreachable skip reason is an inert series reading 0 forever. ⚠ That exposed a gap in my own guard: it checked each label was produced by some arm, hand-built — existence-vs-reachability inside a reachability test.

RelationKind::References(ForeignKey{…}). Making the kind carry data broke the edge identity three ways, because both edge-key sites used format!("{:?}", kind): one reference re-asserted with corrected metadata became 2 edges; the reverse answer returned "References(ForeignKey { nullable: false, … })" as its kind label; and a retraction matched nothing, leaving the edge unremovable. Fixed with discriminant().

15 endpoints, four query shapes (by type / by attribute / by relation / by reverse-relation). Every read returns the full set with its count, never a page — RED-proven at the HTTP boundary by crippling each fold and watching a 200 OK carry a wrong answer: by attribute: 1 of 7, callers: 2 of 5. The acceptance proof runs over HTTP with set equality, including a seventh unknown type accepted-and-reported.

Auth mirrors the server: 503 when unconfigured (no permissive default), 403 otherwise, constant-time. ⚠ The unsafe_code = "forbid" lint forced a better design — the token lives in ApiState, because the std::env version could only be tested by mutating a process global. ⚠ Writes serialize behind a Mutex: the honest shape of EHDB's single-writer assumption.

AC10 narrowed, not ticked: its "no consumer on a serving path" half is closed, but the literal wording is about noetl/server's surface and execute path.

CI caught a race I introduced — the metrics delta test read scanned=+6 for a run that scanned 3, from a concurrent test sharing the process-global registry. The hazard was documented in a comment I wrote, in another file. RED: 2 of 8 runs failed without the lock, 0 of 8 with it; a ~25% rate is why one local run passed. And a test of mine was wrong while the code was right: I expected kind: "Invokes" to be refused, but normalising it to canonical invokes is correct per #429.

catalog#15-#18 merged on green. 120 tests, fmt + clippy -D warnings clean, AC3 still exactly 4 Dataset impls.

The relation path had never run against real data: the adiona 53 are leaf playbooks, so relations=0 there is correct and proves nothing. Over muno/* it is correct, matching ground truth derived by reading the documents — 7 reference occurrences folding to 5 distinct edges (flights-details names duffel twice, profile names firestore twice, and an edge asserted twice is one edge).

⚠ Correction to my own figure: "36 registered muno/* playbooks" was the prod catalog registration count. travel@origin/main carries 12 documents with muno/ paths, 4 of them with child references.

What it surfaced: nothing answered "what calls X" — mcp/firestore has 2 callers and finding them meant scanning every path. Third reverse index (\u{1}to/<path> in c2), still no fifth dataset.

⚠⚠ Three mutations, and the first two did not fire. I predicted the hazard as "folding by edge_key() collapses all callers". Wrong twice: edge_key()'s reverse arm already keys on from_path, and relations_to never calls it. Only mutating relations_to's own closure fired — 1 of 2 callers, and 1 of 40 on a shared dependency. A mutation that passes is not reassurance; it means the mutated thing was not holding the property up.

11 series, six non-test recorder call sites verified by grep. ⚠⚠ With the unconditional pins removed the whole scrape is 155 bytes containing only build_info — all three counter families pruned by Registry::gather, the shape in which the prod gateway served a 200 with zero bytes. ⭐ build_info surviving is precisely why it is the right discriminator: that scrape says "the binary has the code and nothing is pinned or firing", a different diagnosis from empty.

reclaimed is the one that matters — catalog#11 found the merge driver doubling disk and nothing would have shown it. ⚠ The counters are per-process, so a standalone catalog metrics is all-zero by construction; the subcommand says so rather than letting a zero be misread. No new knob, so no deployment-spec row owed.

9 cited, 1 deliberately open, 0 broken. Each names a test function and the audit asserts it exists in the file it claims — proven by mutation (a renamed test + a missing file ⇒ 2 broken citations). ⚠ AC8 was covered but untraceable — zero mentions of AC8 anywhere while the behaviour and its test existed. AC1 and AC9 had no test at all, only prose. ⚠ AC1's naive form false-flags a "SELECT 1;" inside a YAML fixture — SQL in a document the catalog catalogues, not SQL it runs.

• AC10 needs a server consumer — nothing in noetl/server links catalog-store, so the crate has no presence on any serving path.
• The noetl/catalog wiki repo does not exist: has_wiki=true but catalog.wiki.git returns Repository not found, because GitHub creates it only after a first page via the web UI. Content is staged in-repo (#18) and ready to push. Needs a human with repo access.
• Spec §13's four open questions (noetl.registry merge, DDL ownership, Observed write volume, own PVC vs shared) are all decisions.

Recorded in STATUS as the explicit stopping point rather than re-emitting a report.

catalog#13 + #14. 109 tests, fmt + clippy -D warnings clean, AC3 still exactly 4 Dataset impls — the constraint the whole design had to respect.

uses_credential.<alias> was half a feature: the forward index is keyed by path, so "every resource using alias X" — the query that matters when rotating a keychain alias — meant reading every path. And catalog list printed "a full path listing needs an index this store does not yet keep".

A fifth dataset would answer both and break AC3. Instead each index lives inside an existing dataset as a second row kind, distinguished by its index key — which works because ehdb-l0 matches the index key by exact string equality (engine.rs:1489):

The sentinels are control characters: a path reading attr/uses_tool.postgres would otherwise silently answer a reverse query. A forward write intruding on either space is refused, since the path comes from a document.

Partial, and returned successfully. The middle row was unpredicted and is worst: once any resource unsets the attribute the latest op under the shared key is a tombstone, so .last() answers "nobody uses this credential" while 48 do — green-lighting a rotation that breaks all 48. Every assertion is set equality; a count of 49 can still be the wrong 49.

git truth adiona_actor:   49   catalog: 49   SET EQUALITY, byte for byte
git truth adiona_migrator: 4   catalog:  4   SET EQUALITY, byte for byte
union 53, intersection 0 · list --type playbook -> 53 · --type Playbook -> 53

1. Archiving a version is not archiving the path. Got the live/dead rule right, then read the type name from versions() — which folds latest-op-per-version and emits only Registered, so once every version was archived it returned an empty vec, the fallback returned early, and no tombstone was written at all: the path stayed listed forever. The same under-reporting the function exists to prevent, by the opposite route.
2. partition() derived from r.path would send a reader to the wrong shard, returning nothing rather than erroring. Both datasets now derive it from index_key().
3. Synthetic rows in a forward fold are excluded from the input, not the output — a reverse row's name() equals a real attribute name and would shadow it.

Also closed a self-contradiction: ingest left resource_type("playbook") as None while resources_of_type("playbook") returned all 53, so the CLI printed "type playbook is not declared in this store" above a successful listing of 53 of them. Ingest now declares on first use.

Scope, measured: a kind: sweep across travel/noetl/ops finds 183 Playbook plus Deployment/Service/ConfigMap/ScaledObject/Secret/PVC/VMRule — Kubernetes manifests, not NoETL internal resources, so out of scope. No third NoETL resource type exists in the tree to add.

Next, by evidence: the adiona 53 are leaf playbooks, so the relation path has never run against real data — the 36 muno/* playbooks do call children and are the corpus that would test it. And Ticked/Ingested are returned but recorded nowhere.

Net state = the original baseline. I disarmed the three gates at 18:36:35Z under an explicit approval, then the decision rule became "disarm only if it is blocking progress." Applying that test returned NO, so the prescribed state was armed and I restored it at 19:23:33Z. NOETL_CHAIN_{POPULATE,ADVANCE,SOURCE} armed, env 67, generation 56, revision 7c97b8df, image unchanged throughout at 49eb47fc (v3.123.2). ops#323 restores the as-built; ops#322 was the disarm.

Working and failing safe, so disarming mid-flight would remove a measured-good capability for no stated benefit.

• ⚠ Absent is the correct off, not 0. The off-states are different shapes: the booleans are an allow-list, while SOURCE's _ => None is a fail-safe for unrecognised values, so 0 reads off only by landing in those branches rather than as a declared default.
• ⚠⚠ Never (dis)arm by applying the ops manifest. It pins image v3.118.0 against a prod running v3.123.2, and a server-side apply conflicts on .image with field manager kubectl-set — it would roll the image backward into the known-wrong event_id ordering its own header warns about. Targeted JSON patch instead: test ops asserting each name at its index, remove in descending order, dry run diffed whole-object (exactly 6 changed leaf paths, every survivor byte-identical).
• ⚠ Either direction needs a positive control, because the patch touches spec.template: the pod rolls and every counter resets, so chain_populate_total = 0 is also an idle new pod. The cleanest discriminator found — an armed boot logs event-chain DDL skipped as the chain path initialises, where a disarmed pod logged zero chain lines at all.
• ⚠ Honest limit: chain_populate has not moved since the re-arm, and the reason is measured rather than hopeful — chain_head_hydrate is 0 across all outcomes, so no execution has advanced on the new pod and there is nothing to populate (worker logs: last execution 19:20:04Z, previous 16:20:35Z). The proof this config populates is the pre-disarm pod's extended=50 over 3h55m. No synthetic execution was driven to force the counter.
• Collateral from two rolls: exactly 2 heartbeat-failure WARNs per worker, one per roll, each self-healed on the next beat.

adiona/frontend triage — 8 of 8 open issues, zero frontend code touched. One backend fix (travel#133, issue #59: a login-time profile read was writing to Firestore on every login — not data loss, since merging {} changes no field, but a write on the login critical path, and set with merge creates the document, so after one login "never saved a profile" became indistinguishable from "saved an empty one"). One deployment gap surfaced and deliberately not actioned (#5: 53 adiona/v1/* playbooks exist on noetl/travel@origin/main, 0 registered on the catalog the SPA reads, while 36 muno/* are — registering them is a prod catalog change and the owner's call). One blocked on product input (#55: the three use-case .docx yield 22 scenarios, D6 is absent — numbering jumps D5→D7 — and the three documents use three divergent canonical-slot vocabularies). Four frontend-only, one split.

⚠ A false zero in my own verification: the coverage check filtered comments by user.login == "Kadyapam" (the git user name) and reported mine=0 on all eight while I held eight comment URLs. The API login is lowercase. Trusting it would have double-posted every comment; the contradiction between the filter and the URLs is what caught it.

catalog#11 — the merge driver was doubling disk. The previous phase proved the part count fell 25 → 4. It did, and bytes rose 42% in the same tick: 94,784 → 189,568 (2.00x), 54 part files and substrate objects for 4 live parts, because caller-owned reclaim_orphans was never called. The test measured the number the fix was about. Now reclaimed=48, files 49 → 8, 1.02x. ⚠ The test's byte assertion was wrong on first draft while the code was right — a merge consolidates parts holding the same records, so bytes are flat by construction; the property that discriminates is that bytes must not grow. All five caller-owned EHDB lifecycle calls are now accounted for: three driven by tick(), with apply_retention (drops parts below a floor — a resource whose only Registered op fell below it would vanish from latest()) and flush_and_wait_uploads (L0Engine::drop joins the uploader, whose mpsc loop drains before disconnect) deliberately not, reasons recorded on the doc comment.

catalog#12 — nothing could run any of it. Three library crates, zero binaries, and tick() / register_from_source had 0 non-test call sites: 89 tests passed and the catalog had never seen a real document. The new catalog binary reads a git ref, not the working tree, because a population sweep on this very repo read adiona yaml: 0 while origin/main carries 53 (the checkout was 41 commits behind on a side branch). First real run: 53/53 registered, relations=0, attributes=0. relations=0 is correct — leaf playbooks calling no child, after a first regex wrongly counted each document's own metadata.path as a child reference. attributes=0 was a gap: all 53 carry a tool kind (53x postgres) and an auth alias (49x adiona_actor, 4x adiona_migrator), and a playbook yielded nothing because find_attributes only read metadata.labels. Now uses_tool.* + uses_credential.* → attributes=106 = 53 x 2, the four migrator playbooks cross-checked against ground truth derived independently from git. ⚠⚠ Alias only, never a value: a mapping under auth: is an inline credential and is skipped — mutation-tested, the relaxed check fails with the leak in its own output (uses_credential.Mapping {"password": String("hunter2")}). 99 tests, fmt + clippy clean.

Three stale server PRs remain OPEN, not closed (#480/#473/#481): my staleness measurements were wrong. #473's premise is live (two byte-exact == "true" sites plus a third spelling accepting "enabled", no shared truthy()), #481's benchmark is the only record of its conclusion, and #480's examples/chain_cert_bench.rs is unique to it. Instead the evidence was made durable on ai-meta#367 — including a correctness hazard recorded nowhere else: rolling up by commit batch is boundary-dependent, so two replicas that batched differently produce different digests for the same chain.

Prod, both actions the owner approved. v3.123.0 rolled to the server STS by digest (091ff47f), confirmed by the binary's own report — noetl_server_build_info{version="3.123.0"} 1 — not by a note. Revision 7fd4c9bb4f → 5f6cffb8ff, gen 52 → 53, restarts=0, env 67 before and after, sibling deploy/noetl-server-rust untouched at 0/0. Rollback digest 919bc70c recorded and unused. The dead-letter reader was then checked with controls: /api/health → 200 (the probe works), the reader → 403, an unknown route → 404 — so the 403 is the gate, not a catch-all. ⚠ The first probe used busybox nc and returned empty for everything including the control; wget works and curl/python3 are absent from the image. ⚠ Before rolling I checked the chain path at hunk level, because prod has the gates armed: 5 of 23 changed files match chain|parity|populate|advance|cert and every hunk in them is inside mod tests, with an identical NOETL_CHAIN_* read-set at both tags. An earlier note of mine claimed the range touched nothing chain-related — that was asserted at file granularity and unverified when written.

#410 applied and scraping cleanly: up{job="noetl-server"}=1, 8 targets with 0 DOWN, 680 samples, and all six previously-blind families reporting at 0. No pager noise, and none was possible — the conditions were read rather than assumed: the comparator policy needs a rate ratio > 0.5 over 1800 s, which is 0 at zero rates. Server-side apply hit a .spec.selector conflict with manager kubectl-client-side-apply, so plain client-side apply was used instead of --force-conflicts.

⚠ One precondition correction. NOETL_CHAIN_CERT is unset, so the flag the instruction named is genuinely OFF. But NOETL_CHAIN_POPULATE/SOURCE/ADVANCE are set — re-armed on prod 2026-10-01 at 15:06:18Z, which my records never captured. That is the #360 configuration, currently failing safe (~7 "refusing to populate, falling through to Postgres" in 13 h, 0 divergence hits, positive controls passing). Rolling neither armed nor disarmed it; disarming is an owner call.

New repo: noetl/catalog — the generalized, event-sourced catalog of NoETL internal resources, playbooks first. Public + Apache-2.0 + main, matching all 10 production Rust submodules; branch protection identical to ehdb/server/worker (required check rust, strict, no required review, so it is self-mergeable on green). Wired in as a submodule through the pointer gate (#428), which also filled two pre-existing linkedProjects gaps — repos/ehdb and repos/signal-mesh are both Cargo workspaces rust-analyzer had never been indexing.

CI from commit one, 4 of 4 RED-proven — fmt, clippy -D warnings, tests, and a source-hygiene guard, each failed on a planted defect and the tree restored clean. No step is suffixed || true; ai-meta ran its whole life with no CI and five repos carry exactly that suppression (#374). ⚠ The hygiene guard fired correctly on its first run, before anything was committed, by refusing to compare against an empty git ls-files — a scan that finds nothing reports no violations and reads exactly like a healthy workspace.

Design spec (#427, design/catalog-model.md) — storage is EHDB and only EHDB: a write is an event appended to an EHDB dataset, a read is a fold. Three audit findings reshaped it: (1) ⭐ an event-sourced catalog already exists in shadow — catalog_log.rs writes CatalogRecord to StoreTier::Catalog, the event types catalog.registered/.archived/.restored already exist, and CatalogRelation is a pure fold nothing calls on a read path; 1,601 prod records. So this finishes a named predecessor, and ⚠ the RFCs those files cite do not exist on disk. (2) d7_catalog already exists in ehdb-l0 and D1–D10 are all taken, so the spec takes neither D7 nor D11 and defines its own c1..c4 namespace. (3) ⚠⚠ there is no Projection/Fold trait in EHDB — 23 public traits, none a fold; the pattern folds read_index_after(key,0).last(), i.e. latest-wins, which is correct for the entity log and silently wrong for attributes and relations where many entries share an index key. A naive copy keeps one and looks fine, so AC4 plants two attributes as a positive control.

The generalization in one line: adiona needs a new table per entity type (item_attributes, trip_attributes, trip_category); one polymorphic identity (resource_type, path, version) replaces that, so adding a resource type is one appended row — enforced by AC3, which asserts the Dataset impl count does not change. P1 landed (catalog#2): the four model types, 30 tests, version widened to u32 because SMALLSERIAL caps at 32,767 and catalog load re-versions every entry on each run.

⚠⚠⚠ And the audit was wrong about prod, in the direction that matters. /api/catalog/list filtered kind case-sensitively while prod stores both spellings — 'Playbook' 875 rows, 'playbook' 650, 'mcp' 2. So the filter did not return zero; it returned a partial set: "playbook" gave 650 of 1,525 playbooks (43%) and "Playbook" gave 875 (57%). Neither was empty, which is why it survived — an empty result gets reported, a 43%-complete one gets believed. Cause is two halves: registration has lowercased since f0301e86 (2026-06-09) but never backfilled, and the predicate bound the caller's string verbatim. Both are fixed, because each alone is still wrong — predicate only gives an empty result for a caller sending Playbook, parameter only still misses the 875 legacy rows. server#496 → v3.123.1, #429. RED-proven 3 of 3 (column half, parameter half, both). ⚠ Nothing tested it: resource_type appeared in 0 of 27 files under tests/, and the existing shape guard asserted page.contains("kind = $1") — it had pinned the buggy predicate as correct while passing Some("Playbook") to it. ⚠⚠ My own new guard's first draft counted its own assertion messages, because include_str! reads the whole file and the messages name the needles verbatim — it reported 4 uses where the code has 2 and failed with the fix correctly in place. It now slices at the last #[cfg(test)] via rfind, not the first via split, because this file has an earlier #[cfg(test)] with real code after it. ⚠ Also corrected: mcp has 2 prod entries, so the "never registered" conclusion was about the repo's fixtures, not about prod.

P2 landed the same session (catalog#4, 47 tests): the four Dataset impls in a c1..c4 namespace and their folds, on ehdb-l0 + ehdb-core at tag v0.4.5. ⚠⚠ The fold was the whole risk and it is RED-proven three ways — EHDB has no Projection/Fold trait, and the in-tree read_index_after(key,0).last() idiom is correct only when the index key uniquely identifies the folded thing, which here it never does. Planting each naive variant made the matching test fail: the attribute fold returned 1 of 4, the relation fold 1 of 3, the entity fold [1000000] instead of four versions. The naive version returns a plausible-looking record and does not error, so AC4's assertion is on the count. seal_max_age is set and CatalogStore::tick drives seal_aged_parts, because the flag alone is inert on exactly the shard it protects and the catalog — almost always idle — is the worst case for that. AC3 gained a storage half asserting exactly four Dataset impls: the model crate's AC3 still asserts zero there, which stays true but would now pass vacuously. Exact rather than a floor, because the failure it guards against is always an addition.

✅ v3.123.2 rolled to prod, and both fixes verified against the live endpoint. build_info{version="3.123.2"}, digest 49eb47fc, restarts=0, env 67 unchanged with the chain trio untouched. The range was 2 files, both catalog, 0 touching the chain path. #429: /api/catalog/list returns 1525 for every casing (was 650 / 875), and Mcp returns 2 (was 0) — 1525 + 2 = 1527, so the arithmetic closes. #432: a list-form bogus kind is rejected naming step 't1', the tool item's own name, which only the new sequence arm supplies; the old walker would have accepted it. Four controls: a positive (right pod), the mapping-form case naming step 's', a discrimination control (missing metadata gives a different message), and a negative (killing the port-forward made the probe return HTTP 000). No catalog rows were written — every probe was rejected.

⚠ A +33% latency reading that dissolved. A post-roll window read emit 143.4 ms against a 108.1 ms "baseline", but that baseline was a 30-min window and the adjacent 17-min pre-roll window has zero traffic, so no like-for-like comparator existed; an equal-length window earlier reads 177.0 ms, worse than post-roll. With 0 of the 2 changed files on the event-emit path, it is variance on sparse bursty traffic. ⚠ The honest limit: prod traffic here is too sparse to support a tight latency claim in either direction.

⚠⚠ And a real defect in my own store, found by reading EHDB's contract rather than by a failing test (catalog#9). next_seq was initialised to 1 in open(), so a reopened store re-emitted op_seq 1, 2, 3 below the ops already there — violating "ascending sort_key within a partition", which the sparse index, MinMax pruning and merge all rely on, without erroring. The consequence is a WRONG ANSWER: the fold takes the last op per key, so a post-reopen write with a lower key is ordered before the write it supersedes — the test read "dev" where "prod" had been written afterwards. A catalog is the worst shape for it: written rarely, restarted often, so after the first deploy almost every write is a post-reopen write. ⚠ Of the three tests, the one asking "did my writes survive" passed — two attributes with different names both survive, because the defect is wrong ORDERING, not loss. Fixed with the idiom EHDB provides (append_writer_assigned re-keys to global_sequence + 1, recovered from the manifest on open), and the hand-rolled counter deleted rather than patched.

⭐ And then subscription landed as a real second resource type (catalog#7, 83 tests), which turns the generalization claim from structural into measured. AC3 previously proved "adding a type adds no Dataset impl" with a synthetic widget; it now proves it with a live type that shares nothing structurally with a playbook — no workflow: at all, a spec: instead, its reference at spec.dispatch.playbook (9 of 9 fixtures carry one), and a credential alias at spec.auth (6 of 9). The playbooks they dispatch sit in the same fixture directory, so the cross-type graph is closed and checkable. Zero datasets added, RED-proven by planting a c5_mcp_entity — the exact regression AC3 exists to catch and the one that looks most reasonable in isolation. ⭐ RelationKind::Requires finally has a real use: spec.auth names a credential, a type the catalog deliberately does not hold, and the edge is recorded anyway because a dangling dependency is worth knowing about.

⚠ Two API shapes would each have passed a naive test. FoundReference hardcoded Invokes and "playbook", which would have made the credential dependency unreadable while still producing a plausible edge count of 2; and the attribute scalar typing puts bool before integer before string, because a string fallback placed first captures everything and makes every attribute Text — which still round-trips.

P3 landed, and surveying the YAML for it found a live server defect. catalog#5 adds catalog-extract, which reads the playbook references registration throws away: a playbook names a child by path inside a step's tool: block and register discards it, so a playbook can be registered referencing a child that does not exist and nothing says so until the step runs.

⚠⚠ Surveying tool: to write that walker found #432: the server's own collect_tool_kinds read tool: as a mapping only, so validate_tool_kinds ran, returned successfully, and validated an empty set for the sequence form. Measured across the 166 e2e fixtures carrying a tool: key — 20 sequence-only, 109 mapping-only, 37 using both in one file — the walker missed 219 of 1,178 tool kinds across 59 files; hand-verified on test_vars_block.yaml, five list-form blocks each carrying kind: python, extracted zero. Fixed in server#497 → v3.123.2, RED-proven 2 of 2, 0 genuine misses after. ⚠ The probe's raw after-figure is 3, all three my own heuristic counting kind: Playbook inside an HTTP prompt string and two Python code strings — text in string literals, not tool kinds. ⚠ Not deployed: prod runs v3.123.0.

⚠⚠ And the oracle was wrong before the code was. P3's ground-truth heuristic counted any kind: naming a playbook followed by a path:, which matches every document's own opening kind: Playbook / metadata.path — so every fixture appeared to reference itself, and the truth said 2 where the extractor correctly said 1. That is the more dangerous direction of disagreement: the tempting move is to "fix" the code until it agrees with the oracle. ⚠ The negative-control fixture also needed a second half, because "zero references" is exactly what a reader that cannot parse the list form returns — so the test asserts zero references and that the five list-form tools were visited.

⚠ Three shell traps cost time and each produced a confident wrong reading: \s is unsupported in BSD grep -E (a glob matched 1 file instead of 166), zsh does not word-split $FILES into argv (the whole list became one argument, so a probe reported files examined: 1), and cargo's absence from PATH made a clippy run print 0 warnings that was really exit 127.

⚠ cargo fmt --all remains unusable on noetl/server — 138 files are unformatted on main and test.yml excludes a fmt gate deliberately (these crates were formatted with rustfmt 1.9.0). One unrelated collapse rustfmt 1.99.0 introduced was reverted so the diff is only the fix.

Last item of the window, and it came from distrusting my own verification.

Re-running the four checks I had fixed today, two came back DRIFT:

schema-copies   DRIFT  schema_ddl.sql differs between noetl/noetl and repos/server
pinned-sets     DRIFT  5 pinned set(s) are unverifiable        (I had registered 3)

Both looked like regressions I had caused. Neither was:

• repos/noetl's working tree was behind its pointer, so its DDL lacked the table I had ported;
• ai-meta itself was on a feature branch, and although I ran the script from origin/main it loads playbooks/lib/pinned_sets.py from $ROOT — so the branch's copy ran.

Verified clean at the recorded pointers instead: the two DDL bodies are byte-identical under the audit's own comparison, and origin/main's pinned_sets.py carries all three registrations.

Measured cause: 22 of 33 submodule trees in the primary checkout differ from the pointer ai-meta records. So the audit has been describing a tree that diverges in 22 places, silently, for as long as it has existed. #425 makes it the first thing printed and repeats it in the summary — WARN rather than DRIFT, because a dirty dev checkout is normal and counting it would make the finding total useless; unmissable by position instead, since being skimmed past is the failure mode.

• git rev-parse HEAD in an uninitialised submodule directory returns ai-meta's HEAD, because the empty directory is still inside the parent repo. The first version reported 31 of 31 trees differing — every one a phantom whose tree sha was ai-meta's own.
• With nothing readable, "every tree matches" is true and useless. A separate arm now says NO submodule tree is readable (31 of 31), which also explains the 0 first-party .rs of 0 the checks below print. Same vacuous pass a linked worktree produced earlier today.

Four arms proven: nothing-readable → WARN; feature branch + 22 of 33 → WARN; synthetic all-match → OK; synthetic one-moved → WARN with correct shas. ⚠ That last control did not fire on its first attempt — the submodule clone had not fetched the commit, so git checkout silently did nothing. A mutation that was not applied reads exactly like a check that did not fire.

Measured before saying it: 0 currently-failing workflows across all 16 default branches by latest-completed-run-per-workflow; v3.123.0 released 7 of 7 jobs with amd64 + arm64 + manifest; env-docs OK, inert-tests OK over 566 first-party files, schema-copies identical at the pointers, pinned-sets registered on main; wiki sweep green over 15 wikis.

⚠ One caveat on that zero: the per-workflow sweep lists dead Dependabot update-ids as failing for ever, because each update gets its own workflow name and its last run stays that failure. noetl/noetl's four are all from before dependabot.yml was removed.

Everything still open is owner-gated — #410 (pages on first scrape), rolling v3.123.0, #422's gauge (invisible until #410) and its replay design, #406 (accepted wait; 3 is the expected RustSec reading), #234 (do not sweep), #235, #380.

Built the safe half of #422. noetl.event_dead_letter is no longer write-only: server#495, released v3.123.0 with amd64 + arm64 images — so the Dockerfile fix from earlier holds.

GET /api/internal/events/dead-letter?execution_id=&include_resolved=&limit=1..1000
→ { outstanding, returned, limit, rows }

Payload scrubbing — sidestepped, not solved. The query does not read the payload column. A parked row carries the raw event payload, and an event payload can carry resolved credential values; wiring a reader through scrub::scrub_in_place and trusting its coverage is a bet, whereas not selecting the field is not. Enforced by a guard asserting the column constant and the struct, since adding the field to only one would still change what reaches a client.

The cost is deliberate and written into the code: enough to triage, not enough to replay. That keeps the scrub decision inside the replay work, where it belongs.

Auth is the writer's, unchanged — RequireInternalApiToken plus the group's auth_gate::gate layer — because the endpoint enumerates execution ids. Compare #303, where an enumerating endpoint answered 200 with decrypted data and no Authorization header.

outstanding is separate from rows because a list truncated by limit tells you what you can see, not how much there is, and the failure this table exists for is a backlog nobody noticed. It is also the first query ever to use idx_event_dead_letter_outstanding.

tests/auth_gate_wiring.rs searches main.rs for the literal .merge(<name>.layer(. rustfmt splits those merges across lines, so after one cargo fmt the guard reported 12 privileged routers merged without the auth gate — credential_routes, sealed_credential_routes, cross_region_routes and nine more.

Reverted. main.rs now carries an 11-line diff that is only the route, with a comment above it saying not to reformat the file and why. The guard did exactly its job. ⚠ This is the second time rustfmt has blinded a text-scanning guard in that crate, which is an argument about where such guards should live, not just a note.

Fmt accounting, because "I ran fmt" is otherwise unfalsifiable: 139 sites on origin/main → 138; main.rs unchanged at 7, event_dead_letter.rs 1 → 0.

A /metrics gauge — outstanding is in the response, so you can ask, but nothing publishes it, so nothing can alert on it; and it would be invisible until #410's apply lands anyway. And replay or discard, which is where the payload question returns and which needs a design rather than an endpoint.

Follow-through on the three merged noetl/noetl PRs, then the downstream question for #709.

#400 — the publish guard is on main (e18e50d2), verified there rather than on the branch: verify_release.py present, publish.needs = [check, verify-release], and its --self-test printed controls: 10 firing as specified: 10 wrong: 0 on the PR. ai-meta pointer bumped to e18e50d2 (#421).

#708 and #709 noted on their tracking threads — #383 for the dependabot config whose only directory left the repo in March, #201 for the DDL copies agreeing again.

Measured rather than assumed, and the answer has two halves.

It already exists in prod's database, created 2026-10-01T15:06Z. That claim needed care, because ensure_table swallows every error and returns Ok(()) — "Never fails startup" — and the code explicitly anticipates the server role lacking CREATE. So a Ready pod proves nothing. The evidence is the absence of its "dead-letter DDL skipped" warning in the startup window, with a filter control showing the search works there:

textPayload:"dead-letter"  15:06:00..15:08:30Z   -> 0 hits
textPayload:"chain"        same window (control)  -> hits

⚠ And a correction that fell out of it: prod's server is v3.122.0 (digest 919bc70c), not the v3.112.7 my notes and this dashboard carried for weeks. ensure_table first shipped in v3.115.0, so prod has it — but I had been reasoning from a version three weeks stale.

The table is write-only. The only SELECT against it reads back the rows it just inserted, to decide whether the caller may ack — durability confirmation, not a consumer. No GET, no CLI (doctor has 0 references), no ops query, no backlog gauge. The worker counters count parking attempts by a process and reset on restart: a rate with no total.

That matters because parking acks the poison, removing it from the durable log — so the row becomes the only remaining copy. Harmless today (NOETL_MATERIALIZER_DEAD_LETTER is off everywhere, nothing is parked); the moment it is armed we would be deleting events into a table nobody can read.

#422 tracks it, with the two design constraints that make it more than a SELECT: parked payloads are raw event payloads and must go through the response-boundary scrub, and a reader enumerating execution ids needs at least the writer's internal-token gate. Sequencing recorded as build the reader before arming the flag, not after.

Picked up #415 (three unguarded pinned label sets). The guards were the easy part; what releasing them uncovered was not.

Three guards in noetl/server (#493), each a total match so a new variant fails the build, each asserting both directions — every label pinned, and nothing pinned that no caller can produce.

The reverse assertion found a live bug on its first run. EMBEDDED_READ_OUTCOMES pins four values; ReadVerdict::label() produces three. The fourth, engine_unavailable, existed in the const and in a JSON response body — and the handler returned early without touching the metric. So that series was registered, pinned at 0, and could never increment, telling anyone who read it that the embedded engine was always available.

A pinned series nothing can increment is worse than an absent one: absence at least prompts the question. NO-GUARD went 3 → 0 (#418).

My commit body carried Closes noetl/ai-meta#415. @semantic-release/github resolved it against noetl/server, failed, and did so after creating tag v3.122.2 and before the step that dispatches release.yml. That version was tagged, had a GitHub Release, and no image — and nothing reported it.

⭐ The sharp part: GitHub itself resolved the same line correctly and closed ai-meta#415 that same minute. Two consumers, one keyword, opposite outcomes.

⚠ It arms only on a commit type that releases — 1 failure in 60 runs, and the only other commit of that shape was a ci: one that released nothing and went green.

Dispatching release.yml by hand to recover v3.122.2 is what surfaced it:

#14 rustup target add wasm32-unknown-unknown  -> rust-std for wasm32 (under 1.91.1)
#18 cargo build --target wasm32-...           -> syncing channel updates for '1.99.0-...'
                                                 error[E0463]: can't find crate for `core`

rustup target add ran before COPY . ., so the target was installed for the base image's 1.91.1 — then rust-toolchain.toml (1.99.0, which I added on 10-03 in the clippy sweep) took over, and 1.99.0 has no wasm32 std.

Invisible for two days because every commit between was ci:/chore:, which release nothing — so release.yml never ran. The first fix: commit was the one that exercised it, and the image build failed on a change touching three test files. The same shape as an inert gate: broken, and the only thing that would report it does not run on the commit types we had been landing.

Fixed in server#494, verified locally with podman both ways — old → exit 101 E0463; new → exit 0 with the 1,533,325-byte .wasm present. v3.122.3 then built amd64 + arm64 + manifest + github-release; AR carries latest,v3.122.3.

• A mutation battery where no mutant was applied reported 5-of-5 FIRES. The helper never forwarded its arguments; the non-zero exits came from E0308 after a sed changed an array's length but not its contents. Two later mutants had the same shape in reverse. A mutant that fails to compile proves nothing.
• A red baseline from a usage error. cargo test --lib a b c takes one positional filter, so three names is a usage error — which reads as a failing baseline and makes every mutant look caught.
• $MINE did not word-split in zsh, so a loop meant to revert everything except my three files reverted those too; the git stash/pop that followed popped a pre-existing stash from another session and conflicted. Recovered with that stash intact; work re-applied.
• cargo fmt --all reformatted 39 files. main already carries 139 fmt diff sites and CI does not gate fmt. Reverted; the PR quotes 139 before and after.

Continued autonomously through what the gates surface. Three findings, all about instruments rather than code.

findings in that section      : 151
inside repos/server/vendor/   : 150
ours                          :   1   <- also a false positive

The one first-party hit was certified_fold.rs:755:produce_again — text inside a synthetic Rust source string literal, embedded by a real #[test] whose entire purpose is to demonstrate that naive source scanning gets fooled. The check was fooled by the test about being fooled.

Four defects, fixed in #413: no vendor exclusion (server has 11,990 .rs files, 192 first-party); "assert" in L[i:i+40] asked about a 40-line window so a helper reports as asserting when the asserts belong to the next function; the attribute regex missed #[crate::test]; and string-literal lines were treated as code. A fifth came from my own control refuting my own claim — the assert test matched a bare substring, so a fixture named orphan_with_no_**assert**s_of_its_own satisfied it on its own name.

After: 566 first-party files examined, 0 findings, with the battery showing the genuine orphan still reports in both old and new. That row is the control that matters — widening a check to admit legitimate cases is the easiest way to silently delete it.

⚠ And the denominator earned itself immediately: the first run of the fixed check printed 0 first-party .rs of 0 for all six repos and then OK, because I ran it from a linked git worktree where submodules are uninitialised. It examined nothing and said clean.

Audit output overall: 160 DRIFT lines → 9.

19 env vars (7 server, 12 worker) were read by the binaries and documented nowhere — Rule 2a. Now #414, all three components OK.

The first changes what a run does, not what it reports: with it on, a playbook relying on an erroring step to continue begins failing. Default off, armed everywhere, undocumented. The second is an armed background writer whose every metric comes from the unscraped server — so its counters are unknown, not zero, and the row says so.

Reading defaults from source rather than guessing surfaced that the 0-value semantics differ between neighbours: MIRROR_REPAIR_INTERVAL_SECS discards 0, CHAIN_INVARIANT_SAMPLE_SECS treats 0 as an off switch, TIER_SEAL_MAX_BYTES treats 0 as never-seal. Reading one off a manifest and assuming the others is how a knob gets set wrong.

All five pass a variable or method call to their recorder, so registering any would report SHORT 0-of-N. Two have real guards and are now exclusions naming them. The other three — PROJECTION_SERVE_REFUSALS, EMBEDDED_READ_OUTCOMES, EMBEDDED_SHADOW_OUTCOMES — have no guard anywhere, so they stay reported. Excluding them would turn a visible gap into an invisible one, which is the defect the file exists to catch. #415, #416.

Autonomous pass over what the gates actually surface. The headline is not CI: it is that the pod serving production has had no metrics collected since 2026-09-09, and the check that says so has been printing it.

drift-audit.sh scrape reports "pod noetl-server-rust-embedded-0 is selected by NO applied PodMonitoring". podmonitoring-noetl.yaml selected app: noetl-server-rust — the Deployment, 0/0 since the serving workload became StatefulSet/noetl-server-rust-embedded.

Proven rather than inferred, with a positive control:

-l app=noetl-server-rust   ->  0 pods
-l component=server        ->  noetl-server-rust-embedded-0, and only it

noetl_worker_build_info   17 series / 7405 points   <- control: scraping works
noetl_server_build_info    0 series /    0 points   <- the subject

*_build_info is pinned unconditionally at 1, so its absence is what separates a scrape gap from a metric that never fired. That discriminator is the reason this is a 25-day outage and not an "is the metric implemented?" question.

Six of the ten metric families the 19 prod alertPolicies reference have zero data over 6h — every one server-emitted: crossstore parity / divergence / control, eventlog mirror queue / pending, command publish failed. Those are the signals built for #320, #325 and #360. They cannot fire.

Fixed in ops#321 with component: server, which both the StatefulSet's and the dormant Deployment's templates carry — a role label, not an app name, so a variant swap needs no edit. The prod apply is held: resuming a dark scrape feeds conditions that have had nothing to evaluate, so it can page on the first scrape, and picking the hour is the owner's call.

Per apply-safety.md the diff was taken over the whole file, not the field being changed. The workers object came back stale in the losing direction — live already had worker-pool Exists while the manifest still carried the enumerated app In [...] list:

worker-pool Exists   11 pods   (live)
app In [4 names]      9 pods   (manifest)
dropped               noetl-worker-system-pool-shard1-*  x2

Two of the four names selected nothing at all. Applying the file to fix the server would have removed both shard1 pods from monitoring — #238 for the third time, and the 2026-09-08 incident shape exactly: verify one field, apply the whole object, break something adjacent. Aligned before merge.

• "0 of 19 alert policies affected." The filter matched metric.type="prometheus.googleapis.com/…" and all 19 policies use PromQL conditions, so it matched nothing and printed a clean result. Caught by printing the denominator: 0 distinct metric.type references across 19 policies is not a finding, it is a broken scan.
• "Three repos have a red main." I counted bad runs anywhere in a 6-run window instead of the latest run per workflow; docs, doctor and travel were all superseded by later green runs. Then the corrected version flagged gateway, because a tag-triggered run reports the tag as its branch — the latest main run was a stale August failure while v3.12.3 built fine.
• A pytest RED proof that ran no tests. The interpreter had no pytest and the exit code came from the surrounding pipeline, so it printed 0 — a pass.

• #400: the publish guard, RED-proved against live PyPI — noetl#707, green, awaiting review.
• noetl.event_dead_letter ported into the DDL the deploy applies, with a contract test and three controls — noetl#709, green, awaiting review.
• The dependabot config whose only directory left the repo in March — failing weekly for seven months with dependency_file_not_found — noetl#708, green, awaiting review.
• #360 acceptance box 6 un-ticked: it claimed a criterion met on a ~10-minute window against a mechanism that fires hourly. The measurement is kept; the tick is not.
• #234 annotated do-not-sweep: the travel Maps secret exists in noetl-demo-19700101 and is NOT_FOUND in shastaratech-noetl-prod, so repointing breaks the deploy. A grep for a project ID cannot tell a comment from a credential lookup.
• STATUS.md added — there was no present-tense answer to "what is blocked on me right now".

#400: noetl#707, CI green on 14385af6 (3 of 3: ci, pypi, forbid-client-term), REVIEW_REQUIRED — batched for the owner, since noetl/noetl enforces review on admins. #400 stays open until it merges.

Two mistakes produce a green release that ships nothing, and only one involves a mismatch:

• pypi-v5.1.2 cut while packaging/pypi/ still says 5.1.1 → builds 5.1.1. Tag and artifact disagree; a string comparison catches it.
• pypi-v5.1.1 cut again → builds 5.1.1, every file skipped, zero bytes reach PyPI, job green. Tag and artifact agree perfectly. No comparison of the two can see it.

So the guard queries PyPI. The only thing distinguishing releasing 5.1.1 from re-releasing it is whether PyPI already holds those files — which is state, not a string.

verify_release.py gates publish and the disabled publish-oidc: one version in dist/ with floors (≥1 wheel, ≥1 sdist — an empty dist/ is what a broken download-artifact leaves), pyproject.toml == Cargo.toml == built filenames, tag == that version, and not every file already published. A partial overlap is allowed, so a retry after a half-finished upload completes — that is what --skip-existing is legitimately for.

noetl 5.1.1 is genuinely published with 4 files, so the silent-skip case is reproducible for real — 342 versions known:

RED   pypi-v5.1.1 re-tag            exit 1  "every one of the 4 file(s) for 5.1.1 is already on PyPI"
RED   tag 5.1.2, tree+dist 5.1.1    exit 1  "the tag says 5.1.2 but the build produced 5.1.1"
GREEN genuine new 5.1.2             exit 0
GREEN PR mode, no tag               exit 0
GREEN project with zero releases    exit 0  <- --require-new is not just always-fail

⚠ Confirming each RED fired for the right reason earned itself immediately: my first run of the tag-ahead case exited 1 because a shell variable was unset and --dist /dist did not exist. A control firing for an unrelated cause reads exactly like a control working.

Plus --self-test: 10 controls, 7 must fail and 3 must pass, run on every PR and tag. In CI: controls: 10 firing as specified: 10 wrong: 0.

The upload-count regex. It anchored on ^Uploading. Against the actual 5.1.1 publish log it matched 1 of 4 — twine writes a progress bar, and three of the four upload lines begin with an ESC[?25h cursor-show sequence. Had twine emitted that before the first line too, it would have read 0 and failed a good release. Unanchored with ANSI stripped now, verified to read 4. I only looked because I had the real log from this morning's publish; reasoning about the format would have shipped the bug.

The success line claimed more than it checked. The first CI run printed, on a pull request, RELEASE OK — the artifact matches the tag and has something new to upload. There was no tag and no PyPI lookup had happened. That is #400's own failure mode one level up: a confident statement nothing established. It now enumerates what ran and states what did not.

Open: #400 (awaiting the owner's merge) and #406 (the accepted upstream wait). Closed across the program: #361, #374, #375, #378, #385, #390, #395, #398, #402.

Owner direction: rsa/Marvin is explicitly not a blocker, and the same for the two quick-xml DoS advisories. Waiting for upstream is the accepted disposition, so they do not hold a release or an issue open.

#402 closed. Its goal was the patchable half of the oracle gap, and that is closed on the published artifact. The three remaining advisories moved to #406 — a tracked watch rather than a line in a closed issue, because an accepted wait still has to be visible when the upstream release lands.

Two things that split decided this shape rather than simply closing #402 with a note:

• issue-tracking.md treats work blocked on external action as something that belongs in an open issue. The wait qualifies; the blocker does not.
• The two quick-xml advisories are not actually waiting on upstream. The patch exists at >= 0.41.0; what is missing is our decision about which crypto backend ships in the wheel, because object_store 0.14 pulls aws-lc-rs in beside ring and leaves rustls with no default CryptoProvider. Filing that as "waiting for upstream" would have been wrong, so #406 states the two causes separately.

5.1.0 shipped lock   5 vulnerabilities    <- positive control
5.1.1 shipped lock   3                    <- all three accepted waits
informational        10 -> 9
denominator          1270 advisories parsed (0 unparsed), 574 packages
Dependabot, same lock: 290 alerts, all fixed, 0 open

The previous entry recorded that slicing Home's table at the next ## heading swallows Recently closed, and said to bound it at ### Recently closed instead. Writing that sentence put the literal string in backticks, mid-line, above the real heading — so the next str.index("### Recently closed") matched my own note and truncated the slice from 59,491 bytes to 1,485. The table then read as containing no rows at all, and the #402 row I was looking for "did not exist".

The reconciliation numbers were unaffected (they were measured before the note was added), but the measuring script was broken by its own documentation. Fixed by anchoring on a line-start regex rather than a substring.

Two general forms worth keeping:

• A substring search for a heading is not a heading search. Anchor on line start, or parse the structure.
• Prose about a pattern is itself an instance of that pattern as far as any text search is concerned. Documenting a grep trap in the file the grep runs over will reproduce it.

Open from this arc: #400 (the publish guard — --skip-existing plus no tag/version cross-check; real work, not started) and #406 (the accepted wait). Everything else from the CI/debt program is closed: #361, #374, #375, #378, #385, #390, #395, #398, #402.

Three issues closed and one narrowed, all by driving each to a published, measured end state rather than leaving a decision open.

noetl 5.1.1 is on PyPI (noetl#706). h2 0.4.15→0.4.19 (RUSTSEC-2026-0258) and rustls 0.23.42→0.23.45 (RUSTSEC-2026-0285) clear; event-listener 5.4.2 and rustls-webpki 0.103.15 came with the re-lock. ai-meta pointer bumped in #404.

The verification that mattered was not the bump but the control. Reporting "3 vulnerabilities remain" from a scanner that has never produced a positive on this input is indistinguishable from reporting 3 from a scanner that is broken — the same shape as every inert gate in this log. So the previous shipped lock (5.1.0) was scanned first and had to report exactly the five this issue opened on. It did:

5.1.0 shipped lock   5   h2, quick-xml x2, rsa, rustls      <- positive control
5.1.1 shipped lock   3   quick-xml x2, rsa
denominator          1270 advisories parsed (0 unparsed), 574 packages in the lock

⚠ The scan nearly produced its own false clean twice. The advisory files are .md with TOML front matter, not .toml, so a glob for crates/*.toml matches nothing and prints a clean repo — caught only because the script asserts parsed > 1000 before reporting. And my own display of rustls-webpki's advisories used grep -m1 patched with a trailing sed, which showed 0.104.0-alpha.6 as the only patched version and made it look like the scanner was missing three hits; the real field is [">= 0.103.12, < 0.104.0-alpha.1", ">= 0.104.0-alpha.6"], which 0.103.15 satisfies. The truncation was in the instrument I built to check the instrument.

The two deferrals are why #402 stays open. quick-xml's patch is >= 0.41.0, reachable only through object_store 0.14, which pulls aws-lc-rs in beside ring and leaves rustls with two registered CryptoProviders and no default — five TLS tests fail at runtime. That is a choice about which crypto implementation ships in the wheel, not a version bump. rsa/Marvin has patched = [].

And the gap this issue exists for, restated with both denominators: on the same lock, Dependabot 290 alerts all fixed, 0 open; RustSec 3.

Uploading       5    (the "Uploading distributions to" line + 4 files)
Skipping        0
already exist   0

PyPI latest 5.1.0 → 5.1.1, with 5.1.1 confirmed absent before the tag. All four files downloaded and sha256-matched to PyPI's own digests; the sdist Cargo.lock is byte-identical to packaging/pypi/Cargo.lock on the bumped commit, so unlike #395 there was no repo-vs-shipped gap to find.

The issue left a product decision open: index the stale arm64 debs, or drop them. Neither — build a current one. ubuntu-24.04-arm runners were already available, so build-deb became a two-entry matrix rather than a cross-compile, and newest arm64 moved 2.8.7 → 5.0.3 (cli#93). Nothing was deleted; the old debs stay installable as the older releases they genuinely are, which is what --multiversion is for.

Measured on the live published repository after v5.0.3 released through the new pipeline:

.deb in no index        22 of 23  ->  0 of 26
architectures indexed   [amd64]   ->  [amd64, arm64]
Release header fields    1 (Date) ->  8
Release self-entry      present   ->  absent, both suites

⚠⚠ The finding nobody had reported. Without --arch, dpkg-scanpackages writes every .deb into whichever index it is told to produce — architecture is simply ignored. Harmless while only amd64 shipped, and it would have started serving arm64 binaries to amd64 clients the moment arm64 existed. All three findings promoted REPORT→FAIL (apt#3) with a fourth for arch leakage, each injected one at a time against the real tree to prove it fires — a promoted check that cannot fail is just a comment, which is what the REPORT tier already was.

One deliberate omission: Version is not required in the Release header set. It is optional in Debian and the generator does not emit it; Suite and Codename are what apt matches on. Requiring a field the correct output lacks is how a gate earns a permanent || true.

All 7 Rust repos now pin 1.99.0 (gateway#54, cli#94). The measured cost was 2 lints: cli's needless_borrow (one character) and gateway's result_large_err, kept as a function-scoped #[allow] with the reason written down, because axum's middleware signature is Result<Response, Response> and the error variant is a whole Response by construction.

⚠⚠ The pin also found gateway would not compile on 1.99.0:

error[E0464]: multiple candidates for `rlib` dependency `base64` found

base64-compat = "1.0.0" declares [lib] name = "base64", colliding with the real crate. Referenced nowhere in gateway source — dead long enough for the collision to go unnoticed. I first called this a local artifact precisely because it reproduced on both toolchains locally; CI was the instrument that proved the location real. A failure that reproduces everywhere reads like environment noise, which is the inverse of the usual tell.

#395 closed. noetl 5.1.0 is on PyPI — https://pypi.org/project/noetl/5.1.0/ — so the artifact users install finally carries the crates the repo cleared back on 2026-10-03.

Arc completed: 214 → 31 → 7 → 0 in the repo → 0 in the artifact.

Minor: above the published 5.0.0; not a patch because the embedded CLI moved 4.21.0 → 5.0.1; not major because the package's own API (run, cli, main, __version__, the noetl/ntl entry points) is unchanged. Deliberately not aligned to the CLI version — packaging/pypi/README.md documents that the two are separate.

⚠ There are two pyproject.toml files. The root one (setuptools, 4.26.1) is the retired Python distribution's version carrier that semantic-release bumps; the wheel is built by maturin from packaging/pypi/pyproject.toml, which read 5.0.0. My earlier claim that the version line was broken — that pyproject sat below the published 5.0.0 — compared the wrong file.

twine upload --non-interactive --skip-existing dist/*

The published version comes from the packaging/pypi/ files, not the tag — and nothing bumps them automatically, because semantic-release's prepareCmd rewrites the root pyproject and packaging/pypi/** is outside its assets. So tagging without bumping builds the already-published version, --skip-existing skips everything, and the job goes green. || true in PyPI form, and nothing cross-checks the tag against the built version. #400.

So the publish was verified by its upload/skip counts: 4 × Uploading noetl-5.1.0-*, Skipping 0, already exist 0, with PyPI's own state captured before tagging (5.1.0 absent) and after (4 files, none yanked).

Files downloaded from PyPI, sha256 matched to PyPI's digests: the published sdist's Cargo.lock (524 packages, all four patched, none in a vulnerable range, with a control proving the range logic flags 0.102.8), strings on the published wheel binary, and the publish-run compile log — which is what closed the jsonwebtoken gap, since it left no string in the stripped binary.

⚠ pypi.yml builds without --locked, so a wheel's resolution need not equal the committed lock. Demonstrated: the 5.0.0 run compiled both rustls-webpki 0.102.8 and 0.103.13. Which is how a strings probe briefly convinced me the old artifact was fine — finding a patched copy is not evidence that a vulnerable one is missing.

The sharper finding, and it came out only because the release verification used a second tool. A full RustSec scan of the same shipped lock Dependabot reports as 0 finds 5 patchable vulnerabilities in 4 crates: event-listener (memory-corruption), h2 (DoS), rustls (crypto-failure), quick-xml ×2 (DoS) — plus rsa (Marvin, no patch) and 8 informational unmaintained notices.

Two oracles, same input, different answers (#402). #395's closure is therefore scoped to the seven alerts it tracked, and says so.

That is the issue's own lesson recurring one level out: the fix was verified against the oracle that raised the alarm, and a different oracle sees more.

Completes #378's pin work. All 7 Rust repos now carry a rust-toolchain.toml.

ehdb and signal-mesh were the last two unpinned repos, and both gate with -D warnings and no allow-list — neither had anything to absorb a new lint. Pinning them turned up a live case:

So the next push to ehdb would have failed the gate, with nothing regressed and no commit responsible. Found by pinning, not by the build — the failure was simply waiting for someone to trigger it.

The fix was deliberately not try_update. That method does not exist before 1.99, and the workspace declares rust-version = "1.82" — adopting it would raise a declared MSRV by 17 releases to silence a pure rename with identical semantics and no behaviour change. That is a policy call about what the crate promises consumers, so it was left alone. A scoped #[allow(deprecated)] carries the reason instead, placed on the expression rather than the crate so a different deprecated call still fails. Proven load-bearing: removing it reproduces the exact error.

Both ehdb and signal-mesh gate cargo fmt --all --check, and the pin also fixes which rustfmt runs — 1.99.0 ships rustfmt 1.10.0. Both pass --check under it, verified before the pins landed. A pin that fixes the lint floor and breaks the fmt gate is not an improvement, and this fleet already has a note that rustfmt version drift reformats large parts of a tree.

#378: fleet clippy -A entries 23 → 2. server 14→0 (#492), worker 7→0 (#347), tools pinned (#111), pointers bumped in #396.

⚠ The count was previously reported here as 27. It was 23 — that grep matched -A below, -A entries and -A is from comment prose rather than the command.

Two tests carried no #[test] and had never run: labels_are_a_closed_set and insert_column_and_bind_counts_agree. Both now armed, both pass, #361 closed.

The mechanism in both files: a stray #[test] sat above a doc block with a second #[test] below it, so both attributes bound to the earlier function — reported as duplicate_macro_attributes, also allow-listed — while the next function was left unmarked and inert. The two entries that hid it sat adjacent in the list.

A second gap came out with it: unused_variables covered high, a member of a deliberate up(500 ns)/down(499)/nearly(999)/high(798) family in which every other member is asserted. The lint's own suggested fix — rename to _high — would have preserved the gap and marked it intentional.

The lesson is sharper than "pay down debt": an allow-list hides the next finding of the same kind, not just the ones it was created for.

Mechanically: remove it and see whether the gate fails. Each surviving allow was removed individually — all produced exit 101; all restored, exit 0.

Three classes survived, each scoped with its reason: dead_code on two worker statics that are read only from a #[cfg(not(test))] arm (so --all-targets, which compiles the test cfg, sees them as dead while production uses them); non_snake_case on three server test names whose capital is the verdict; and cli's large_enum_variant — where the documented claim "boxing changes the published DSL schema" was then tested and confirmed, because boxing ToolSpecDoc::One makes schema_guard fail despite Box<T> being transparent to serde.

In an issue about inert gates:

1. #[allow(clippy::assertions_on_constants)] placed directly on an assert! invocation is silently ignored — rustc says so explicitly. The lint still fired while my comment claimed a scoped suppression that did not exist. Attributes belong on items.
2. The first load-bearing battery reported 2 of 4 as decorative. Wrong — the shell split cases on :, which also appears inside #[allow(clippy::...)], so the wrong text was removed and the label printed empty.

• const { assert!(..) } fails cargo build, not cargo check. Check is metadata-only and skips const evaluation of function-body inline consts — a first mutation test of this read as a pass.
• cargo test <bare_fn_name> matches nothing and still exits 0 (0 passed; 1178 filtered out). My first verification of the two armed tests ran nothing and looked green.
• The toolchain action's banner is not the toolchain cargo used. It prints what it installed; the pin still wins. Reading the banner briefly convinced me pinning did not work, and that claim had to be retracted — the lint's own URL (rust-clippy/rust-1.91.0 on pinned cli vs rust-1.99.0 on unpinned tools) is authoritative.

#385 closed — open alerts on noetl/noetl are 0, control-verified (state=fixed returns 290). Arc 214 → 31 → 7 → 0.

⚠ The published wheel still embeds all seven: pypi.yml publishes only on a pypi-v* tag and none was cut, so the artifact users pip install is noetl 5.0.0 from 2026-07-26 with the 4.21 CLI and every vulnerable crate. Dependabot reads the lockfile in the repo; users install the artifact. Opened as #395, with the version line left for the owner because pyproject on main (4.26.1) sits below the published 5.0.0.

#375 CLOSED. Every repo that can host a PR gate now has one, and the 15 wikis — which cannot — are covered by a scheduled ai-meta sweep (#392).

wikis declared:  15      markdown files:  243
wikis checked:   15      relative links:  1684
not cloneable:    0      RESULT: pass

It started at 3 findings. Two genuinely broken links that nothing had ever checked, because *.wiki.git has no runner: runtime/topology in noetl-wiki (resolved to a nonexistent path — GitHub wikis address pages by slug regardless of directory) and worker/src/ehdb/guard.rs in ehdb-wiki (a relative path resolving inside the wiki; source links are absolute GitHub URLs). Both fixed upstream so the sweep starts green.

The sharpest instance of this entire arc's theme, and it was mine.

trap 'cleanup; on_exit' EXIT replaced the earlier trap, so rc=$? inside on_exit captured the status of cleanup — always 0 — instead of the script's. Two controls printed RESULT: FAIL while exiting 0: a green job announcing its own failure, because CI reads the exit code and not the text.

Caught only because the controls asserted the exit code rather than grepping the output. Had they grepped for "FAIL" they would have passed, and a permanently-green sweep would have shipped looking like thorough work — the exact defect #374 was about, reintroduced by the person fixing it.

The general form: when testing a gate, assert the thing CI actually consumes.

The sweep checks the live wiki, not the submodule pointer (a pointer can be months behind; the question is whether a wiki is broken now), clones over anonymous HTTPS (declared URLs are SSH, a runner has no key, the wikis are public), derives its list from .gitmodules rather than enumerating, and fails below a population floor. Bare page names stay exempt because wiki links are slugs — flagging them would make it red on arrival.

Continues the entry below. #375's three gateless repos now gate, all merged: ops (#320), e2e (#93), apt (#2).

Each was measured green before being made gating, each prints its denominator, and each fails below a population floor — because 0 failures is also what a check that examined nothing looks like. Every floor was proven to fire by running the checker against an empty repo.

All 36 are Helm templates: Go template syntax is not valid YAML by design. Shipping that would be a gate crying wolf on arrival, which is how a check gets switched off. helm lint covers them instead, and a control proves it is not a blind spot — breaking a template fails charts while yaml stays clean. Templates are found by a sibling Chart.yaml, never a path pattern.

docs/playbooks/** is generated from fixtures/playbooks and regenerated by npm run build, so the published site was always right while the committed copies rotted: the committed index claimed 144 playbooks against 162, and seven category pages had never been committed at all. The check is git status --porcelain docs/ rather than git diff, because most of that drift was files never added.

Also: onBrokenLinks was 'warn' — a build could not fail on a broken link, in a docs repo. Measured at 0, flipped to 'throw'.

New issue #390:

• 22 of 23 .debs are listed in no index — dpkg-scanpackages without --multiversion.
• arm64 is in pool/ but indexed nowhere — binary-amd64 is hardcoded.
• Release carries none of its 8 identifying fields and lists itself at 38 bytes against a real 1792.

And the finding behind them: ops/scripts/publish_apt.sh already discovers architectures, uses --multiversion and writes a full Release header — and nothing invokes it. The release pipeline uses its own inline commands. "Exists but is not reached", applied to a script rather than a code path.

Reported rather than failed, because the arm64 call is a human one: newest arm64 is 2.8.7 against amd64's 5.0.1, so indexing it would make arm64 look supported years behind.

The e2e gate's first CI run failed on my own workflow — ModuleNotFoundError: No module named 'yaml', because the docs job set up Python without PyYAML while the files job did. Locally yaml was already importable.

Separately: my CI-wait helper reported a failure that belonged to the previous head. Right after a push, gh pr view can still serve the old SHA, so the loop matched the old completed run and exited on a stale verdict. It now requires the head to be stable across two reads before trusting any result. The run on the real head was green all along.

Picks up the CI arc from the entry below. Two tracked issues closed; the headline is that every number here was checked against a control, because several of the obvious measurements were wrong.

The wrapper has exactly two direct dependencies (noetl, pyo3), so all seven were transitive and each needed a breaking bump one or two repos up.

Re-locked in noetl#704. Arc across the whole effort: 214 → 31 → 7 → 0. The 24 npm alerts were already gone with the e2e suite removed in #701.

⚠⚠ jsonwebtoken 10 broke at runtime, not compile time. It requires an explicit CryptoProvider; the code compiled clean and failed only when exercised. One existing test caught it — a green cargo build would have shipped it. Fixed with features = ["rust_crypto"].

⚠ lru is still in the tree and that is correct. ratatui 0.30 does not drop it; ratatui-core 0.1.2 still depends on it. The advisory clears on version, not removal — so "did lru go away?" answers no, and that is not a failed fix.

⚠ ratatui 0.30 exposed a duplicated crossterm — ratatui on 0.29 while cli declared 0.27 directly, i.e. two independent copies of raw-mode/alternate-screen state driving the same tty. It compiled and tested fine because the types never cross. Collapsed to one version.

Every RED proof was done in Actions, not locally — a gate seen only green is indistinguishable from one that cannot fail, which is the defect being fixed.

• extract_auth0_tenant_from_domain was dead — 7 passing assertions, zero callers. First read was that the Auth0 region fix had been left unreachable; it had not. This was the superseded region-dropping wrapper, and the live path is correctly wired. But deleting it naively would have cost real coverage: one of its tests was the only check of the domain-side trim/lowercase, so it was re-pointed at the live function rather than removed.
• Two stale doc comments had drifted onto the wrong items — one described a function deleted in R-1.1 PR-2c-3 and had landed on resolve_duckdb_path.
• if_same_then_else was a dead branch — contains("kind: Playbook") returned the same value as the default, so the check never changed the outcome.

1. A -D warnings run is not an inventory. On cli it reports 15; the truth is 39. Warnings-as-errors aborts compilation at the first failing crate, so the main noetl crate — holding 23 of them — was never linted.
2. clippy served cached warnings. A "39 → 6" reading still printed (8/7) after the threshold was raised to 8. The control on a recompiled crate said (9/8). True figure: 39 → 4.
3. clippy --fix introduced a hazard worse than the lint. manual_flatten → reader.lines().flatten() is lines_filter_map_ok: loops forever on repeated Err. An auto-fix is a suggestion, not a proof.
4. A local RED proof returned exit=127 three times and looked clean — the command sat in a shell variable and zsh does not word-split, so clippy never ran. Gating on "0 errors in the log" would have recorded a passing proof of nothing.
5. git checkout -- <file> to drop a probe reverted four real fixes in the same file; the assertion only checked the probe was gone.

Local cargo was 1.97.1, CI ran 1.99.0 — two minor versions, a different lint set. And tools, server and worker have no rust-toolchain pin while CI installs @stable, so their lint set floats: the tools proof run logged stable … updated - rustc 1.99.0 (from rustc 1.98.1) mid-run. gateway and cli pin 1.91.1. Pin decision tracked on #378.

cli's first gate PR looked like a clippy failure and was not: ci/check-member-versions.sh (#331) refused it because the PR changed executor/ while leaving executor/Cargo.toml at 0.10.0 — the release would have published a noetl-executor without the change. It runs before the toolchain install, so build/test/clippy all reported skipped. The companion RED-proof branch died at the same step and proved nothing until it was rebuilt.

Follow-on to the entry below. Everything here is the gate being made trustworthy rather than merely present.

The gate deadlocked itself. CHECK 2 reports a pointer sitting on an unmerged side branch. The fix is to move it back onto the tracked branch — which is never a fast-forward, so CHECK 3 blocked it. The gate reported a defect and then refused to let anyone fix it. That is precisely where #372 sat for a day.

Fixed in #379: a non-fast-forward is allowed when the new pointer is on the tracked branch and the old one was not, reported as a corrective re-point and counted separately. It keys on the old pointer being off-branch, which a genuine regression never is — so the case CHECK 3 exists for still fails. All four controls re-run: corrective pass, regression FAIL, normal fast-forward pass, detached bump FAIL.

The general shape is worth keeping: a check that reports a defect must leave the correction reachable. Two checks can each be individually right and jointly produce a trap.

repos/noetl.io and references/chatui are both private, so an unauthenticated runner could never fetch a SHA from either — they sat permanently unverifiable (no CI access), holding coverage at 31 of 33. Neither carried a live dependency (checked before removing: one dated memory-archive mention, and AGENTS.md, updated in the same change). Removed. Measured on main by the runner itself: declared 31, examined=31 of 31, check 1 31/31, unverifiable 0.

clippy ran as cargo clippy … || true on five Rust repos — present in every log, costing full runtime, unable to fail. The obvious fix was refused by the evidence: a bare -D warnings gives 22 errors across 14 lint kinds on server and 15 across 7 on worker. An outright flip would have landed red on arrival, which is how a gate becomes something people skip — and is almost certainly why || true was written.

Both now gate with the existing kinds allow-listed by name, so a new lint of any other kind fails. Proven by planting a non-allow-listed clippy::len_zero: the clippy step went red while cargo test stayed green, isolating the lint gate. Sweep: #378. cli, tools and gateway deliberately left alone until each is measured.

Two things the inert step had been hiding: an unused std::sync::Arc import in server's tests/chain_log_sourced.rs, introduced by this repo's own recent chain work; and the step's name, clippy (visible, non-gating), which stayed accurate right up to the moment it started gating. Both fixed.

• Re-point repos/docs rather than merge noetl/docs#188. #188 has no CI checks at all — the docs build gate did not exist when it was pushed (it is what #195 added). Merging an unverified RFC to make a pointer valid is the larger, less reversible action, and four docs PRs are open at once, so merging one out of band inverts the dependency. The RFC reference is dropped until #188 lands; re-bump is #380 and will be an ordinary fast-forward.
• ci: released nothing on server and worker, as release-versioning.md says. Verified rather than assumed: both newest tags (v3.122.1, v6.2.1) predate the merge commits.

playbooks/332-embedded-cutover/STATUS.md does not exist — checked every local and remote branch and the working tree. That index entry was always wrong. The live status surface is this wiki: Home, Sessions-Log, and Continuous Integration.

Home's Also open table reconciled again: 1 closed row dropped, 5 open issues added, now 58 rows with 0 gap and 0 closed-listed-as-open. Five of those had been referenced only in the previous Last refreshed prose — replacing that header turned them into durable table rows, which is the more robust place for them.

ai-meta had zero workflow files. That is how #370 merged five submodule pointer bumps with nothing checking them. A gitlink is a 40-hex string and git validates nothing about it, so a typo or a never-pushed SHA produces a repository that cannot be cloned — and nothing anywhere says so.

The ai-meta gate: CHECK 0 remote reachable → CHECK 1 commit reachable from some ref → CHECK 2 ancestor-or-descendant of the tracked branch tip → CHECK 3 fast-forward. Treeless fetches (--filter=blob:none) — ancestry needs the commit graph and nothing else.

The gate asserted "dangling, unpushed, or wrong SHA" for a cause it had never established. Prove access before blaming the pointer.

⚠ I compounded it: I called chatui public because gh api repos/mlflowio/chatui returned the repo — that call carries my token. The probe that matches an unauthenticated runner is an anonymous git ls-remote, which fails. An authenticated probe cannot answer a question about an unauthenticated runner.

Verification coverage on main is 31 of 33, not 33. Two private submodules are permanently unverifiable without a secret, so the gate prints that count on every run rather than leaving the gap implied.

PRs and pushes to main block on pointers that change; the full 33-pointer sweep reports, non-blocking. Failing main on legacy drift a push did not introduce would park main red, which is how a gate gets ignored — the exact failure this gate exists to prevent.

Not only locally. ai-meta: an all-f pointer plus a broken-link/unbalanced-fence markdown file turned both jobs red (check 1 0/1; files=1, links=1, 2 problems), then reverted — by git revert, no force-push, so the proof stays in the branch history. docs: reintroducing the real #364 MDX {#custom-id} produced Could not parse expression with acorn / micromark-extension-mdx-expression. doctor: a clippy lint failed the clippy step while cargo test stayed green, isolating the gated step.

The question asked was "does anything trigger on pull_request?", not "are there workflow files". 16 repos examined.

• ⚠⚠ clippy is || true on server, worker, cli, tools, gateway — it cannot fail, while costing its full runtime on every PR. It passes any "do we run clippy?" grep. ehdb and signal-mesh gate it with -D warnings. #374
• No pull_request trigger at all: doctor, ops, e2e, apt, docs, ai-meta. #375
• ⚠ noetl/noetl runs no pytest anywhere, and check-pr.disabled sits on its default branch — a check deliberately switched off, reason unrecorded.
• ⚠⚠ *.wiki.git repos cannot host Actions. Platform constraint, so "add CI to the wikis" is not achievable as stated; the option is a scheduled ai-meta job running check-docs.sh over the wiki submodules.
• ✅ #232 is stale-corrected: cargo test --all-targets does run on PRs on all seven Rust repos. My memory index had asserted the opposite for months.

doctor: test exit 0 and clippy -D warnings exit 0, but cargo fmt --all --check exits 1 with 396 diff lines. Copying ehdb's ci.yml wholesale would have landed a job that was red on arrival. fmt ships report-only and announces itself when clean. Same discipline for docs — typecheck and build were verified exit 0 on the base first.

• The first script printed unbound-variable errors and still exited 0 — a vacuous pass in the gate meant to prevent vacuous passes. Fixed with a COMPLETED marker and an EXIT trap that turns any abort into a failure.
• Three separate counts were wrong while the verdict was right. My harness grepped ✓ on main and reported "ancestry 3 of 5", missing the two wikis that track master. A workflows=0 column was broken by a failed zsh glob. And the wiki reconciliation first reported 84 stale rows / 67 missing — both wrong, because the measurement crossed into Recently closed and could not see the prose links in Active umbrellas. The real numbers were 12 and 19.
• comm silently produced garbage in both directions because I fed it sort -n when it requires lexicographic input.
• ⚠ CHECK 2 needs deliberate isolation: a side-branch bump also trips CHECK 3, so the old pointer must be the side branch's merge-base — then the fast-forward check passes and only CHECK 2 can fail.
• ⚠ A false-alarm fix can weaken the alarm. Every control was re-run after the ref fallback went in, to confirm an all-f SHA still fails.

Home's Also open table was reconciled against gh issue list --label ai-task: 12 closed issues were listed as open, and 19 open issues appeared nowhere. Both fixed; the table is now 54 rows and matches the query exactly.

⚙️ Shipped: ehdb v0.4.5 (#377) + server v3.118.0 (#477) + ops #317. Pointers bumped.

The defect. event_id is a snowflake minted BEFORE the insert, so commit order is not id order. An event can commit into the middle of an ORDER BY event_id read and shift every position after it. order_by_links never compares two event_ids: it indexes the set, finds the single root, and walks successors — so an interior insert is just a link.

Every failure mode is its own variant (MultipleRoots, NoRoot, Fork, Dangling, Unreachable), because they need different responses and collapsing them into one "cannot answer" is how the previous two divergence classes went unexplained.

(b) hydrator off max(event_id) onto the link tip; new HeadAmbiguous, which still returns a head because stamping NULL on ambiguity adds another root. (c) events_materialize stamps the link it never had — it wrote 10 columns and prev_event_id was not among them, so every row became a root. project_events deliberately unchanged: it is a replication path and re-stamping would make the projection disagree with its source. (d) one-root invariant as a metric + gated endpoint + a sampler, because a gauge only written when a human calls an endpoint sits at 0 — the shape of a healthy system. It ships with sampled_at, pinned at 0 = never.

It forces the condition by constructing a fresh ChainHeads, because a restart's only effect on this path is an empty in-memory map. The pod-delete version is weaker: it only fires if an execution happens to be emitting across the window, and #227 records that in-flight executions often stall. That is the same "wait for it instead of forcing it" failure that let #472 ship and never fire — prod recorded head=0 across 4,270 hydration decisions.

#357 could serve a post-restart execution with several NULL-prev rows by recomputing the edge from log order. That worked — and it was the same trusting of id-order that caused the false divergence. A forked execution is now refused and falls through to Postgres. Safe, but the store covers fewer executions until the write path stops producing forks: 2 of 63 recent plus the whole legacy era.

⚠ So comparator-green is not sufficient on its own — a source that refuses everything also reports zero divergence. The gate is green and coverage.

Rolled 05:41:15Z, armed 05:48:45Z. First reading 16 engagements / 0 divergence / 0 refusals, so it is engaging. The duration-sized re-ramp is running and must span several ~hourly cycles before anything is called green — the last time a 772-comparison ten-minute window was called green it failed an hour later. Rollback is a ~17s three-gate flip, and is preferred over an image roll because every build below v3.118.0 orders the chain by event_id.

• ⚠ /api/events/batch returns {"status":"ok","event_ids":[…]} while persisting nothing — the insert 500s on event_catalog_id_fkey (#363).
• ⚠⚠ Two of my own guards were decorative on first write: one matched link_batch in the comment above the call; the other matched a bare route path, so …invariant-DISABLED still contained it.
• ⚠ A fixture bug (a terminal event with prev = NULL, a two-root execution by accident) and four tests seeding unlinked events, which made them measure the refusal path instead of what they were named for.
• ⚠ cargo fmt -p <pkg> -- <one file> reformatted 39 files and broke a test. The path argument does not scope it.

⚠⚠⚠ CORRECTION 2026-09-30 00:15Z — THE GREEN DID NOT HOLD AND THE GATES ARE ROLLED BACK. The 772/0 reading was real but the WINDOW was too short: an hour later the same deployment read 13 divergences over 839 engagements (98.45%) and the three gates were removed (~16s env flip, env 67 → 64). The prod cause is a THIRD mechanism — an event committing into the MIDDLE of an ORDER BY event_id read, because a snowflake event_id is minted before insert so commit order is not id order (#362). The store was missing one interior row, proven by position arithmetic (store held at position 72 what the log holds at 73). So #360's two defects were real and its fix stands, but it was not what was breaking prod, and the earlier 5-in-79 was most likely this too. A large denominator over a short window is not a sustained result. #360 is REOPENED.

Full detail of the mechanism in #362. What #360 still owns: its own two defects (a behind snapshot is staleness; coverage is monotonic), both merged and independently proven.

⚠⚠ Two more of my own checks lied while diagnosing this. GET /api/executions/{id} returns ORDER BY created_at DESC LIMIT N — the most RECENT N — so "neither conflicting id is present" was computed on a truncated tail. And created_at is stamped at MINT time, so "0 inversions between created_at and event_id order" can never detect commit reordering; I ran it as a refutation and it refuted nothing. ⚠ services/execution.rs also points readers at /api/executions/{id}/events for the full history — that route does not exist.

⭐ The ramp that failed at 6.3% now reads 100.00%. Two defects behind the false divergence, each proven by its own RED control, and prod re-ramped with all three gates armed.

What was wrong. The earlier ramp reported 5 divergences in 79 comparisons and was rolled back — with nothing actually wrong: 60 partitions, 0 persistent mismatches, 24/24 executions COMPLETED. Two causes:

1. populate_from_log classed "the store holds more than my snapshot" as Diverged. Under concurrent reads of one execution the task that reads first can reach the store lock second and find a fresher snapshot already applied; the extra events are real, committed, and the better answer. New FromLog::StaleLog — benign, partition stays servable.
2. Coverage was not monotonic, so a stale snapshot could record a smaller total and make the partition unservable through the guarded read — a false divergence becoming a false refusal. Fixing only the classification would have moved the bug, not removed it.

⚠⚠ My first fix weakened the #358 alarm. "Store longer than my snapshot" is also exactly how a second writer looks, and my reasoning — "populate_from_log is the sole writer" — assumes that case away. The #358 regression test failed on the first run with Got StaleLog { stored_len: 4, log_len: 3 }. From one snapshot the two causes are genuinely indistinguishable, so the verdict moved to where a second read is possible: chain_for re-reads the log and escalates to diverged when the log does not catch up. The populator now reports StaleLog as unresolved rather than deciding.

⚠ And the escalation introduced a new false alarm of its own — releasing the store lock between populate and the guarded read let another reader advance the store, so the chain came back longer than the terminal-flag vector: 2 length_disagreement refusals in 800 concurrent reads. Safe (it declines rather than answering wrongly) but the same false alarm one step downstream. Fixed by holding one lock across both. Diagnosing it needed refusals counted by reason — a bare refused == 0 gave no way to tell a transient DB error from a residual symptom.

Mechanism CONFIRMED, on the intermediate non-atomic build:

10 false divergences in 800 — same mechanism and direction as prod's 5 in 79.

⭐⭐ And then the shipped build stopped exhibiting it at all — stale_log=0, and the pre-fix classification also passes the burst. Making populate and the guarded read one lock hold lengthens the critical section enough that readers queue in read order, so the reordering largely stops happening. A real improvement, and precisely why that test can no longer discriminate: it is documented as a soak, with the discriminating evidence in deterministic tests.

⚠⚠ An INERT test found in the very path being ramped. tests/chain_advance.rs::with_the_flag_on_and_a_source_the_chain_decision_is_used ended }); instead of }).await;, so the async block was built and dropped — the one test asserting the chain decision is actually used, the entire reachability claim, asserted nothing. rustc emitted unused implementer of Future that must be used on every build and it went unread. Armed (it passes), plus a guard that fails on an unawaited block, with a RED control proving the mutant compiles — a non-compiling mutant reads as caught. Two further tests carrying no #[test] at all → #361.

⚠ Gate 1 is now INERT on its own. Since the #358 emit-path call site was removed, the populator's only caller sits behind the source gate, so the old "arm populate, measure, then arm source" sequence has no measurable first stage. Verified on prod at 17:25Z: populate armed, source off, every chain_populate outcome stayed 0. The order still matters — NOETL_CHAIN_SOURCE=chain resolves to OFF without populate, deliberately.

⚠ And this issue postdates the commits that cite it. "noetl/ai-meta#360" was written into a merged ehdb commit, a merged PR body and the pushed v0.4.4 tag before the issue existed; gh issue create revealed it by handing me #360 for something else. The issue was written to match the immutable history, and the other one re-homed to #361. A straight violation of Issue Tracking Rule 1.

PROD RE-RAMP — all three gates armed 17:30Z on server v3.117.2:

versus 5/79 (6.3%) on the rolled-back ramp. Postgres authoritative throughout; rollback is a ~17s three-gate env flip, cheaper than an image roll — and safer, since v3.117.1 predates this fix and rolling to it restores the false-divergence behaviour.

Landed. ehdb #376 → 720c1d2, v0.4.4. server #476 → f630e9a6, v3.117.2, both ehdb pins moved together. ops #317 records the armed gates as-built (env 64 → 67, verified against live in both directions, volumes and PVC compared because that is the #323 outage shape) and admits a ledger gap rather than backfilling rows it cannot verify — the ledger had not moved since v3.109.3 on 2026-09-16. Not applied; prod was changed with set env / set image, single-field edits, per Deployment Validation and apply-safety.

1304 DB-free tests + 14 + 15 DB-gated against a real Postgres fixture.

Headline. All four PRs merged in order, ehdb v0.4.3 cut, both pins moved together, and CI verified the tagged combination. The only remaining step is the prod flag flip, which is owner-gated because it is the one step that touches live decide().

Tag v0.4.3 → c1b78e91, annotated, on the CI-green main tip, verified by object: populate_from_log, the coverage gate, the watermark tripwire, ChainError: Display, LogEvent carrying no prev field, and FORMAT_VERSION still 1 so the on-disk layout gate does not refuse an existing store.

Both pins together. ehdb-feed and ehdb-l0 → v0.4.3 in one commit (3bd3289). A split pin puts two ehdb-l0 versions in one graph; the compile error is the lucky outcome, two engine versions on one layout is the unlucky one.

⭐ The local paths override was removed before that commit, so #472's CI went green against the published tag rather than a local source tree — the one gap the kind proof could not close. The kind suites reproduce against the tag too (11/11 + 15/15), as do 1170 lib tests.

Prod is untouched and the code is inert there. Swept all 6 prod workloads for the four chain env vars: 0 set. The ops IaC does not set them either.

Two records worth keeping.

• ⚠ noetl/server's tag and Cargo.toml disagree as steady state — v3.113.0→v3.116.0 all carry version = "3.112.3", and every release-server run passed anyway. Pre-existing by five releases. Do not "fix" it with a manual bump; that is the 2026-08-03 incident. Read the tag.
• ⚠⚠ Three false-greens from one gh family, all caught before they mattered. --json conclusion yields "" not null while a run is in flight; --limit N spans several pushes so a success can belong to an earlier commit; and --commit <sha> can return 0 rows while --limit shows runs with that headSha — an empty list satisfies "0 not completed", so an until loop exits instantly and prints a blank that reads exactly like a pass. Every green reported was re-verified by pulling --limit, filtering headSha client-side, and requiring a non-empty match set. The merges and the tag were each re-checked this way after the fact, and all held.

Headline. The owner chose "both" for #357. Both halves are implemented and kind-proven; the repoint holds for the explicit prod-canary go. Four stacked PRs, none self-merged. Nothing flipped in prod.

The fix, in one line. Recompute the chain edge from log order — the log's own prev_event_id is never an input, because it is NULL on 643,420 of 645,677 kind rows — plus a coverage record that makes populated from the beginning distinguishable from populated from the middle, plus chain-head hydration so a cold in-memory map stops stamping prev = NULL on running executions.

Kind proof. 11/11 in the new suite, 15/15 in the pre-existing one. RED control (revert the edge recomputation) fails 5 of 9. ehdb battery 8/8, server battery 6/6. Flag-off writes 0 files. The poller path proven with all three gates on: chain_source_from_env yields log_sourced_chain_store and poller_action advances the post-restart execution that used to freeze.

Live-path RED→GREEN, twin fixtures, identical scenario: without the hydrator a running execution's next event is stamped prev=NULL → 2 null-prev roots; with it, prev=2053 → one root and hydrate{head}=1.

Lessons worth keeping.

• ⚠ The root signal cannot be an event-type name — only 62 of 595 executions start with playbook_started; 533 use playbook.initialized. The first guess would have been inert for 90% of them. The root is "position 0 in the log".
• ⭐ Two mutants survived first, and each named a missing test. M7: nothing exercised the log path's failure case. S5: the Pg hydrator mapping a read error to Empty changed no test — the G4 lesson, documented in a comment and never tested. Both closed, S5 against a real database lacking the table, with a positive control that it really is absent.
• ⚠⚠ Six metric labels were not pinned, found by reading /metrics on a running server rather than the code. A guard now computes the recorded set from the source and prints the denominator (13 recorded, 16 pinned).
• ⚠ Two pins that depend on each other are one decision. ehdb-feed pulls ehdb-l0; bumping one put two ehdb-l0 versions in the graph. The compile error was the lucky outcome.
• ⚠ A proof I could not re-run. The fixture was hand-built in a scratch database that was later deleted, leaving chain_source_pg.rs with no fixture — its failures then read like a code regression. Now committed as tests/fixtures/chain_fixture.sql, with 1002's deliberate hole documented.
• ⚠ I nearly shipped a second copy of one error message as a new Display impl beside the existing message() — the drift this repo's rules are about, self-inflicted. And the module-level doc still asserted the pre-fix watermark ordering, one level above the guard that already pinned the function doc.
• ⚠ A wiki row I wrote yesterday was wrong: NOETL_CHAIN_POPULATE is not strict == "true" (it accepts 1|true|yes|on), unlike NOETL_EHDB_EMBEDDED one row above, which is. Corrected — the divergence matters exactly when reading an armed state off a manifest.

Headline. The populator call site landed and was kind-proven, and the same proof blocked the step it was meant to unblock. Two defects make a reader of the chain store wrong on a running execution — the exact cliff the authorized sequence existed to avoid — so the repoint is on hold (#357). Nothing was flipped in prod.

Landed.

Kind proof. Real noetl-control-plane against the kind cluster's Postgres, kind server scaled to 0 so exactly one writer existed, forward identity proved by matching count(*) against an in-cluster query, probe carrying its own negative control, and the forward killed at the end to confirm the probe died with it.

• flag off → 0 files written. Inert, measured rather than asserted.
• all seven metric labels pinned at 0 before anything fired.
• fresh execution → Postgres 6 events correctly chained; store 6, seqs [1..6], watermark 1:6, guarded read all six, root correctly prev=None. Exact agreement with the independent oracle.

What it found. Shared cause: prev_event_id is stamped from an in-memory head map with no DB hydration (server/src/state.rs:260), so after a restart a still-running execution's next event carries prev = NULL and the store cannot tell that pseudo-root from a real one. 534 of 595 kind executions already carry more than one null-prev root.

• mid-flight arming truncates, confidently — Postgres 3, store 1, authoritative first=1 through=1, guarded read Some(1) with root_prev=None. first=1 is true of the store and false of the execution, and the answer is indistinguishable from a complete chain. Open.
• a restart freezes the chain while the watermark advances — Postgres 7, store 6, watermark 1:7. Every later event is rejected for the same not-the-head reason, so the prefix is permanently stale and still authoritative. Fixed in ehdb#374: that disagreement is the only evidence the store has that the log moved on without it.

Lessons worth keeping.

• ⚠ The root marker cannot key off playbook_started — measured, only 62 of 595 executions start with it; 533 start with playbook.initialized. Building on the first guess would have been inert for 90% of executions.
• ⚠ Two pins that depend on each other are one decision. Bumping ehdb-l0 alone put two ehdb-l0 versions in the graph. The compile error was the lucky outcome; the unlucky one is two engine versions opening one on-disk layout.
• ⚠ A doc comment that contradicts its code is worse than none — populate's doc still described the pre-fix ordering after the fix landed, and is now source-guarded.
• ⚠ cargo test --lib <filter> returned a green 0 passed for an integration test — the populator's tests live in tests/, so the filter measured nothing and reported ok.
• ⚠ gh project item-list --limit 200 returns exactly 200 as a CAP, so "not on the board" was an unreliable zero; at --limit 1000 the item was there.

noetl/server v3.112.7 (#462) reached prod by digest at 06:32:24Z and was measured inert, exactly as v3.112.6 (#461) had been. The reconcile give-up cap has still never fired in production.

Measured 34 minutes after the deploy, against a pre-deploy baseline taken on an 80-minute-old pod (not a restart):

Root cause. dispatch_offserver_stateless_drive calls DriveTombstones::clear() as the else if condition, so it runs for every real event whether or not a tombstone exists — and clear() removed the budget along with the tombstone. retrigger_recorded ran at 79.6/min across ~101 in-flight executions: one real event per execution every ~76 s, against a budget needing 1800 s of uninterrupted no-ops. Unreachable by construction.

This is the same defect three times — the budget living somewhere another path resets it. #461 moved the give-up record out of the evictable orch_cache slot; #462 moved the budget out of it; both were necessary and neither was sufficient.

Fix: noetl/server#463 — clear() drops the tombstone and leaves the budget alone. A real event is not progress; an applied drive is, and that already zeroes the budget via reconcile_decision(advanced = true). PR open, not merged — the standing authorization covered #462 only.

⭐ What made this round catchable: the new guard was run against the unfixed code first and failed with after 901 polls (90 real events) the execution is STILL being driven. That negative control is what the two previous rounds never ran. A positive control is included so "preserve the budget" cannot be satisfied by making clear() a no-op. Suite 1213/0.

⚠⚠ Tooling trap worth keeping: a podman COPY --from=builder was served from cache while the builder stage genuinely recompiled for 2m44s, so a rebuilt binary never entered the image and two gate arms were byte-identical to a 45-hour-old build. Fixed with an ARG CACHEBUST before the COPY, plus a content control asserting the arms differ in the right way.

Wiki: noetl/server reconcile-give-up (new) + the NOETL_RECONCILE_MAX_NOOPS row on deployment-specification, which now warns that a giveup of 0 with offserver_retry climbing means the cap is inert rather than idle.

The A2A/ReAct signal-mesh POC and its two design documents left noetl/ehdb for https://github.com/noetl/signal-mesh (public, Apache-2.0, main). It was a consumer of EHDB, never part of it — it folds an EHDB-shaped log and uses ehdb_core::plan::ReadConsistency, and no EHDB crate uses it back. In-workspace, every EHDB build compiled a POC nothing there depended on, and every EHDB release tagged a component with its own lifecycle.

⚠⚠ A ported guard can go vacuous on a layout change without failing. ehdb's hygiene guard tests src_path.contains("/src/bin/"). In a single-crate repo the relative path is src/bin/demo.rs with no leading slash — the src/bin control would have passed over an empty set. Replaced with an is_src_bin predicate plus a test pinning it against both layouts and the near-misses. The population floors also had to drop 20/100 → 3/10; borrowed floors fail for the wrong reason. The guard then fired for real, reporting its own file as untracked before it was staged.

⚠ The src/bin control that stayed behind in ehdb survives only because crates/ehdb-reference/src/bin/ still exists — confirmed by running it, not assumed.

⚠⚠ Three mermaid defects the source could not show. GitHub accepted all seven fences as mermaid, so every source-level signal was green. Rendering the pages found: a hard failure (periods inside a -.label.-> dotted-link terminate it early — the blueprint's dotted labels survive only because they happen to contain none); a diagram that rendered and was illegible (two wide subgraphs scaled to micro-text); and two overlapping edge labels. "It rendered" and "a reader can use it" are independent questions.

⚠ And a near-miss in my own instrument: I first measured all five blueprint diagrams stuck at the iframe's 180px floor and nearly rewrote five working diagrams. The heights were read before the cross-origin resize completed; a control against the same diagrams on repo markdown showed 971/261/388/624/180. The measurement was wrong, not the diagrams.

⚠ gh project item-add returned rc=0 printing nothing, and gh project item-list --limit 500 then did not list the item it had just added (285 scanned). A GraphQL query on the issue's projectItems showed it present with Status Done. Same shape as the gh issue list --limit N cap: the listing was the lying instrument.

⚠ git push over HTTPS is refused for .github/workflows/* when the gh OAuth token lacks workflow scope. SSH pushes fine. There is no GitHub PAT in any of the 8 GCP projects' Secret Manager.

⚠ Repo Map had no noetl/ehdb row at all, despite ehdb being a production repo since 2026-07 — found while registering signal-mesh, fixed in the same change set. The page is a representation, and nothing forced it to agree.

server v3.106.0. The server opens an ehdb-l0 event-log engine in-process, appends what Postgres accepted, and compares counts. It serves nothing — Postgres remains the system of record. The serve flip is a later checkpoint.

Dependency moved rev → TAG (ehdb v0.2.0): the first git-tag-semver pin, closing the 70-commit skew between consumers. FORMAT_VERSION fires at this open.

⚠ A kubectl diff --server-side appears in the audit log as a deployments.patch — it is a dry-run apply. I nearly read my own diff calls as an outside actor.

⚠ A test that cannot tell "disabled" from "broken" is not testing the flag. The flag-off test first pointed at the /data default, where the open fails anyway — None came back for the wrong reason and a mutation deleting the flag check passed. Fixed with a writable temp dir and a positive control.

⚠ System-pool restarts (2 and 26) are the known #322 liveness-kill pattern — exit=0 reason=Completed, last kill 92 min before the deploy, both Ready since. Pre-existing, unrelated, not caused by this.

Docs: server wiki [Embedded EHDB Shadow], ehdb wiki [first consumer], memory topic updated — per the new change-documentation.md mandate.

#332. Reorientation from hardening EHDB-as-a-networked-service to embedding it per shard. Three RFCs, six merged PRs, nothing deployed.

Three components this program needed were already built and never wired — the ehdb-l0 projection engine, D8 RuntimeDataset, and the affinity routing layer. EHDB was built ahead of its consumers, so "does it exist" and "does it work" are independent questions, and only the second is ever answered by shipping. Each was exercised before being built on; all three worked.

Embed, not sidecar — API and storage are one workload, so shared fate is correct. Topology lives in EHDB as D8, an event-sourced projection reusing the same cursor/snapshot/replay machinery. Gossip = library: foca, chosen over chitchat because chitchat is Scuttlebutt + phi-accrual and does not do SWIM indirect probing — the property that distinguishes "I cannot reach X" from "X is down". Registry = git-tag semver, no public crates.io names.

Principle codified: self-sufficiency.md — no external database, not no dependencies.

Recorded because the pattern is the point — in each case a guard was written to prevent a failure class and then exhibited it:

1. the #[non_exhaustive] guard matched its own doc comment — prose counting as code;
2. the format guard's blanket Err(_) => Ok(None) turned every read failure into "absent" → re-initialise, and no test had a substrate whose read fails;
3. the OpOrigin seam test called the authorizer directly, so deleting the call from the append path stayed green — a seam nothing calls;
4. a rename-order test asserted the end state, which is identical either way.

Plus two redundant checks found by mutations passing (a whitespace rule the charset already covered; an unreachable is_some()), and ⚠ a mutation harness that misreported six catches as compile errors because grep '^error' matches cargo's error: test failed.

foca bring-up on prod, step 5's prod checkpoint (server opens a local engine). Prod untouched throughout — server v3.104.3.

#332 · design + gap analysis in docs/rfc/ehdb-embedded-state.md. Read-only; nothing built.

We have been hardening EHDB as a separate networked service. That topology — not a series of unrelated bugs — is the shared root of #320 (relay loss), C5 (election), ehdb#332 (remote tier on the writer's own PVC), #325/#326 (two stores to compare), and the phantom-tier-volume incident earlier the same day.

All 11 crates are libs; there is one binary in the whole workspace. The thing running as noetl-cmdbus-writer is the noetl-worker binary in writer mode — our own process wrapping a library, not a separate product. Every engine has open(config, substrate) / cold_load(...) over a DurableSubstrate seam.

The gap is one seam. The server links ehdb-l0 and never opens an engine — D1EventLog is only a type parameter to PublishRouter, which holds PhantomData and can only connect(shard_count, addrs) over TCP.

Decided on the contention question from two code facts:

• the engine already runs replication/manifest/upload on its own named OS thread behind an mpsc — "The append path never does this" is in the source;
• the API is 100% sync (0 async / 236 sync), so placement cannot be accidental — the type system forces the isolation decision rather than leaving it to discipline.

The sidecar's kernel-enforced isolation is a real advantage, stated honestly and rejected because it keeps an IPC boundary on the hot path — the same shape that produced #320's two mechanisms and #325/#326's two oracles. The purpose is to delete a boundary, not improve one. Reversibility kept by adding the local router as a second impl of the existing seam.

ProjectionStore already describes itself as an append-only log of projection snapshots whose latest is the state. ⚠ Two snapshot layers that must not be conflated: the manifest (which parts exist — the thing that filled the PVC on 09-01) and the projection (proj_seq, the Flink-checkpoint analogue). cold_load is restore; replay-tail and a durable per-shard cursor do not appear to exist — the piece most likely to be underestimated, because the primitives existing makes it look done.

What happened. Applying the reworked prod manifests (ops#301, #323) left noetl-cmdbus-writer-0 unschedulable for ~55 minutes:

FailedScheduling: persistentvolumeclaim "noetl-ehdb-tier-0-data" not found

That pod hosts both the command bus and the event bus, so every worker pool sat in EHDB claim connect failed and dispatch stopped.

Root cause. The manifest declared a fourth volume for a PVC that has never existed in prod. The live StatefulSet had three. The volume was cruft: the tier writes to /data/eventbus/ehdb-tier, a subdirectory inside the existing eventbus-data volume, so the storage was already there and the extra claim duplicated a path that needed nothing.

⚠⚠ 1. Proved one field, applied the whole object. The gate was a server-side dry-run across five workloads showing the image digest preserved — with a negative control proving the method could detect a change. Genuine evidence, about image. It was generalised to the object. kubectl diff against live would have shown the added volume outright, in the same second, at no cost.

A dry-run answers only the question you ask it. The gap between what you verified and what you changed is where the incident lives.

⚠ 2. Predicted a shape-only apply would be restart-free. Three of five workloads rolled. restartedAt and the watchdog annotations live under spec.template.metadata, so removing them changes the pod-template hash and rolls the workload by construction. The mechanism was visible; it was asserted away rather than checked.

Owner patched the live StatefulSet to drop the phantom volume and mount. Writer scheduled and went 1/1 in ~50 s. The system pools showed CrashLoopBackOff with 11 and 17 restarts — not crashes: exitCode=0, SIGTERM received, liveness kills accumulated during the outage (the #322 pattern), with log timestamps predating the writer's recovery. They drained their backoff and reached 1/1 unaided within ~6 minutes.

No data lost, no rollback. All three real PVCs stayed Bound; all five image digests matched pre-apply exactly. Dispatch verified end to end afterwards (probe execution COMPLETED in ~18 s), 0 ERROR lines, 0 5xx.

• agents/rules/apply-safety.md — a gate: diff the full rendered object against live before any prod apply, never the field you changed. Plus: if the diff touches spec.template at all, the workload rolls — state what will restart, and check it.
• drift-audit.sh manifest-vs-live — flags anything declared in a prod manifest and absent from live (volume, claim, mount, container), and checks that claimed PVCs exist. Verified both ways: it fires on the pre-fix manifest with three findings naming the phantom volume, and passes clean on corrected main with 5 workloads compared. It also prints the rollout prediction, so error 2 is detected too.
• ops#302 — main corrected to prod's actual three-volume shape, verified field-by-field with a YAML parse. Until it merged, anyone applying from main reproduced the outage.

server v3.103.5. services::internal::project_events now persists parent_execution_id and worker_id — additive, no schema change; the emitter already published both and they survive into the JSONB through the envelope's flattened extra.

events_projected_total    0 → 60      (exactly the parent+child's 30 + 30)
events_materialized_total 0 →  0

Then on a fresh child: widened comparator match, 0 content divergences; fold-diff's own SELECT diff_paths: 0; parent clean as negative control. All workloads Ready, 0 restarts.

First post-fix run reported authoritative="354748240001769472" ehdb=354748240001769472 — the same value in two spellings. My comparator stringified the i64 while the tier carries a JSON number, and it would have flagged every child as divergent from now on.

The absent values had hidden it: the field was never present on both sides at once, so the spelling never came up. A defect that only becomes visible once the upstream fix lands — fixed in v3.103.5, with the same accommodation ehdb_projection_fold already carried for worker_id.

No INSERT into noetl.event may persist fewer columns than the row carries — mirroring or not. The old invariant was stated in terms of a behaviour the writer happens to have, and the live writer did not have it.

Two sites stay truncating as asserted exclusions (each must still truncate, so a stale entry cannot excuse a future offender): events_materialize (10 cols, not live) and ⚠ db/queries/event.rs::insert_event, which drops error and prev_event_id — the one-level chain link, so a row written there is not walkable. New finding → #327.

⚠⚠ Which writer is reachable is a configuration question, not a code question. The first fix targeted two writers that genuinely had the defect but are not on prod's path; a counter settled it, not reading.

2,370 executions compared, 0 errors: still exactly 15 missing (14 real). The count did not grow across 18 new executions including three post-fix validation children — which is the evidence new children carry it. Backfill held.

#315 is unblocked for new executions; the 15 historical ones still read as unparented until backfilled.

server v3.103.3 deployed, healthy, 0 restarts — and the validation failed to produce the column. Reporting that plainly, because the deploy itself was clean.

/api/execute accepts parent_execution_id, so a child was made cheaply rather than running the planner:

PARENT 354547251244179456
CHILD  354547308949413888
event 354547309033299968  parent_execution_id: authoritative=<absent>  ehdb=354547251244179456

Two independent readers agree it is absent from Postgres — the widened comparator and the fold-diff's own separate SELECT. The parent execution is a clean match with 0 content divergences: a proper negative control.

emit_events: "Gate-off does a single multi-row INSERT; gate-on publishes each row". Prod's bus is EHDB, so the direct INSERT never runs.

noetl_events_projected_total     60   ← exactly the parent+child's 30 + 30
noetl_events_materialized_total   0

That is services::internal::project_events, which lists 21 columns and omits parent_execution_id and worker_id. Filed as #327.

⚠⚠ The method lesson. I identified writers that could drop the column and treated the guard-covered ones as the ones that run. The guard's invariant — "a writer that mirrors must persist what it mirrors" — is sound, and the two sites it covers genuinely had the defect. They are simply not the path prod takes; project_events does not mirror, so it sits outside the invariant exactly as events_materialize does. Which writer is reachable is a question about configuration, not about code, and only a runtime counter answered it.

The invariant is too narrow. The real property is an INSERT should not persist fewer columns than the row it is given carries, mirroring or not — stated that way it points at project_events immediately.

v3.103.3 is still correct, additive and healthy; it closes the gate-off path and the in-tx command.claimed site. Not wasted — just not sufficient. Rollback (8a222c8f) staged and unnecessary.

All 2,352 executions compared, 0 errors:

missing parent_execution_id in Postgres: 15
  automation/agents/mcp/hotelbeds  10
  automation/agents/mcp/firestore   4
  test/simple_loop                  1   ← the validation child

14 real, one event each. A backfill is small — and #325's widened comparator is what verifies it, since it is what detects the condition at all.

Read-only. The residual #325 found turns out to matter more than its size suggested.

noetl.event has three writers and they do not persist the same columns:

handle_batch_events also mirrors — and mirror_rows serialises the full EventRow. So it sends 17 fields to the tier and persists 12. The tier is not enriching; it is faithfully copying a row Postgres then truncates.

⚠ There is already a guard, every_in_tx_event_insert_is_mirrored, counting INSERT sites against mirror sites so a new writer cannot bypass the mirror. It checks that each INSERT mirrors — nothing checks it persists the same columns the mirror sends. That is the gap this fell through.

Not latent: 8 SELECTs in db/queries/event.rs, the projection fold (:173/214/377/931, compared at :1485), the orchestrator sets it when spawning children — and nonconvergence_sweep.rs:556 uses it for skipped_parent_active:

-- `parent_execution_id` is carried on the child's own events, so the parent id
-- comes from there

The comment states an assumption the data does not satisfy. NULL column ⇒ subselect yields nothing ⇒ EXISTS false ⇒ active false. The guard fails in the unsafe direction: a child with a live parent reads as having no active parent and becomes eligible for termination.

That lands squarely on the pending decision to arm the sweep for the 100 stuck executions. Those 100 are test/simple_loop (top-level, not at risk), but the affected children are automation/agents/mcp/* — real agent work. Cross-linked to #315.

Surfaced, not built — persisting the missing columns is a write-path change (narrow: values already on the row, column already exists, no schema change). Inert and offered: a guard asserting every INSERT site persists the column set the mirror serialises — the check that would have caught this.

Scope unmeasured: 5 of 200 sampled, but the affected population is child executions specifically, so the per-child rate is likely far higher.

server v3.103.2, still inert. Same sweep each time (newest 200, narrow vs widened, 0 errors):

• context — the PERMANENT_LOG_LEAN externalisation: authoritative holds {"__context_ref__":…,"__context_bytes__":…}, the tier the inlined object. The same reference-vs-inlined class as result, in a different spelling, which is why the first collapse missed it.
• worker_id — a location difference. The column is often NULL while meta->>'worker_id' carries the value and the mirror copies it top-level. Read off a prod record, top-level and meta.worker_id are the same string. A missing value that was never missing.

I reported that the fold's differ already classified all six field diffs as benign, because input_event_diffs_post_normalisation was 0. It is not that: that list compares created_at only — (a.created_at != b.created_at).then(|| …) — so it is 0 by construction whenever timestamps agree. normalise_event_precision truncates timestamps and nothing else. No existing normalisation reconciles those fields. I read a narrow check's zero as a broad claim, which is the exact shape representation-drift.md exists to catch.

All 5 are playbook_started on a child execution, where the tier holds a parent_execution_id and noetl.event does not — and it is not in meta or context, so it is not another location difference. The system of record knows less than its mirror, and it is the first genuine thing the widened comparator has found.

Still held: flag off, alert untouched. 2.5% is not zero, so enabling still needs flag + alert to move together.

⚠ Build note: publish-image (arm64) failed and publish-manifest was skipped on this release, so GHCR has no multi-arch image — but publish-ar (Cloud Build) succeeded independently. The deployed AR image was verified architecture: amd64 against amd64 prod nodes before rolling, rather than assumed.

server v3.103.0 deployed inert. It adds ?content=true on the on-demand parity route so the widened comparison can be measured without arming the sampler — the env flag arms the process, the process includes the sampler, the sampler records, and recording is what pages. There was no way to find out what enabling would surface without first paging on it; now there is.

Newest 200 executions, compared twice each, 0 errors:

narrow  : match 53   divergent 146
widened : match 23   divergent 176
MATCH -> DIVERGENT: 30 of 200 = 15.0%
fields: worker_id 762   context 622   parent_execution_id 5

result never appears, so the reference-vs-inlined collapse does its job. But three other classes do, and none is corruption:

• context — authoritative holds {"__context_ref__":"noetl://…/__command_context__/…","__context_bytes__":1011} against the tier's inlined content. The same pattern under different key names — the NOETL_PERMANENT_LOG_LEAN externalisation, which the collapse does not recognise.
• worker_id, parent_execution_id — authoritative absent, tier populated.

The proof it is benign is a measurement, not a hope: the fold's own differ reports exactly these six field diffs on the same executions and then input_event_diffs_post_normalisation: 0. The platform already classifies them as equivalent; the comparator merely lacks that normalisation.

So the two oracles disagree in a second way — not just narrow-vs-wide, but different normalisation. Enabling today would page on 15% of executions over differences the system already knows are equivalent, which is the exact outcome the flag exists to prevent. Held. Alert untouched.

Fix before it can be enabled: teach the collapse __context_ref__; port normalise_null_json / normalise_event_precision; re-run this sweep.

NOETL_RECONCILE_MAX_NOOPS is unset, so the default 225 applies (225 × 8 s ≈ 30 min). ⚠ My earlier "the reconcile bound is OFF on prod" conflated it with NOETL_NONCONVERGENCE_SWEEP_ENABLED, which is a different, execution-terminating thing and stays off. noetl_reconcile_giveup_total{max_noops} is present and 0 — present matters as much as the value.

The waste shape is not recurring: 8 drive dispatches in ~3 h. There are 100 non-terminal executions up to 97 h old, and they are not the #235 frozen-column artefact — sampled ones have 3 events, last command.issued, last activity 2026-09-01. Inert residue of the truncate incident, not an active loop. Clearing them is the nonconvergence sweep's job, which terminates executions; not enabled under discretion.

#313, #314.

server v3.102.0, merged inert. Not enabled anywhere.

#325 built: MirroredEvent deserialised five fields and payload_divergence compared three, so match meant "the identifying projection agrees", not "the tier can stand in for Postgres" — and the comparator, not the fold, is what feeds the paging alert.

NOETL_EHDB_CROSSSTORE_PARITY_CONTENT (default off) widens it to context, result, meta, error, worker_id, parent_execution_id. Off, every verdict is byte-identical and the widened columns are not even SELECTed — an unconditionally-widened query would put whole tool payloads on the wire every parity tick.

Postgres stores a result as a reference where the tier stores the inlined context, systematically. Comparing those raw would report a divergence on essentially every execution and mean nothing. collapse_result_representation collapses an object carrying exactly one of the two to a marker — and leaves an object carrying both alone, because that is not a representation choice. null / absent / {} / [] all agree, because the authoritative column is a SQL NULL while the mirrored payload carries {}.

⚠ The collapse asserts the two forms are equivalent, not equal — the pointer is never dereferenced. A tier whose inlined payload does not match what its reference resolves to is still invisible.

Two controls, content and result_representation, both driven through compare_cross_store_with_content, which forces the comparison on regardless of the env — reading the flag there would make the controls silently vacuous on exactly the deployments that have not armed it. Five mutations, each caught.

⚠⚠ Enabling is not merging. Arming the flag reclassifies a large share of today's match as divergent and pages on a condition that has been true for as long as the mirror has existed. Procedure lands in the same change set: playbooks/325-content-parity/ALERT-RETUNE.md — measure with the Inspect-only route first, re-point the alert off kind="content", then arm the sampler. #325 stays open until that has run; the Closes keyword auto-closed it on merge and it was reopened.

#314 on its stated goal — iteration_command_ids is already a BTreeSet. #313 — the duplicate mechanism is structurally prevented by the tier's event_id dedupe, with the 8 surplus events out of 60,673 written down rather than waved away as the reopen trigger.

Read-only. The answer is the fold, and the comparator is not wrong — it is narrow by construction, which is worse, because its verdict reads like a guarantee.

MirroredEvent (ehdb_parity.rs:188) deserialises five fields; payload_divergence compares three — event_type, node_name/step, status. It never parses context, result, meta, error, worker_id or parent_execution_id. So a match verdict means "the identifying projection agrees", not "the tier can stand in for Postgres".

GET /api/ehdb/projection-fold/diff/{id}?source=tier — a parameter that exists precisely because the WAL differ could not diagnose this — returns the same shape on every disagreeing execution:

/steps/<step>/result/context/result/context    (only in tier)
/steps/<step>/result/context/result/reference  (only in postgres)

Execution 354320837223915520 shows it across four steps. Reading the tier record directly confirms it: the tier holds the inlined payload (content, control_data, …) where Postgres holds a reference. build_result_object (events.rs:1560) selects between the two by result_kind, so the stores are recording one logical result in two representations.

⚠ input_event_diffs_post_normalisation is 0 — the differ's own normalisation makes the inputs equal while the folded states still differ, so the raw-input diff understates it. The fold is what surfaces this at all.

The comparator is the weaker oracle and it is the one wired to the paging alert — a tier could diverge in every payload and still report match everywhere. The event-log tier is primary-serving, so a tier-served read returns inlined content where Postgres returns a pointer. And it defeats NOETL_PERMANENT_LOG_LEAN=true inside the tier, which is retaining exactly the payloads that policy exists to keep out.

Filed as #325. The inert fix — widen the comparator to cover content — is proposed with the caveat that it converts a large share of match into divergent and would page on a condition that has been true all along, so it needs a flag or a re-tuned alert in the same change set. Whether the mirror should instead capture the post-substitution row is a design question about what the tier is for: surfaced, not proposed.

MEMORY.md compacted 24,900 → 19,973 bytes (5,013 headroom, was 86). All 61 topic links preserved, zero dropped, zero broken; long lines 10 → 3; prod state and the owner-gated list refreshed against the cluster rather than copied.

iteration_command_ids is already a BTreeSet (orchestrate-core/src/state.rs:198, changed by server#388), so the order-sensitivity this issue was filed for no longer exists.

Worse, I had commented on #314 that the ~951 order-only executions from the cross-store sweep were "the same shape as this issue". They are not, and that error propagated into a plan to fix them by canonicalising iteration_command_ids. Classifying all 1,325 order occurrences by their detail string across the 2,138-execution sweep:

A: "tier records were not returned in ascending global_sequence order"      0
B: "shared events differ in order — tier […] vs authoritative […]"       1325

Zero type A. All type B — the tier genuinely holds shared events in a different relative order than the authoritative log, most likely via the mirror queue's own documented rung 3 (queue_full_inline reached 404 in one load run). Nothing to do with a set field inside WorkflowState.

⚠⚠ And the live residual is a third thing again. The fold equivalence on prod reports 9 disagreed of 39 comparable, n=14 vs n=14 — same version, same event count, different digest, disagreements_explained_by_missing_context = 0 — while the cross-store comparator calls one of those same executions a match with zero payload divergences. Two parity surfaces disagree with each other, and ordering is already handled on the tier side (ehdb_projection_fold.rs:346 and :699 both sort by event_id). That is a real investigation and it should not inherit #314's now-false title.

"Nothing reads it" was true of the worker and false in general. The server reads NOETL_OFFSERVER_ATTACH_TAIL through envy into AppConfig.offserver_attach_tail — no string literal exists to grep, which is exactly why it looked unread — with four readers on the hot path. A blanket delete would have removed a live server flag, and the flag it imitates is the #156 tail-attach flag, deliberately off because turning it on broke auth.

Four worker-side declarations removed (two per manifest — it had been duplicated into the wait-for-api initContainer), server declaration kept, live env aligned, both pools rolled and healthy. Inert: the removed value was false, which is also the serde default.

POSTGRES_PASSWORD is unbound — and so is every other key in noetl-secret, and so is the Secret itself: zero secretKeyRef/envFrom bindings across every workload in the namespace, consistent with the Secret Manager cutover.

Not acted on: NOETL_ENCRYPTION_KEY sits in the same object, and #309 already records that a DR apply would replace the live encryption key. "Unbound" is not "unused" — a key nothing binds today may still be the only surviving copy of something that decrypts existing data. Disposition belongs with #309, as a whole-object decision, after confirming the key exists and matches in Secret Manager.

Follow-on to the #320 close. Three residuals worked, one of them corrected.

ehdb-env-drift-check.sh (which prints its own positive control) found six variables live on prod and declared nowhere, i.e. a DR re-apply would silently remove them. The dangerous one: NOETL_EHDB_EVENTLOG_MIRROR_DRAIN_CONCURRENCY=1 — losing it reverts to the code default 8, the configuration that produced ~99% relay delivery failures. Next worst: NOETL_EHDB_SEAL_MAX_AGE_MS=5000, which has an alert on the seal trigger not firing, so losing it disarms the behaviour and the alert watching it.

Declared in ops#299, verified both ways: the LIVE but NOT declared class is now empty and three of four workloads report agree. Placement was checked structurally — the first attempt put one value into the wait-for-api initContainer, which a by-eye review would have missed.

⚠⚠ The manifests are still not safe to apply. Every image pin is behind prod: server v3.90.0 vs v3.101.3, writer v5.122.2 vs v5.131.0. Applying would roll back past the #320 fix and past v5.129.1 — reintroducing the unbounded cmdbus manifest growth that caused an 8-hour silent dispatch outage. Filed as #323.

Corrected by separating the classes over all 50 lifecycle events: of 15 Killing events, 5 were preceded by Unhealthy on the same pod (genuine liveness), 10 were rollouts or evictions. Three causes, only one a probe issue:

1. 5 real liveness kills — and the probe is not misconfigured in the obvious way. I expected the classic "idle pool looks wedged" bug; claim_loop_wedged is deliberately keyed on failures, not claim recency, with the reasoning written out at metrics.rs:2732. That trap was already found and fixed.
2. Rollouts — noetl-state-builder-watchdog (a 3-minute CronJob with cooldown + flap-stop + MAX_ATTEMPTS) doing designed remediation, plus deployments.patch by [email protected] from outside this session.
3. ⚠ Preemption under capacity pressure — shard1 shows Preempted + 24 FailedScheduling. That is capacity (#318), not configuration. ⚠ Found only after noticing --limit is a cap: the first query fetched 1000 events that excluded shard1 entirely.

Still real and surfaced, not applied: readiness is a tcpSocket probe while liveness is HTTP, so a listening-but-wedged pod stays Ready and keeps receiving mirror traffic for the full 90 s before the kill.

Its rationale called this "an auxiliary verification path" that must not add "latency on the caller's hot path". Both premises are stale: NOETL_EHDB_EVENTLOG=primary makes the tier primary-serving, and the mirror has been async since #155. The constant outlived its reasoning.

The two have opposite risk — a slow read should give up fast; a slow append that gives up early means the record was not written. Split in worker#310 (v5.131.1, deployed to the three Deployments; the writer StatefulSet deliberately not rolled). Four mutations, each caught; the behavioural gates measure on the wire against a listener that accepts and never replies.

Over the 11 hours after v3.101.3: 35 events short across 2 real muno/itinerary-planner executions at 17:44, with no pod churn in the window — the writer-slowness class, which is exactly what v5.131.1 targets. Repaired 35 → 0, idempotent (already_complete / repaired: 0).

So the floor is: 63 executions, 1663 mirror-expected, 2 affected, now 0 short — but a sink outage longer than the retry window still loses events, and the single-endpoint relay makes such outages routine. That part is #322 / #318 and owner-gated.

drift-audit.sh reported two gaps this session created or exposed: the new attempt-label set read NO-GUARD (pinned_sets.py cannot read a computed label — reaching for "the next quoted string" yields dropped, a terminal outcome, which the first draft of the replacement guard did), and the env-docs check demanded the spec document two NOETL_TEST_* fixtures. Both closed; the fixture filter prints its count, because an earlier attempt at that same filter had a syntax error that made the comparison emit nothing while printing OK.

Three releases and a repair. The clean floor the session was aimed at.

1. No retry. deliver POSTed once and returned () — structurally unable to report failure — and all three failure arms ended in a warn! and a discard. Concurrency was never the mechanism, only a rate multiplier: measured over six hours with the queue completely idle (enqueued 144 = drained 144, queue_full_inline 0), 9 of 144 batches — 6.25% — were dropped.
2. ⚠⚠ Stale pooled connections. relay_client() was reqwest::Client::new(), which reuses idle keep-alive connections for 90 s. The relay target is a ClusterIP Service in front of a single pod killed by liveness several times a day (#322), and mirror traffic is bursty — so the pooled socket from the last burst often points at a pod that no longer exists, and every retry draws another dead one from the same pool. Retry alone could not fix it: after v3.101.1 a batch still dropped with all five attempts failing inside four seconds against a pool Ready for 100 minutes. This is what finally explained the 2026-09-03 muno losses, which happened at light load with no pod churn at all — a window the earlier diagnosis had to report as unpinned.
3. A third, upstream: under load the worker's tier append returns 502 … "timed out after 2s" (serve_state=no_durable_service). Absorbed by the retry window, not fixed.

dropped is now a terminal counter meaning these events will never reach the tier. It had to exist because discards were filed under unavailable, and none of the 18 prod alert policies referenced it: all four mirror alerts watch ..._queue_total — the enqueue ladder whose own module header says "never a drop" — and not the delivery step that drops. The queue read perfectly healthy for days while events were lost. Drops now log at error with the event_ids, so a loss is repairable rather than only discoverable by diffing both stores.

The 131 recoveries are the point: the retry path was exercised, not idle — a clean run with no failures would have proven nothing.

⚠ The widened window trades against drain throughput. With DRAIN_CONCURRENCY=1, a batch retrying for 80 s blocks the drain for 80 s: a 42-execution burst backed the queue to 446 batches and pushed 404 to inline delivery (rung 3, i.e. back onto the request path). It drained fully with zero loss — but I first read that backlog as a 468-event regression before checking pending_events. Widening further without raising drain concurrency would make it worse.

⚠ Defaults were raised to the measured values (7 / 500 ms) rather than left below what prod ran by hand — a default quietly worse than the deployed value is the #267 shape, where a DR re-apply silently reinstates a losing config.

354018056919326720  32/27 short 5  muno/hotel-cards      ->  32/32  short 0
354008288930570240  32/30 short 2  muno/hotel-cards      ->  32/32  short 0
354005901364961280  32/30 short 2  muno/hotel-cards      ->  32/32  short 0
353750965720981504  14/ 7 short 7  system/scheduled_...  ->  14/14  short 0
TOTAL 16                                                 ->  TOTAL  0

Zero missing_event remains; the surviving order divergences are the #314 class. A second call returns already_complete / repaired: 0. The ~5,409 synthetic test/simple_loop events were deliberately left. ⚠ With the async mirror the endpoint's own after-count is taken before the queue drains, so it honestly reported partial on a repair that completed seconds later — the outcome being keyed on the post-repair missing count is what made that visible instead of a false success.

Still open: #322 (the liveness kills that trigger it), the writer's 2 s append timeout under load, and #267 (neither retry var is declared in any ops manifest).

The cap was not a cap. GET /api/ehdb/executions accepts offset, so the whole population is reachable read-only. ⚠ It serves 100 per page even when limit=1000 — the body's limit echoes the request, not what was served — so a loop that stops on a short page stops at 100. That is the entire reason the morning's figure was a bound.

Swept the real comparator over all 2,138 executions: 434 s serial, 0 errors, every row controls_ok=true / metrics_recorded=false / tier_query_source=service. Safe to sweep only because #264 was fixed on that route — otherwise 2,138 probes would have inflated the divergence counter they measure.

authoritative_expected  60673      ehdb_count  55256
pending_tier (STRUCTURAL)    0      surplus          8
DEFICIT  5425 events / 548 executions   (25.6% of executions, 8.94% of events)

Self-consistency: the newest-100 slice reads 316/51 today vs the 318/52 reported this morning — same window, same answer.

• 99.7% of the loss is test/simple_loop — synthetic load traffic. Non-synthetic loss is 16 events across 4 executions (three muno/playbooks/hotel-cards, one system/scheduled_cleanup). Sizing a repair off 5,425 overstates real exposure by ~340×.
• 951 order-only executions are 100% complete and 100% test/simple_loop, spread evenly across all three days — the #314 shape, corroborated on a second independent surface. They are what makes the raw "1,507 divergent" look alarming: any gate keyed on divergent is dominated by ordering, not loss (2.75× false-signal on this population).

DRAIN_CONCURRENCY=1 is live (rolled 2026-09-03T19:28Z). After that cut, 3 of 9 muno executions on prod lost events, and a run today at 11:22 lost 6 of 30. #320 should not be closed on the mitigation — the 04:00 load hour was 422/422 executions affected (4,674 events), but the residue outside it is still happening. n=38 post-cut, so the rate is not estimable; the existence is.

The morning reading attributed 6 events to 09-04; that execution is clean now, so it was in-flight lag, not loss — the "structural gap: 0" was slightly optimistic. With a 30 s tolerance and an async mirror, sweep executions that are hours old, not minutes. The 318/52 headline was unaffected.

MEMORY.md compacted 28.7 KB → 23.0 KB (it was truncating on load at a 24.4 KB limit); all 58 topic links preserved, 1 added, 0 broken.

Held: the repair run stays owner-gated — it is a write into the primary-serving tier. The endpoint (v3.101.0) remains inert and unrun; prod stays on v3.100.4.

Two things were asked for and both are done, read-only and inert respectively.

The figure in circulation was "~12k dropped records". That number came from failed batch deliveries during the #320 window, which is not a completeness measurement — a delivery can fail and be retried, and a batch is not an event. Measured against the cross-store comparator instead, over the 100 executions the parity API exposes:

mirror-expected events:   2432
tier holds:               2114
STRUCTURAL gap:           0      <- measured, not assumed
REAL DEFICIT:             318
executions affected:      52 of 100 (52%)
executions with SURPLUS:  0      <- no duplicates
by day:   2026-09-03: 51 executions / 312 events
          2026-09-04:  1 execution /   6 events
04:36 cluster (#320 load-run window): 48 executions / 303 events

The separation the owner asked for holds cleanly: structural mirror lag is 0 in this window, so the entire 318 is real loss rather than in-flight lateness. ~95% (303/318) sits inside the #320 load-run window, so the loss is #320-attributable rather than ordinary operation.

⚠ 318 is exact for the visible window and a lower bound overall. The parity API caps at 100 executions regardless of limit, and the affected set cannot be enumerated from logs — the server pod is 4h56m old and the #320 window predates it. Anyone quoting a total should quote this bound, not extrapolate from it.

POST /api/ehdb/repair/executions/{execution_id} (server#399, v3.101.0) re-mirrors only the event IDs the comparator reports absent. CrossStoreReport gains missing_event_ids (skip_serializing_if empty, so the wire shape is unchanged when nothing is missing) — that list is the endpoint's input.

Four properties, each held by a gate that a mutation breaks. All four mutations were applied, verified to fail the intended gate, then reverted:

⚠⚠ The non-inflating property is the one worth carrying forward. #264 already recorded that the parity endpoint writes the counter its own paging alert reads, so investigating a divergence inflates it. A repair endpoint that compared under Record would do that twice — the divergence counter would climb during a successful repair, and the operator watching the alert would read the fix as the problem getting worse.

Suite: 1040 passed, 0 failed; clippy 0 errors; CI green; merged as v3.101.0. Inert: new route, no caller, no sampler, no change to any existing path. Not run against prod. Prod stays on v3.100.4 (digest 03cecf3f, imageID-confirmed on the running pod).

• ⚠⚠ An in-pod /metrics probe reported build_info lines=0 on four separate ports. All four zeros were false: curl is not in the container. kubectl exec … curl fails with "executable file not found", and piping that into grep -c yields a confident 0 that reads exactly like "the metric is absent". Only the positive control (-w http=%{http_code}) surfaced the real error. The prod version was resolved from the Artifact Registry digest instead.
• ⚠ rustfmt given a path recurses into newly-declared modules; given stdin it cannot. A fmt baseline that pipes git show HEAD:file and compares against rustfmt --check file is not like-for-like — mod.rs jumped 28 → 1019 diff lines purely from adding one pub mod line. The measurement was wrong, not the code.

Tier re-aim backfill run, further worker rolls beyond those done, the cutover flip, C5 election, #332 migration, quorum / replication / double-writes, system-pool autoscaler enable, further Cloud SQL changes, IAM / rotation, and the pgbouncer duplicate-identity fix.

Next decision for the owner: whether to repair the measured 318 (52 executions), and whether to widen the measurement past the API's 100-execution cap first.

Pointers: server#399 · ai-meta@a42b823 (gitlink-only bump to a0d4278) · server wiki deployment-specification · #313 · #320

Headline: server v3.100.4 is LIVE on prod. The p95 tail and the run-to-run bistability on POST /api/execute were the event-log mirror falling back to inline delivery — not the command-publish retry budget I had named in #319 §4.

§4 identified the 10 s publish retry as the tail. It was assembled from constants read in source — a 10 s deadline, 32 attempts, an exponential backoff. That supports a claim about what could happen; it was published as what was happening, with no instrument that could contradict it.

server#393 shipped the fix (v3.100.3) and the counter it added read:

noetl_ehdb_command_publish_deferred_total 0

Zero, across a full load run — the publish always succeeded inside 500 ms — and p95 stayed at 14.9 s.

A hazard visible in the source is not the cost visible in production. Build the instrument before publishing the conclusion. server#393 stays as a correct defensive change that fixes nothing that was happening.

emit_mirror measured 1.68 s mean on the request path; everything else was noise (emit_publish 50 ms, emit_should_publish 2 µs). MIRROR_ASYNC=true was set and the queue was armed — and still:

queue_total{outcome="enqueued"}             2463
queue_total{outcome="enqueued_after_wait"}   327
queue_total{outcome="queue_full_inline"}     908    <- 24.6%

A quarter of all batches went back onto the request path, at up to 10 s each (5 s enqueue timeout + 5 s POST), three events per request. Mean queue lag 170 s; only 56 of 2,790 batches drained inside 10 s.

deliver_pass coalesced per execution and then POSTed strictly one at a time — throughput 1 / relay-round-trip regardless of how many independent executions waited, while its own comment already said different executions are independent. It explains the bistability better than §5 did: queue fills → submits go inline → requests slow → more in flight → queue stays full.

• Ordering is safe by the queue's shape, not by assuming order does not matter: a pass holds at most one batch per execution_id and passes do not overlap. NOETL_EHDB_EVENTLOG_MIRROR_DRAIN_CONCURRENCY=1 is the rollback.
• P2 introduced no cache — the row is read fresh every request, so a re-registered version is picked up on the next one. A negative control fails the suite if content is ever retained.

Mutations, each verified to fail the intended guard: reordering without coalescing (the existing end-to-end ordering test), a silently-serial spawn loop, ignoring the concurrency env (the negative control), a re-added snapshot catalog read, a request-spelled system-path memo, and an accidental catalog read-cutover.

Live freshness control on prod: 25/25 recent executions resolved test/simple_loop to the highest of its four versions.

Full clean curve, 1/2/4/6/8/10 × 2 reps, nothing probing the server:

rep conc submit ok  err p50ms  p95ms  sched
1   1    46     46  0   120    2820   5s
1   2    46     46  0   140    11010  6s
1   4    20     8   12  30040  30080  6s
1   6    0      0   0   -      -      64s   <- SKIP, no generator scheduled
1   8    91     81  10  3430   30040  49s
1   10   36     7   29  30040  30050  54s
2   1    19     19  0   150    12050  6s
2   2    35     34  1   150    8000   6s
2   4    32     20  12  7550   30150  6s
2   6    18     0   18  30030  30110  6s
2   8    24     0   24  30040  30210  5s
2   10   102    84  18  7230   30030  49s

c=8 goes 81 ok → 0 ok and c=10 goes 7 ok → 84 ok between two runs of the same configuration. That is the original cliff, not a softened one.

I had reported p95 14,910 → 3,070 ms at c=1 from a single rep; the repeat read 12,050 ms. One sample of a bimodal distribution, published as the distribution — the same error as §4, one level down. p50 is the part that is genuinely reproducible (510 → 120/150 ms at c=1; 5,730 → 140/150 ms at c=2).

Run-wide deltas: inline fallback 24.6% → 8.3%, emit_mirror 1.68 → 0.78 s, lag 170 → 67 s, publish_deferred +0. Everything ~2-3× better; nothing eliminated. The queue was still pinned at 512 ten minutes after the load stopped.

So the tail is now cleanly P1-bound. More drain concurrency cannot help — 8 parallel POSTs still land behind one writer taking ~4 ms fsync per append.

Checking the mirror's success counters after the run:

v3.100.3 (serial)         mirrored=3541  unavailable=70    degraded=84     ->  4.2% fail
v3.100.4 (concurrency 8)  mirrored=101   unavailable=10087 degraded=1882   -> 99.2% fail

The relay is a single-pod headless service and does not tolerate 8 concurrent POSTs; a direct probe from the server pod succeeds instantly, so it is the concurrency rather than the relay being down.

This is why the drain looked fast. queue_full_inline 0 and emit_mirror 0.000 s read as the fix working; the drain was fast because deliveries were erroring out quickly. A failure path faster than the success path flatters every latency metric that measures it — and there was no success-rate series beside the latency one, so an improvement and a fault were indistinguishable.

⚠ Compounding it: deliver handles a relay error by logging and dropping the batch — no retry, no re-queue. Pre-existing, but this moved it from 4% to 99%, so roughly 12,000 batch deliveries were lost from the tier in that window. That matters for anything reading the tier as an oracle ([#258], [#263], [#313], [#314]).

Filed as #320. Not an active incident — prod idle, queue depth 0, zero deliveries in 60 s, and the tier is not serving reads on this server. Mitigation is NOETL_EHDB_EVENTLOG_MIRROR_DRAIN_CONCURRENCY=1 (keeps server#394) or an image rollback to v3.100.3; applying it needs an operator — the change was blocked for this session.

⚠ Two rig defects surfaced: a SKIP row reads like a collapse (0 0 0 with - latencies is the only tell), and levels run back-to-back with no drain interval, so each inherits the last one's backlog — the two best rows both had ~50 s of Autopilot scheduling in front of them, which is free drain time.

• I perturbed my own measurement. Pulling a 75 MB catalog listing off the server pod mid-run collapsed that run's c=4 level. Discarded and re-run clean, touching nothing during the run — the harness rig exists to stop exactly this, and the probe was outside it.
• cargo fmt -- <paths> formats the whole workspace, not those paths. It rewrote main.rs into a shape auth_gate_wiring cannot parse — the same failure as server#390. Reverted; touched files were checked against their rustfmt baselines instead, which were already non-zero.
• Two new scan idioms were wrong on first run and said so loudly: a doc comment counted as a SQL query, and &'static str matched a search for retained static state.

P1 — the ~4 ms per-append fsync — is parked behind ehdb#322. The drain cannot outrun the writer it feeds, so the queue still saturates under sustained load and ~2.8% of batches go inline at higher concurrency. DRAIN_MAX_BATCHES (more records per POST, fewer fsyncs) is the likelier next lever than more concurrency, but it risks the tier's page cap and wants measuring first.

Pointers: ai-meta@baae005 (gitlink-only) · server@8b39283 (v3.100.4) · noetl-server-wiki@72f7696 · Umbrella: Request latency

Owner lifted the hold for the rollout. statefulset/noetl-cmdbus-writer moved to @sha256:a6e578c6… (v5.129.1, verified pullable HTTP 200 before rolling — never roll to a tag or an unbuilt digest).

Space was reclaimed, not merely held. drift-audit.sh volumes went from 2 DRIFT to all OK — the same mechanical check, before and after.

Bonus: the eventbus L0 engine pruned too (manifest 356 MB → 28 MB). Both engines in the writer process share the retention policy.

⚠⚠ Pruning is not visible immediately after a roll and that is not a defect. At 1 append it read pruned 0 / retained 0 and the snapshot count had risen 172 → 173. The sweep runs once per retain manifest writes, so it had not fired. A 120-execution burst (108/120 at p50 277 ms, no 500s) drove it, and pruning completed. Drive traffic before concluding the fix is inert. Note also that retained is only written by the sweep, so until the first sweep it reads 0 — indistinguishable from "0 retained".

⚠ The burst's backlog briefly took system-pool lag to 107 and it grew across three samples before draining. That reads exactly like the #315 re-drive loop; it was not. Only 2 re-drives in 5 minutes, both old pre-truncate execution ids. It resolved on its own: 100/100 executions terminal, lag 0.

Dispatch verified before and after the roll. Pod Ready, 0 restarts, all listeners bound, 0 errors. Rollback to @sha256:1c7d5387… was staged throughout and not needed.

The owner lifted the publish hold for worker#304 only. Merged as 5e8d006; semantic-release produced v5.129.1 (fix(ehdb): → patch), verify-version passed, and all seven release.yml jobs succeeded including publish-ar.

Verified rather than assumed: the tag contains the merge commit, the ehdb pin inside the tagged build is 0980183 (both ehdb#346 and ehdb#347), and 11 metric render sites are present. The AR image @sha256:a6e578c623dce… is linux/amd64, matching the prod nodes — and the current v5.129.0 is single-arch too, so that is the established AR pattern rather than a defect (the multi-arch index lives on GHCR).

⚠⚠ Publishing changed nothing on the running system. The writer StatefulSet is pinned by digest to @sha256:1c7d5387… and still reports noetl_worker_build_info{version="5.129.0"}. The pruning code only executes inside the writer process, so /data/cmdbus continues to grow at the measured ~12 GB/day against 18.6 GB free — about 36 hours from 07:00Z. Until the rollout happens the hand-relocation stopgap is still required, and it buys ~36h per application.

⚠ Worth recording: the fleet is already split — the three worker Deployments run digest 42145c01… (v5.124.0) while the writer runs v5.129.0. Rolling the writer alone does not widen that.

The rollout itself was NOT performed — it is a separate owner decision. Blast radius is one pod; rollback is one kubectl set image back to 1c7d5387….

A prod outage, found and fixed. Every POST /api/execute returned 500:

EHDB command publish: EHDB publish failed: ehdb publish connection closed before ack

/data/cmdbus was 100% full, 0 bytes free. The actual command data was 71.8 MB; the manifest was 19.4 GB. ehdb-l0 writes a full manifest snapshot per version, nothing ever reads one (LATEST is the only manifest the engine loads), and nothing pruned one — so snapshot size grows with part count while snapshot count grows with write count. 6,770 snapshots, 75 KB → 5.0 MB each. Quadratic.

⚠⚠ The reason it took hours: the writer looked perfectly healthy. serve_ingest answered append_batch failure with a bare return — error discarded, socket dropped, no log, no metric. Pod Ready, restarts=0, all nine listeners bound, 0 ERROR and 0 WARN lines. Diagnosis burned rounds ruling out DNS, endpoints, selector drift, pod-IP staleness and image mismatch, all of which checked out fine, because the one component that knew was silent. A full volume and a serde-incompatible record produced byte-identical symptoms.

⚠ I first told the owner my writer restart caused it. It did not — the manifest growth long predates it. Read the timeline off part/manifest mtimes and controllerrevisions, not off "what did I just touch".

Mitigation (during the incident): 6,747 superseded snapshots relocated, not deleted, to /data/eventbus/cmdbus-manifest-archive-20260901. 100% → 1%.

Then a second problem, which the clean-slate truncate caused. Dispatch worked but the system pool sat at lag 78–86 and never drained; a burst degraded to 37/60 with 23 timeouts at p50 1997 ms. 53 pre-truncate executions were in an infinite re-drive loop — WAL chain incomplete; returning no-op, server reconcile will re-drive — with freshly issued command ids each cycle. spawn_orchestrator_reconciler polls orch_cache.active_executions() every 8s with no attempt cap, and OrchStateCache is an unbounded map whose only removal is evict() on terminal. Its own doc comment carries the broken assumption: "progress is guaranteed" — true only while the missing events are still coming. Fixed by a server restart after verifying nothing rehydrates the cache from the 4,647 surviving noetl.execution rows. Lag 82 → 0; next burst 30/30 at p50 199 ms.

Equivalence on the clean slate — 30 executions, prod v3.99.2:

Data equivalence is 100%. All 22 divergent executions diverge at one path, /steps/loop_test/iteration_command_ids/N, and a multiset compare proves all 22 are pure permutations — same ids, reordered, genuine divergence zero. The fold digests an order-sensitive array whose order is not deterministic across stores (#314). Until canonicalised the oracle cannot gate a serve-flip: it reports ~73% divergence on identical data, and a real divergence would be lost in that noise.

⚠ I nearly reported 0% — the real keys are /postgres/* and /comparand/*, and input_event_diffs_post_normalisation (empty) is distinct from the raw list. Dump the key structure before computing a rate off a nested response.

Landed

• ehdb#346 — bounded manifest retention. L0Config::manifest_retain (default LATEST + 32). Growth goes quadratic → linear in part count. ⚠⚠ A mutation disproved my own design note: I documented the O(1) per-write delete as what bounds growth and the sweep as "only for backlogs". Removing the sweep leaves 6 snapshots instead of 4 after 40 writes, with stragglers arbitrarily old (v6, v49 among v77–v80) — two producers write versions, so a file can land after the delete meant to evict it looked. The sweep is the guarantee.
• ehdb#347 — tracing::warn! + ingest_append_failed / ingest_decode_failed at both silent return sites, counted separately because the remedies are unrelated. Logs throttle 1-in-256; counters never throttle, with a test.
• ops#294 — three alert policies declared (not applied: the metrics do not exist in GCM until worker#304 ships, and applying over absent series is the same "absent looks healthy" failure this whole thread is about).

Held at the merge gate (each fires a publish)

• worker#304 — ehdb pin 229ab6a → 0980183 + four series rendered unconditionally so they appear at zero. CI green, 736 tests pass.
• server#386 — #264: the parity endpoint stops writing the counters its own alert reads. Also gates the control counter and the ratio-feeding outcome counter, and the response now says "metrics_recorded": false. 968 tests pass.

Opened: #314 fold-digest ordering, #315 unbounded reconcile re-drive, #316 projection-parity has no sampler — its divergence series only moves when a human queries it, so a real divergence there is invisible today.

Method notes worth keeping

• ⚠⚠ A mutation that fails to apply reports a false green. One LATEST-protection mutation hit a Python AssertionError on a wrong indent assumption and the suite returned "5 passed" — the unmutated result. Assert the mutation landed in the source before trusting what the tests say.
• ⚠ A sanity control caught a broken command: a repo-wide grep returned 0 for a symbol I knew existed (zsh glob-expanding --include), which would otherwise have read as "the codebase is clean".
• ⚠ Election/fencing is not the gap it looks like. ShardElection has no call sites — but that is already deliberate and documented, and made legible on the scrape as ehdb_election_active 0. Fencing is wired in the worker. My first grep missed it by being scoped to the ehdb repo alone.

Deployed. Prod is on server v3.99.2. Three fixes landed and one was gated back off:

• worker_id / context reader asymmetry (server#380) — the tier read worker_id only as a JSON string and filtered a jsonb null context that the Postgres side kept. Fixed symmetrically.
• created_at producer unification (server#384) — the column is microseconds, a DateTime<Utc> is nanoseconds, and the two writers reduced differently: sqlx's binary bind TRUNCATES, the SQL text cast in project_events ROUNDS. Now reduced once at construction, so both writers and the mirror carry the identical value and the fold rule is a no-op. ⚠⚠ This reversed a code comment that forbade reversing it — and both prior measurements were right; the question simply had two answers.
• ⚠ The sink mirror is gated OFF (NOETL_EHDB_SINK_MIRROR, server#385). server#383 closed #307's missing-event class and introduced the opposite defect: on one execution the tier held 45 records against 30 authoritative — 11 duplicates plus 4 orphans. Mechanism not pinned; #313 tracks it. Post-gate executions verify clean (auth == tier == matched, zero divergences). A tier holding more than the system of record is the worse failure, so the missing-event fix was traded away deliberately.

gs://shastaratech-noetl-prod-eventlog-archive-20260831 — UBLA locked, public access prevention enforced, 0 public bindings. 6.95 GB dump carrying 1,020,651 event rows across all partitions, catalog 1,469 (cross-checks live exactly), plus the three MD5-verified tier stores.

⚠ The first export reported DONE with no error and contained ZERO rows — 1,974 bytes. noetl.event is PARTITION BY RANGE, so --table=noetl.event dumped the parent definition and none of the data. Trusting the exit status would have deleted a million rows against an empty archive.

The TRUNCATE is owner-run and not executed — there is no SQL write path from an agent session (/api/postgres/execute is disabled, psql has no password, the admin surface offers only report/rehearse). SQL, FK analysis and the projection vs projection_snapshot hazard are in playbooks/event-history-archive/.

95.6% of the 1,020,651 events predate the tier's ~2026-08-13 enablement. Within its own window the tier holds 47,060 against 45,109 authoritative — 104%, a surplus matching #313, not a shortfall. Retention/GC and selective-mirroring were both ruled out with evidence.

⚠ Three magnitudes were reported wrong before this landed — "19 days", "78 days" and "5,001,000 events" — each from measuring a copy (the tier) or a broken paginated count whose cursor never advanced. The reliable number came from the pg_dump output, i.e. the source.

noetl 4.23.1 and noetl-executor 0.9.0 published to crates.io — the executor needed a version bump because cli#82 added a json-schema feature that the published 0.8.0 did not have, and the workflow's skip-guard therefore never republished it. ⚠ v4.23.0 is tagged but never published and never will be; the registry went 4.22.0 → 4.23.1.

server#382 widened the #263 insert guard to walk src/**/*.rs at test time rather than one file, and server#379 brought the platform DDL into the server repo behind a drift check. The SAFE-INTERNAL naming pass landed as worker#303 (42 symbols) with the env-var tier retracted to zero, and two dead GCPToken* types were deleted after a cross-repo reachability check.

Landed. The legacy Python platform is retired: noetl#697 (the macOS fork crash fix), #698 (stop building the ghcr.io/noetl/noetl image) and #699 (523 files, 132,650 deletions) are all merged. What remains of noetl/noetl is the thing it was supposed to be — the maturin/PyO3 CLI wrapper (packaging/pypi, PyPI noetl 5.0.0) plus schema_ddl.sql, which the Rust deploy still provisions from. The wheel was rebuilt from merged main and installs clean: noetl/ntl --version -> 4.21.0, import noetl -> 5.0.0.

⚠ Deleting the source did not close the DR path — that was the surprise. ghcr.io/noetl/noetl:v2.8.9 and :latest are still pullable (verified, HTTP 200), and the fresh-stack playbook deploys a published image; it does not build one. So ops#290 was the thing that actually closed it, with explicit refusals rather than silent repointing — the real deploy_noetl step body was extracted and executed, exit 1. ci/manifests stayed byte-identical, so nothing production applies changed.

A machine-readable playbook schema exists again (cli#82): noetl schema [--output], draft 2020-12, generated from a real Rust document model in noetl-executor — 22 new types covering the policy/output/loop surface that had no Rust type anywhere, while policy: appears in 61 of 202 real playbooks. 279/279 real playbooks validate; a malformed one fails 11 of 12 mutations. A staleness guard regenerates and diffs the artifact, so it cannot silently rot. ⚠ Step.next is documented, not constrained — the legacy shorthand forms still parse, so a malformed next: is not caught.

The D1 durability window is observed on production, not asserted. Idle it reads 0.000 forever. Under a bounded 180-execution burst it sawtooths and resets, peaking at 5.251s against SEAL_MAX_AGE_MS=5000. ⚠ The bound is ~5s plus one sweep interval, not a hard 5.000s ceiling — alert above it, not at it. Catalog verify agreed 180/180. No writer/server/pgbouncer restarts; the 12 client timeouts were KEDA scaling the user pool 2 -> ~20 and cold-starting.

Two content-level fold defects were found and merged, both inert on prod:

• The tier reader disagreed with the Postgres reader on the same value — worker_id serialised as a JSON number read as None, and a jsonb null context that only one side filtered. Fixed symmetrically.
• ⚠⚠ The microsecond normalisation was reversed, against a comment that forbade reversing it. That comment cited one execution showing truncation and said "re-run the diff endpoint before changing this". It was re-run: across 11 executions / 263 events, 9 round, 1 truncates, 1 mixed. Both observations are correct — noetl.event.created_at is written by more than one path and they disagree. Rounding is chosen because it is right for 9 of 11, and the minority execution is kept as a live assertion. The earlier sample could not distinguish the rules because every event it examined had a remainder below 500 ns, where the two agree.

⚠⚠ The real blocker is a mirror gap, not the fold. ~60% of disagreements are missing events — Postgres n=30, tier n=29 — pinned to event 352701148234289152 on execution 352701018137436160, with pending=0 and unmirrored_by_design=0, i.e. a genuine drop. Cause: three noetl.event INSERT sites never mirror, and one of them, services::internal::project_events (the projector sink), had written 5,088 events on prod with none reaching the tier. The #263 guard could not see it because it counts inserts only in events.rs.

The durable fix landed (server#382): the guard now walks src/**/*.rs at test time, so a new file is caught, and every insert site must be registered as mirroring, mirrored-by-a-named-caller, by-design, or pending-fix — with no default. UNMIRRORED_BY_DESIGN is deliberately empty. The actual mirror (server#383) is staged and unmerged: it changes runtime behaviour on a hot path, so it wants a human awake. Its deploy runbook is on the PR.

ops#292 finished the deprecated-verb sweep in ci/bootstrap — ⚠ scope widened deliberately, because a verb-only substitution would have shipped commands that still do not run: noetl run takes no trailing --host/--port (they are global flags before the subcommand) and the adjacent register lines were broken the same way.

⚠ noetl/cli has a v4.23.0 tag with no publish. semantic-release tagged it on the cli#82 merge and release-cli (which runs cargo publish) failed; crates.io is still on 4.22.0. Not retried — left for the owner.

Landed. The EHDB #324 remediation set: five inert PRs (ehdb#333–#337) closing F1–F5, plus worker#291/#292/#293 putting the instrument on prod (v5.125.0 → v5.125.1 → v5.125.2).

⭐ The finding, measured on prod. First reading of ehdb_l0_unreplicated_age_seconds on the command-bus engine: 387 → 404 → 451 → 497 s, monotonic, 4 acknowledged records on one disk, replicated_lag_seconds_count 0 — nothing had ever reached the substrate. At the same instant ehdb_feed_total_lag 0 and ehdb_l0_out_of_order_appends 0 reported perfect health. F3 stopped being an argument from reading should_seal() and became a number.

⚠⚠ Two of the three findings were in my own work, and both came from deploying rather than reading.

1. The window shipped on the wrong log. The writer hosts two L0 engines — /data/cmdbus (:9102) and /data/eventbus (:9106) — and v5.125.0 instrumented the command bus while the tier that is primary on prod is the event one. Both render ehdb_l0_* and share the same helpers, so a grep finds one wiring and stops. Fixed in worker#292.
2. Crash-recovered records were invisible. The tracker was seeded only by append_record, and recovery does not go through it — so after an unclean exit a shard reported 0 pending. The instrument was quietest at the moment of greatest risk. Fixed in ehdb#338 + worker#293. Notable: PartWriter already restarted its age clock on recovery, so the two halves had drifted and one had to be wrong.

⚠ A self-correction worth recording: my first probe of :9106 got Connection refused and I nearly filed that as the defect. It binds ~37 s after start, after the three group resumes — a timing artifact. The real finding was the missing families.

Everything ships inert. Fencing counts and refuses nothing; the election issues tokens without being authoritative; the age-seal trigger is default-off; no prod storage changed.

🔴 Four gates remain, owner-gated, three touching the live writer path on an already-primary tier — playbooks/324-four-gates/. Two ordering hazards recorded: enforce-before-election is an outage, not a degradation, and G4 would fail on prod as it stands (the tier dir is nested inside the writer dir on one PVC).

Also: playbooks/261-durability-soak/ — ⚠⚠ a saturating soak cannot detect the defect it is run to measure, so the design mandates a deliberately quiet arm; ehdb wiki invariants refreshed (the page claimed "detected by nothing" for two invariants that now have detectors).

server v3.94.0 · worker v5.124.0 · cli v4.22.0 · gateway guards (ai-meta pointer bdf2fda1). Everything inert by default.

#311 — resolved, and then proved. pgbouncer now seeds its userlist from Secret Manager via a CSI mount before the image entrypoint runs, so the stale password in DATABASE_URLS is never consulted. Hours later Autopilot rescheduled the pod again — the exact trigger of the 12-hour outage — and the new pod re-seeded "noetl" → 44 chars with 0 auth failures and 0 workload restarts. The hourly cleanup CronJob's 18:00Z run also succeeded, confirming the earlier Error pods were pre-recovery residue.

Catalog step 2 + 3. noetl.catalog was never event-sourced — register is a direct INSERT — so the relation had nothing to fold. Registration now emits into StoreTier::Catalog, deliberately not the event log: a catalog record has no noetl.event row, so appending one there would make the cross-store parity comparator report extra_event, which pages.

The backfill, and the trap it avoids. Re-posting the existing catalog through register would run MAX(version)+1 and create a second version of all 2,528 entries — describing the catalog by doubling it. POST /api/catalog-log/backfill emits each row's existing version and never touches noetl.catalog. Coverage went 4 → 2,528 / 2,528, fold_missing 0, 2,528 digests compared with 0 mismatched, and a re-run emits nothing.

Three truncation defects the gate caught, each reporting a wrong answer as a smaller one: the worker's scan hardcoded limit=100 so a fold over 2,528 records folded a 100-record prefix (and the server's own cap-check could not see it — a cap the requester cannot observe is worse than a small one); a whole-log scan cannot work at all against the 1 MiB frame cap on a 6,553,796-byte log, so the fold pages adaptively because one entry can be 267 KB; and verify_endpoint did its own unpaged read, becoming a verifier that stopped working at exactly the coverage it exists to confirm.

CLI + gateway. noetl catalog load <paths…> bulk-loads a tree in batches with a per-file outcome and refuses credentials rather than registering secret material as catalog entries. The gateway needed no routing change — the proxy is generic /noetl/{*path} → /api/{path} — so the property is now tested rather than assumed, including that the proxy stays auth-gated. ⚠ The first mutation run there proved nothing: noetl-gateway is binary-only, so cargo test --lib runs no tests and the mutations appeared to pass.

server v3.92.0 (server#369, ai-meta pointer 938d92d5). Option 2 of #307: the recovery fold now falls back to the durable EHDB event-log tier when the worker's in-memory spine cannot answer.

Why it read as healthy. Recovery folded only from the state-builder index, which holds in-flight executions and evicts them on completion — while the comparator by its nature asks about completed ones. Coverage was ~0 by construction, not by failure, and nothing said so except a zero, which is indistinguishable from quiet.

The blocker that made it more than a source swap. mirror_payload never wrote context, and WorkflowState::apply_event reads it — it is where workload, path and version come from on the start event. A tier-sourced fold was therefore guaranteed to digest-differ from the Postgres fold. Simply repointing recovery would have reported divergence on every execution. Records written before the fix now refuse with their own reason (payload_too_old) rather than folding short and being scored as diverged — a false alarm that would have fired hardest immediately after shipping, on the one signal the comparator exists to make trustworthy.

Kind gate — controls before and after, one variable at a time. Baseline (3.91.1) reproduced spine_refused; the same new binary in spine mode refused identically, so the binary was not the variable; verify folded a completed execution from the tier while the verdict and served_tier stayed put; the equivalence sweep read 4 agreed / 0 disagreed with 9 pre-v2 records refusing. Two positive controls proved the gate can fail: an injected phantom event gave equivalent=false, and an injection at a lower event_id (version unchanged, digest moved) gave digest_mismatch, is_fault=true. Plus eight unit mutations, each restored.

Prod — inert default proven first (v3.92.0 with the mode var unset: mode gauge spine=1, coverage all-zero, e2e COMPLETED 13 events), then verify. Coverage moved off zero on completed executions; equivalence 5 agreed / 0 disagreed (10 pre-v2 refusing as payload_too_old, filling in as new executions land); served_tier 0; READ_SOURCE=wal, event-log primary and mirror source all unchanged; restarts 0; cross-store parity resumed after the roll (65 events, 5 executions, all match, 0 divergence, all 10 self-test controls firing).

Two things deliberately not claimed. served_tier could not move in kind — projection_read_total is 0 across all outcomes there because load_latest is never called by those playbooks, so the read path is unexercised regardless of the change. And the in-path verdict machinery cannot see tier-vs-Postgres divergence: by design it never consults Postgres, and wal_projection_state materialises then refolds, so it re-materialises from a divergent tier and agrees. Two other layers do catch it, both demonstrated: the continuous cross-store parity comparator and the equivalence sweep. That is why the flip to tier serving is held as a separate owner decision.

Also fixed in passing: the recovery tier fold ran on a 20s timeout inherited from the diagnostic endpoints (now 2s); the tier is append-only behind a retrying drain and apply_event is not idempotent, so duplicate records are now deduped by event_id — Postgres cannot produce that, which is why it had to be handled on the way out of the tier rather than assumed away; fold_from_wal_spine removed as now-uncalled; the projection record's mirror_source names the source that actually answered instead of a constant.

NOETL_ENCRYPTION_KEY, POSTGRES_PASSWORD and NOETL_INTERNAL_API_TOKEN now reach prod only as CSI-projected Secret Manager files. All four secretKeyRef bindings removed across noetl-server-rust and both system pools. server v3.90.0, worker v5.123.0; ops#275, #276, #277.

Two things made it safe rather than lucky:

• Rolled inert first. Each binary went out with no _FILE variable set and reported source="env" on prod. That is the control: it shows the metric can report either value, so the later source="file" is evidence rather than a pinned constant.
• Digest-checked before removing the fallback. Every mounted file was compared by sha256 against the secretKeyRef value it replaced, computed inside the pod. All five matched — so removing the env could not change the effective value, and the removal was a change of plumbing, not of content.

Verified functionally, not by provenance alone: the Postgres pool is created after the three source="file" boot lines, credential decryption returns structure on a two-minute-old pod (fetched fresh, not a warm cache), and a real execution reached COMPLETED. The three variables are now absent from the container environment entirely, which is the actual proof the file is the only source.

The durable lesson: a no-op diff and a completeness check answer different questions. The diff asks "would an apply change anything"; it cannot see an object no manifest declares, an annotation absent from last-applied, or a declared object that was never applied. Verification is now both checks.

Correcting an in-session claim: the internal token is enforced — /api/internal/outbox/pending-count gives 403/403/200 on prod. The mechanism is a per-handler extractor, so it is opt-in, and the handlers that forgot are indistinguishable from ones that decided not to. /api/credentials/{id}?include_data=true answers 200 with decrypted data and no Authorization header, reachable from the user worker pool, with zero NetworkPolicies. Two /api/internal/* routes are open for the same reason — which makes the prefix a representation of protection that nothing enforces.

server v3.91.0 adds a router-level Layer (which a handler cannot forget) across 14 privileged groups, in shadow — it rejects nothing, recording what it would reject. Mutation-proven: removing the credential_routes layer makes the structural guard fail.

With READ_SOURCE=wal live: every refold refused, served_tier=0, and the projection comparator has compared 0 events against 8,476 on the eventlog tier.

The reason is spine_incomplete, which reads as transient — and is not: executions nine hours old refuse identically to ones seconds old. A positive control found the mechanism:

t+4s   complete=True   count=6    indexed_executions=1
t+8s   complete=False  count=0    indexed_executions=0

The spine walk works; the window closes when the execution finishes. state_spine_handler reads the in-memory index and never calls the bounded rehydrate path the drive uses — though REHYDRATE_ON_MISS=true and the WAL holds 38,611 events. 2,281 incomplete builds against 7 rehydrate attempts. So the comparator asks a structure that holds in-flight executions for completed ones: coverage is ~0 by construction, not by failure. Owner decision; three options in the issue.

server#362 #363 #364 · worker#286 · ops#275 #276 #277 #278 #279 #280 · draft ops#281 · ai-meta pointer bumps · new issues #304 #305 #306 #307.

Three issues landed on prod, each proved by real executions. travel#107/#108/#109 → itinerary-planner v95 → v97, ai-meta@865e70d.

#293 — transfers quote the picked hotel. No extra provider call was needed: HotelBeds already returns per-hotel lat/lng and the card already kept them; they just were not carried across the pick. ⚠⚠ The bug only execution found — slot state round-trips through Firestore and stringifies the numbers, so isinstance(x, (int, float)) rejected a good location on every turn after the pick while picked_hotel_location sat in state looking correct. Coordinates are coerced, not type-checked. Proof: to_code 48.8566,2.3522 (city centre) → 48.87389793495718,2.347064647742627 ("priced to Peyris Opera").

#292 — no cross-currency cap. No FX source exists in the catalog, and inventing a rate would be the same defect class as inventing inventory, so the cap is not applied when currencies differ and the turn says so. budget.currency_explicit mirrors dates_explicit_range: a bare "under 300" still filters.

#286 — provider hops inlined. A kind: playbook hop costs ~9s of orchestration regardless of work. hotels 9.3s→3.0s, activities ~9s→1.9s, transfers ~9s→1.7s; hotels turn span 24.9s→17.8s. ⚠⚠ Lifted from the registered catalog content, not repos/ops — the two had drifted (hotelbeds 47,295 chars live vs 37,945 in the repo), so the repo copy would have shipped stale provider code with every test green.

Validated the three together end to end, not just individually: hotels → pick → transfers still resolves to the hotel's coordinates after inlining.

New follow-up: #295 — the provider code now exists in two places (three counting repos/ops); which copy is authoritative is an owner decision, filed rather than left implicit.

Wiki: noetl-travel-wiki@10f539a.

Documentation only; no prod, playbook or behaviour change. Four new pages on the travel wiki, cross-linked into Home and the sidebar rather than duplicating the existing per-page detail:

Two things worth keeping from writing it up:

• ⚠ The travel wiki's deployment page listed a NATS deployment in a nats namespace. Checked against the prod cluster before "correcting" it — the namespace genuinely does not exist, so the line was stale and is now fixed. Verifying before editing is the point: the same instinct applied to the agent memory index two weeks ago would have caught #238's stale text.
• The anti-fabrication page documents the positive-control method explicitly, because it is the part most likely to be got wrong later: a nonsense city gates before the provider, so a clean run tests the gate, not the render branch.

Wiki: noetl-travel-wiki@e982b88.

Completed what #289 started. Landed travel#106 (ab430a9) + ai-meta@25f6d39. Live on prod: itinerary-planner v93.

The synthetic fallback was a house pattern, not a flight bug. Every surface invented inventory when its provider returned nothing, and reported COMPLETED:

⚠ They invented the inputs too. Hotels and activities defaulted dates to today+30 for four nights, pricing a stay nobody asked for. Activities are the subtlest defect found in either session: the tours returned are real, just real on the wrong dates — every field genuine, nothing reading as wrong.

⚠⚠ Transfers fabricated three inputs at once: the outbound date; the drop-off, using city-centre coordinates from a five-city hard-coded map even when the user said "to my hotel"; and outside those five cities the pickup, using a city code as an IATA airport code alongside an empty destination string. Now gated, with the city-centre drop-off disclosed rather than implied — a picked hotel stores only its id, so re-quoting against a real property is not yet possible.

⚠ Method note — proving an empty-provider branch. Asking for a nonsense city proves nothing: it gates at region and never reaches the provider, so a clean run is a claim about the gate, not the render branch. Proved instead with a positive control — the same code at a throwaway catalog path with the provider list forced empty. This is the reachability lens again: existence of the branch is what a naive test measures.

One apparent flights regression was not one — removing the fabricated return leg dropped the fare 86.11 → 38.37, so an "under 50 EUR" cap stopped being impossible. Re-verified with under 10 USD.

• Follow-up: #292 — the budget cap is compared across currencies with no conversion; it only looks correct because Duffel happens to return USD.
• ⚠ muno/playbooks/_probe-empty-results is still registered on prod; /api/catalog/delete returns 403 (protected since the soft-delete incident), so it needs an operator to archive. Nothing calls it.
• Wiki: noetl-travel-wiki@67241ba

Validated the owner-supplied Flights Training Use Cases spec (10 rules, scenarios F1–F10) against what the planner actually did, then implemented the MVP. Landed travel#105 (d259981)

• ai-meta@912d5da. Registered and verified live: itinerary-planner v88, flight-planner v4.

⚠⚠ The headline is a fabrication, not a gap. When Duffel returned nothing, show_flights rendered {"offer_id": "synthetic-offer-1", "price": {"total": "220.00"}} and the UI presented it as bookable inventory. The execution reported COMPLETED, the widget rendered, and nothing anywhere said the fare was invented. It also fabricated the inputs: a search missing origin, destination or date became NYC -> PAR departing today+30 (SFO -> MIA on the guided branch), so a user who never named a city got a plausible route back.

Implemented per the spec: required-input gating, platform-default origin with explicit override, three result groups (best value ≠ cheapest — lowest price-per-hour over the faster half), the duration→radius heuristic as a ranking hint only, budget as a post-filter that empties the set rather than silently dropping the constraint, stale-result invalidation, and lap-infant passenger mapping.

⚠ Two defects only execution found — a read of the playbook missed both.

1. A single date derives a checkout at +4 days (right for a hotel stay), and that derived value was sent as the flight return_date — pricing a round trip the user never asked for. On the same query: 86.11 → 38.37 USD. The durable shape is representation-drift: check_out_date was a copy with no provenance, holding two different facts ("the user stated a range" / "we derived an end date") in one field, so every reader downstream had to guess. Fixed by storing the provenance, not by teaching readers a heuristic.
2. "actually make it business class" carries no flight keyword, so the correction left the flight branch entirely and asked for travellers — abandoning the search being corrected. Only visible on a second turn of the same thread.

Proof is real executions on prod against live providers for F1–F9 plus a regression pass over places / hotels / activities / transfers — no unit tests. The corrected F9 turn is also the clearest evidence the ranking works: Cheapest 342.10 (1 stop) · Fastest 475.60 (nonstop 2h25m) · Best value 378.59 — neither of the other two.

Not faked: multi-city, baggage-aware filtering and fare-rule semantics are not expressible on Duffel's offer_requests (no price parameter, no baggage filter) and are recorded as roadmap.

• Coverage matrix: playbooks/travel-slm/flights-use-case-coverage.md
• Wiki: noetl-travel-wiki@94dae43
• Follow-ups: #290 (the same synthetic fallback still fabricates places / hotels / activities / transfers — flights were the only surface cleaned), #291 (flights roadmap).

⚠⚠ Correction, added after the fact and the most useful thing here: #238 was already CLOSED — on 2026-08-05, two weeks earlier. It was never open during this work. The "ZERO alerting on prod / 0 notificationChannels" text I acted on came from the agent memory index, which had carried #238's pre-fix description ever since and was never updated when the issue was fixed. I propagated that stale claim into ops#262 and two ai-meta#155 comments before checking the live project.

The drifted representation was the notes about the system, not the system — representation-drift.md applied to the very file used to decide what to work on. The work itself still stood up (the six async-mirror policies were genuinely missing, and all eight pre-existing policies were genuinely undeclared), but the framing was wrong for a day. Memory corrected; a topic file now records the verified state.

Delivery is PROVEN, not inferred. A temporary firing policy produced a real email at [email protected]; the policy was then deleted (404 on a direct get). That test was necessary rather than ceremonial: verificationStatus is absent from notificationChannels.get and not selectable, so an unverified channel and a working one are indistinguishable through every readable signal — both report enabled: true and attach to policies happily — and Cloud Monitoring does not log incidents to Cloud Logging, so there is nothing to grep. Attachment is not delivery.

Headline. ops#264 declares prod's Cloud Monitoring alerting as code and wires the six async-mirror alerts. Applied: 6 created, 8 updated, 14 declared, 0 without a channel.

⚠⚠ #238's premise was wrong, and I had repeated it. The issue says prod has "0 GMP rules, 0 alertPolicies, 0 notificationChannels". Checked rather than inherited: prod had 5 Rules objects, 8 alert policies, 2 enabled email channels, and the server was being scraped. I quoted the zero-channels claim in ops#262 and in two #155 comments before verifying it. The real defect was narrower and one layer down — which is the whole lesson of representation-drift.md: the issue text is a copy, and nothing forced it to stay true.

⚠⚠ The finding that outranks the rules: prod has TWO alerting paths and only one delivers. From inside a repository they are indistinguishable.

The GMP OperatorConfig references configSecret {name: alertmanager} in namespace gmp-public, and that namespace contains no secrets at all. Five applied Rules objects evaluate happily and their alerts go nowhere — including the ones added in ops#262 the same day, which were written as Rules on the assumption that applying them would make them page. They would not have. Anything that must reach a human belongs in alertpolicies.json until a receiver credential (SMTP / Slack / PagerDuty) is provisioned, which nobody has.

The undeclared-infrastructure half. The eight live policies and two channels existed only in the running project, in no repository — a rebuild would have silently lost prod's entire alerting. Same shape as #267, where the deployment manifests omit 84 live env vars. They are now exported and declared, applied by an idempotent script that matches on displayName and never deletes: a policy dropped from the JSON stays live and is reported, because silently removing a paging rule is worse than leaving a stale one.

⚠ Unresolved and honest about it: delivery is unconfirmed. verificationStatus is absent from the notificationChannels.get response and the field is not selectable, so the API cannot say whether the email channel is verified. An unverified channel accepts configuration and silently fails to deliver — #238 one layer deeper again. A temporary firing policy is live pending inbox confirmation; attachment is not delivery, and the README says so rather than assuming.

Also live today: server v3.83.1 confirmed on prod (7h, 0 restarts) with the async flags intact (async_enabled=1, window 30 s), match 173 / divergent 0, conservation 243 = 243, and the paging-counter fix verified by six in-flight probes leaving divergence_total unmoved at 0.

Headline. Both stages shipped. noetl-server v3.83.0 + user-pool noetl-worker-rust v5.120.0; async mirror ON with a 30 s comparator window. emit_mirror went from 78.6–87.6 ms/call to 0.1 ms/call — ~6.0 s of inline mirror per Muno turn down to 0.007 s. Median warm turn 16.9 s → 13.0 s (−22%).

⚠ Both numbers reported, not the flattering one. Removing 6.0 s of mirror bought ~3.9 s of wall clock, because not all 76 mirror calls sit on the turn's critical path. The mirror cost is gone; the turn-level win is smaller than the mirror's raw size.

The window was measured, not chosen. Async went on first with a deliberately wide 300 s window (both flags in one rollout, so there was never an instant with async armed and the window at 0). Real prod lag over 241 observations: mean 84 ms, p50 67 ms, p95 374 ms, p99 699 ms, max < 1 s. The window was then set to 30 s ≈ 43× p99 — deliberately below the sampler's 120 s settle, so the background sampler compares every execution exactly as before and the window only guards the on-demand endpoint. Verified on the flip: settled turns went pending_mirror → match 63/63/63, 0 held back.

Gates all clear. Conservation 241 enqueued = 241 drained, 0 pending, 0 depth; queue_full_inline 0; enqueued_after_wait 0 (backpressure never engaged at prod's load); controls 10/10 expected; served_primary 684 across replicas with every demote outcome 0; sampler match 14 / divergent 0.

Deployment scope was deliberate. The cmdbus-writer StatefulSet stayed on v5.119.1 — it carries the durable log, rolls as its own watched step, and the Option 1 runtime cache lives there; rolling it risks the known in-flight-step re-execution for no benefit here. System pools stayed on v5.118.2, not being in the mirror path. Option 1 verified holding and better than reported: emit_mirror 78.6–87.6 ms/call against 1141 ms/call pre-fix (yesterday's write-up said 110 ms).

⚠⚠ A defect found ON PROD within minutes of the cutover — #264 in a second place. Probing an in-flight execution through the on-demand endpoint returned pending_mirror (correct) and still left divergence_total{count}=1, {extra_event}=1 behind — the counter ops#257's EhdbCrossStoreDivergence pages on. So investigating a running execution inflates the number its own alert reads, while reporting no divergence. Two causes: compare_execution recorded report.kinds() unconditionally (an untaken comparison is not a finding), and the divergences were read skew — the authoritative page and the tier page are separate reads, so a still-emitting execution legitimately has a tier record the authoritative page never saw. Fixed in server#354 → v3.83.1: recording guarded on a Divergent verdict, and tier records above the authoritative page's maximum id excluded (sound because the log is append-only with snowflake ids). ⚠ The bound must come from the full page, not the comparable prefix — when the window holds a whole execution back the prefix is empty, which is exactly the observed case (auth=0 ehdb=1); written the wrong way first and caught by re-reading against the observation. Positive control included: a real extra_event below the page maximum must still report, so the fix is not "ignore anything unexplained".

Final prod flag state: NOETL_EHDB_EVENTLOG_MIRROR_ASYNC=true, NOETL_EHDB_CROSSSTORE_PARITY_LAG_TOLERANCE_SECS=30. Revert is one flag.

⚠ #238 is now the blocking follow-up: prod has zero notificationChannels, so every alert guarding this pages nobody. Fine while a human watches the metrics; it must be wired before this is left running unattended.

Headline. The event-log mirror ran inline on the event-write chokepoint, costing a measured 110 ms per call × 76 calls ≈ 8.4 s per Muno planner turn on prod — after ehdb#316 had already cut it from 85.6 s. It now goes through a bounded in-process queue drained by a single task (server#353 → v3.83.0). Both flags default off; prod is NOT rolled — the prod rollout was blocked on permission and awaits the owner's go.

Why it could not ship as just a queue. The cross-store comparator is the event-log tier's only correctness evidence, and it had no recency bound — correct while the mirror was synchronous, because an event was in the tier before emit_events returned. Async breaks that: an execution sampled while its newest events are still queued reports missing_event. That is #263 inverted — #263 was a tier reporting completeness it did not have. So the change is a queue and a lag window, together.

⚠ One correction to the scope as previously written. Checked rather than assumed: the comparator has exactly three consumers — two HTTP endpoints and the sampler — and emits metrics and logs only. There is no automatic code path from a cross-store divergence to primary_serve::decide, which reads parity_held from the worker's own per-append sequence check. The hazard is real; the mechanism is one step less direct than "the serve policy demotes".

The window is a latency concession, never a coverage one. A prefix cut at the first event newer than the window, excluded from both sides so a fast mirror is not scored as holding extras. The horizon is MIN(event_id) - 1 over the recent events, not MAX over the old ones — the two differ when event_id and created_at order disagree, and only the first cannot leave a still-queued event inside the compared set. An execution wholly inside the window scores pending_mirror, not match.

What gives it teeth: two controls on ONE fixture, driven twice with only the horizon moved. lag_within_window and lag_beyond_window. A comparator taught to ignore everything passes a "clean parity under tolerance" test perfectly; only the pair asserts a discrimination. Both run on every sampler tick, and both were verified 10/10 expected in the running pod.

Kind validation, on built images. Flag off: inert, every series pinned at 0, 61/61 match. Async on under a 30× burst: 3,270 enqueued + 390 enqueued_after_wait = 3,660 drained — exact conservation, zero loss, backpressure engaged and queue_full_inline 0, so the one reordering-capable rung was never needed. Settled: 68/68 executions match at 61/61/61 (4,148 events), byte-shape identical, event_id and global_sequence both strictly increasing. Worker serve path unaffected: parity_mismatch 0, primary_divergence 0 across all 3,660. SIGTERM with 174 events in flight: 25/25 match, tier count equal to authoritative count for every execution.

Forced real divergence, end to end on one execution. Relay killed mid-run so 61 events genuinely never reached the tier: at t+21 s (inside a 120 s window) pending_mirror, 61 held back, not match; at t+148 s divergent with missing_execution + missing_event naming all 61 ids. The window delays detection by exactly the window and then reports forever.

⚠⚠ The queue's own first implementation lost events, and every counter still read correctly. The backpressure path used tx.send(batch), which moves the batch into the future, so the timeout cancellation discarded it — 32 of 60 records never reached the relay, while queue_full_inline incremented by exactly the right amount, because the events were counted at the moment they were lost. tests/mirror_queue.rs phase 3 caught it; reserve() fixed it. That failure is also the mutation proof that the test has teeth.

⚠ The window is not a free parameter. Driven hard enough that mean lag hit 5.76 s against a 5 s window, the comparator did report divergence — 42 samples — and was right to: the tier really was behind by more than the declared bound. The window must be sized from measured mirror_lag_seconds, which is why the tolerance is published as a gauge and ops#262's alert compares against it rather than a constant.

Two pre-existing defects surfaced, both of the same family. The #263 mirror guard (the_chokepoint_mirrors_before_it_forks) has been failing on main since server#352 hoisted should_publish into a let for phase timing — and because no cargo test runs in CI on any Rust repo (#232), a red guard and a green one were the same observable. And append_batch — worker#281's entire premise, "async will produce multi-record batches" — was unfalsifiable from /metrics; worker#282 adds noetl_ehdb_tier_append_records_total{path}.

⚠ Three guards caught themselves before catching anything else. The batch-path counter counted its own search literals (4 != 3) — the same trap as the #263 INSERT counter and its doc comment. The batch metric's first version rendered inside the tier_service_up gate, making it invisible on exactly the workers that serve the mirror, while its own doc comment already claimed it rendered unconditionally; the second version broke the stronger render_lines()-must-be-empty invariant and had to become a pin at the handler's bind site.

Alerts. Six, plus a recording rule, in prod-GMP and kind-VMRule twins with byte-identical spec blocks. EhdbMirrorQueueStalled watches the pending gauge, not the lag histogram, because a stalled queue emits no observations at all and its histogram reads exactly like an idle healthy system. ops#263 adds promtool unit tests, both ways — a rule that has never fired is indistinguishable from one that cannot, and the negative cases (silent at tolerance 0, silent when merely busy) are the ones with teeth. ⚠ These alerts currently page nobody: prod has zero notificationChannels (#238).

Artefacts. server#353 → v3.83.0 (publish-ar green, in AR at sha256:17a281e8…); ops#262 merged; ops#263 + worker#282 open; server wiki deployment-spec (Rule 2a); ai-meta pointer #280. #155 added to the roadmap board (it was absent) and set In progress.

Headline. A 10-hop probe playbook put 79% of a turn in two command-dispatch gaps (~510ms p50), paid twice per hop because the orchestrator itself is dispatched as a worker command. Four hypotheses argued from code shape were all refuted by measurement — the 250ms claim poll (16–33µs), NATS round-trips (NATS is deleted), the writer's fsync posture (0.39ms p50), and the unmerged #205 group-commit branch, which describes this exact symptom but whose model predicts ~5ms at this load.

What landed.