Home - noetl/ai-meta GitHub Wiki

Last refreshed: 2026-10-10 (Claude session — ⭐⭐ B3 + D2 are LIVE ON PROD and proven with real data. Server v3.143.0 / ehdb v0.9.0, tick 60 s (not the shipped 15 s — chosen from prod's measured 314 appends/hour, where 15 s would carry ~1.3 records per object and write 5,760/day), cleanup default ON. A 32-minute sampler caught the entire chain: 188 unsealed records left the node in 3 batches, objects appeared under tail/ counted externally with gcloud, the age seal fired at ~900 s, and D2 then reclaimed all 6 objects (3 per replica × 2) as a contiguous run of durable parts covered them — retained 6→0, bucket 3→0, tail_failed=0 throughout. ⚠ The first ~400 s read appends=0; that was the window, not the code — measured independently, prod had no execution traffic at all. ⚠⚠ The D2 safety property is the design: the watermark stops at the first non-durable part, because max-over-durable skips a local-only part in the middle and would authorise deleting the only off-box copy of those events; both mutants are caught by that one test. ⚠⚠ And a defect I introduced in B3 and caught before enabling: the driver held the engine lock across the remote put, and that lock is taken by shadow_append on the live write path — p50 77 ms / p99 234 ms per colliding append, forever; ehdb v0.8.1 splits prepare / upload / commit. ✅ #457 CLOSED on measurement — /data 31% → 9%, local 1.2 G → 11.9 M — with the one goal item that was NOT done (⚠ 0 of 19 alert policies watch any volume metric) carved out as #466 rather than buried. ⛔⛔ The prod worker roll is BLOCKED by a broken release, not by its accepted risk, and my first fix was the wrong diagnosis. release-worker failed twice (v6.2.2, v6.2.3), each ~45 min in, on commits whose cargo test --locked was green: noetl-tools = "~4.0" floated to 4.0.6, which needs async-nats 0.47 against our 0.38 ⇒ two incompatible Context types. ⚠ Adding --locked did not fix it — cargo chef cook works from recipe.json and writes its own lock over the one copied in, so the cook stage resolved 4.0.6 anyway. The fix is an exact pin (=4.0.1), verified by building the real image (1,008 crates, 0 errors, one async-nats). ⚠ Also corrected: the pools already run v6.2.1, not v6.0.0 — I had read build_info off the writer, the only reachable :9090, and generalised it. C2+C3 HELD; election-before-enforce absolute. · Earlier today: B3 closes the last in-flight durability gap: the UNSEALED tail now leaves the node. A2/A3 made sealed history survive node loss; a part is local-only until it seals, so the window was the seal interval — B2 bounds it and cannot close it. Measured on prod while building: SEAL_MAX_AGE=900 with oldest_unsealed_age=719 and unreplicated_oldest_age=0, i.e. nothing waiting to upload and ~12 minutes of records living on one disk. ehdb v0.8.0 writes the pending records as write-once per-batch objects under tail/ — never parts/, or plan_retention could drop them — and cold_load_replicated replays them, deduping against the manifest's max_sequence because a part that seals after its tail objects makes them redundant by design. ⭐ Window: 900 s → the tick. Bounded, NOT zero — no remote ack gates the local append, so it is neither consensus nor a quorum, and the tick is the number to quote. ⚠ The RED control is the load-bearing test: with the flag off, 16 of 21 records come back, so the loss is real and every green is falsifiable; with it on, destroying the local root and cold-loading from the replica alone returns 21 of 21, and 13 of 13 through a real GCS emulator. ⚠⚠ A defect I introduced in v0.8.0 and caught before enabling anything: replicate_tail performed the remote put while holding the engine lock — and that lock is taken by shadow_append inside emit_events on the live write path, so every append colliding with a tick would have waited out the put (p50 77 ms, p99 234 ms, once per tick, forever). A durability feature becoming a write-path latency regression. ehdb's uploader thread had documented the rule the whole time ("the substrate writes happen OUTSIDE the lock so a slow store never blocks appends/reads"); v0.8.1 splits prepare (no I/O) → upload (free fn) → commit (no I/O), pinned by a test whose RED control restores the defect. ⚠⚠⚠ And a correction to my own shipped default: prod's measured rate is 314 appends/hour, so a 15 s tick carries ~1.3 records per object and writes 5,760 objects/day — the count is driven by the tick, not the data. 60 s is the better choice (still 15× better than 900 s, a quarter of the objects) and the 15 s default I shipped is likely mis-set. B3 is emulator-proven and READY but NOT enabled on prod: enable needs D1 (tick) and D2 (tail-object cleanup — a data-deletion policy on the event log), both owner calls, now with measured numbers on #460. D3 resolved as recommended (a failing tail put fails nothing). C2+C3 HELD; election-before-enforce absolute. · Earlier today: the off-box replica was armed on prod and the acceptance proof found the defect instead of rubber-stamping it. A2 reached a genuine second failure domain (domains=[local-device:66320, remote-gcs]; survives_node_loss 0→1, single_point_of_failure 1→0) and parts sealed from then on replicate — the age seal fired at t=934s of a 1853s window and the part plus manifest landed in the bucket, counted externally with gcloud. ⚠⚠ Then the proof refused: read back from the remote's own manifest, 40 parts listed and 39 at replica_count 1. Uploads are enqueued on seal and nowhere else, so arming a replica replicates nothing that already exists — and every durability gauge read healthy the whole time because they describe the declared replica set, not the parts. ⚠⚠⚠ Worse than an empty remote: the bucket holds a manifest naming all 40 parts and the bytes of one, so it reads as a complete store and is not a recoverable one — which makes the backfill a correctness prerequisite, not an optimization. Fixed rather than redefined: ehdb#400 → v0.6.0 backfill_under_replicated, reading bytes back out of a replica that holds them because durable_view nulls local_path, so a repair path that only knew fs::read would have compiled, run, reported success and copied nothing — the same defect one level down. Backfills also carry their own counters: a part sealed in an earlier process has no seal→durable interval, and folding it into upload_lag_micros_total would divide the real total by a larger denominator and deflate a mean a capacity decision reads. ⭐ The worker's v0.3.2→v0.5.2 pin bump had exactly one compile break across two minors — the new ShardLag.inflight — and refusing to default it to 0 was the whole point: render_snapshot publishes that field and high-lag-with-zero-inflight is specifically the stalled consumer reading, so a hardcoded zero makes a healthy bus publish an outage signal on every scrape (ehdb#402 → v0.7.0). ✅ #463 CLOSED — vendor/ removed (19,670 files, 572 MB) via server#528. ⚠ Three of my own mutants did not compile before one did, and a mutant that does not compile is not a caught mutant. ⚠⚠ Rule 0a drift found and fixed on this very page: the Active-umbrellas section was missing #457, #459, #460, #462 and #463 — every umbrella of the last two sessions, including the one the session was about. C2+C3 remain HELD; election-before-enforce absolute. · Earlier (2026-10-09): a retention + object-store archival tier for the event log, built end to end and proven against a real GCS emulator; the hot volume becomes a WINDOW rather than an accumulator (#459, closing #457). Audit first: four things already existed and were not rebuilt — the retention planner (plan_retention, drops whole parts below a sort-key floor, never splits a part), its executor (apply_retention), a GCS client live in prod (ObjectBackend/GcsBackend, ADC via Workload Identity, 21,569 objects written) and the archive bucket itself. ⚠⚠ And four that did not: no cloud DurableSubstrate (only LocalFs + InMemory, so nothing goes off-box and prod's "second replica" is the same PVC); apply_retention had ZERO callers in the server — built-and-dormant, the #332 pattern; no time- or completion-based floor; no per-execution layout, date index or config. ⭐⭐ The constraint that shaped everything: partition = shard_for_execution with shard_count=1, so a part holds many executions interleaved — an execution owns no part, execution_bloom cannot enumerate, and dropping an old part would destroy the early events of a still-running execution. Resolution: archive per-execution (exact), prune per-part (coarse), with a floor as the bridge. P1–P6 merged: config, archive to execution_id=<id>/, a by-date index that is one object per execution (GCS has no append, so a per-day manifest would be read-modify-write and lose entries under concurrency — 60 concurrent archives on one date all survive), prune-after-durable, retrieval by id and by date, and the metrics. ⭐ Proven on a real L0Engine: with a live execution unarchived the floor pins at sequence 2, dropped=0, 20 of 20 of its events survive; archive it and the floor rises, dropped=9, parts 9→0, 10,736 bytes reclaimed. ⭐ A real bug in my own floor was caught only because the test asserts on reclaimed bytes rather than on the call returning Ok — the all-archived branch used the highest FIRST sequence, so a fully-archived log reclaimed nothing while every call succeeded. ⚠⚠ Scope correction: the store this prunes — /data/ehdb-embedded, 2.4 GiB — is a SHADOW; verified with a control grep that its only readers are the shadow writer and the parity verifier, and GET /api/executions/{id} reads Postgres noetl.event, which is append-only and never purged. So pruning it cannot make API history unreadable, and wiring a fall-through there would have been an instrument that cannot see its own target. ⚠ Blocked on one owner action: the server SA has no IAM binding on the archive bucket, so archiving fails closed; the exact gcloud command is on #459. Flags default-off — nothing in prod deletes anything. · Later: P7/P8 audited and specced (#460, design only, nothing built). ⚠⚠⚠ PROD RUNS RF=1 AND EVERY DURABILITY SIGNAL READS HEALTHY — both "replicas" are on device 66320, yet replica_domain_violations=0: L0Engine::open wraps the substrate in a replica set of exactly one and the check is gated on len() >= 2, so the 0 was never evaluated. FailureDomain::Remote exists and is constructed in tests only, so survives_node_loss is false by construction everywhere. ⚠⚠ The exposure is the unsealed tail, and on the tier it is unbounded in time — verified in code: with_seal_max_age is called at exactly three sites and tier_store.rs calls it zero times, so the two buses get the 5 s bound and the store at risk gets 256 MiB and no clock. ⭐ Bounds stated rather than softened: EHDB is not etcd-with-sharding, it is a WAL with a lease (etcd has consensus and no shards; EHDB has shards and no consensus); P8 gives mutual exclusion, not consensus, and reduces write availability under partition; and by CALM there is no coordination-free linearizable cross-shard read, ever — a theorem, not a gap. ⭐ The #459 archive is not P7 (terminal+expired only, lagging, downstream of sealing) but it retires the dependency blocker, because DurableSubstrate is a sync trait written on the uploader's dedicated thread, so GcsSubstrate can live in the server and be injected. Measurement phases unblocked; the three semantics changes are owner-gated. · A1/B1/C1 now MEASURED (server#511): four always-computed replica gauges seeded pessimistically (an unevaluated set must not read as safe) with the alert keyed on survives_node_loss rather than a count; the tier's active catalog.jsonl sits 7.1 MiB unsealed for ~7 hours, so a byte-only trigger bounds the busy tiers and leaves the quiet ones exposed; and five durable-state mutation paths are enumerated, none wrapped. ⚠⚠ Two of my own framings corrected: the tier is not an L0 engine so with_seal_max_age does not apply (age-based segment sealing would be a new feature — stays specced), and I had checked the wrong binary for the fence — the writer links ehdb-reference and the fence is already a selectable decorator, so what is missing is the election, not the fence. ⚠⚠⚠ Which sharpens C3: with no election live, enforcing is an immediate outage, so C2 strictly precedes C3. · C1 guard landed (worker#348, zero src changes): a second store-construction site or a new unregistered append* now fails a test. ⚠⚠ Three further guards were written and deliberately not shipped — the plain bypass and the read-after-wrap are compiler-guaranteed (move semantics; a match data dependency), my mutants would not compile, and a mutant that does not compile is not a caught mutant. ⭐ The ehdb skew is drift, not a correctness risk (FORMAT_VERSION 1 at both pins, ehdb-service/src unchanged) — the cost is capability, which is why C2 needs a pin bump first. ⛔ P7/P8 is now blocked on user decisions (#460) + the IAM grant — no further safe autonomous work; prod is 4 releases behind (v3.131.0 live, v3.135.0 tagged) and one deploy delivers the archival tier, P5 retrieval and the A1/B1 instruments. · Later (same session) — THE ARCHIVAL TIER IS LIVE AND DRAINING ON PROD, and three of the claims earlier in this very line are now STALE. ⚠ Corrections first, because the dashboard contradicting reality is the thing Rule 0a exists for: the IAM grant is DONE, prod is not 4 releases behind (it runs v3.137.2, rolled by digest a3bb8087, rollback c46a032a, 1/1, 0 restarts, env 71 = baseline), and P7/P8 is no longer blocked on no-further-safe-work — A2 and B2 both shipped. ⭐⭐ Archival verified with real data: ARCHIVE_ENABLED=true, 1,472 of 6,251 archivable executions archived, every one also verified_durable and indexed, 0 failures, 4,431 real GCS puts. ⚠⚠ But the reclaim number is NOT measured and is not claimed — the floor correctly refuses while any archivable execution is unarchived, so enabling prune today would reclaim nothing; reclaim is gated on the drain (~4 h), not on code. ⭐ The GCS-put latency that #460 decision (i) was taken WITHOUT: p50 77 ms, p90 98 ms, p95 139 ms, p99 234 ms, n=300 in-region — ~19x a ~4 ms local fsync, which closes synchronous per-record remote append harder than the "tens of ms" estimate did. ⚠⚠ Four defects found by controls rather than reasoning: (1) compute_prune_floor returned NoFootprint when everything is archived — the exact steady state the drain produces — so the whole prune path would have reclaimed zero while every call returned Ok; caught only by a positive control asserting a fully-archived log DOES reclaim. (2) collect_candidates capped at 200x100 and returned Ok on a silently truncated list, and a floor from a partial population can sit above an execution never classified. (3) The put metric's "pin" was observe(0.0) on a histogram — a FABRICATED sample; in prod the only reading below 50 ms was that invented zero, on the very number the decision turns on. (4) My own B2 gauge understated its exposure: it stops at sealing, but replication is async so a sealed part still has a local-only window — and unreplicated_snapshot() has computed the whole durability window all along with zero server callers. ⭐ A2 is proved by moving bytes, not metadata: the log cold-loads from GCS alone (6 sealed parts, sequences 1-48, confirmed by listing the bucket outside the test; control bucket 0 objects) — a substrate that declared Remote while dropping every upload would pass the unit tests. ⚠ Two GCS buckets are ONE domain, and they do survive node loss, so the gauge that would catch a two-bucket plan is domains_distinct, not single_point_of_failure. ⚠⚠ B2 existed in ehdb and was unreachable from the server — no config, no caller outside ehdb's own tests; ehdb's own doc names the trap (a configured seal_max_age with no timer is "inert on exactly the shard it was added for"), so both halves now ship together, guarded including the spawn in main.rs. ⚠ Owner-gated: A3 needs a replica bucket + IAM (exact commands on #460); and PR #355 has 38 commits stranded on a design branch, including the pointer bumps for several prod rolls, now CONFLICTING — all four gitlink conflicts measured as "this branch is strictly ahead", STATUS.md needs a human call. ⚠⚠⚠ C2+C3 remain HELD; election-before-enforce is absolute. Releases v3.138.0-v3.140.1 are cut and not deployed.) Refresh cadence: every session that lands meaningful cross-repo work (per agents/rules/wiki-maintenance.md Rule 0a)

Standing direction (2026-06-04). Per memory entry, Python tiers are deprioritized. Forward Rust-only e2e work is tracked under [#54][i54] (Phase F R5). Python pieces stay deployable for backwards-compat on GKE but are NOT a target for new feature work.

Single pane of glass for the NoETL platform. Every active umbrella, every submodule, every release lands here so a single page shows what's in flight, what shipped, what's next.

Convention. This wiki is the cross-repo dashboard. Per-repo wikis (e.g. noetl/server wiki, noetl/ops wiki) document that repo's surface; this wiki documents the system of repos. See Wiki convention for the split.

Everything in this block was read from the live system or the repo in one pass. The Sessions log below is an append-only history where each entry was correct on its date; where it disagrees with this block, this block is current.

⚠⚠ CORRECTED 2026-10-07: there is NO mirror loss, and the shutdown drain already exists and works. I previously wrote that unavailable counted lost batches, "one per roll". It does not — it is AttemptOutcome::Retryable, a failed attempt that gets retried. The arithmetic over a 22-hour window closes exactly:

enqueued 11330 = drained 11330,  shutdown_abandoned 0,  queue loss 0
mirrored 11323 + recovered 7     = 11330   (every event reached the tier)
parity   match 1332 / divergent 7 = 99.48% agreement

flush_on_shutdown() (ehdb_eventlog_mirror_queue.rs:449) is called from main.rs:1464 on SIGTERM, and shutdown_abandoned is 0. ⚠ I also attributed unavailable=2 to rolls that happened before this pod started — a counter cannot record those. divergent = 7 equals recovered = 7, consistent with a comparator sampling during a retry window (a timing artifact, the #325 family), though counters cannot prove the sets identical. #343 — no code change; the residual is comparator tuning, which is owner-gated.

⚠ The chain gates were disarmed at 18:36Z on 2026-10-06 and re-armed at 19:23Z when the "is it blocking?" test returned NO. Net state is the original baseline. #410 is applied and CLOSED.

All 8 Rust repos gate clippy with -D warnings. All 8 pin the toolchain at 1.99.0. Only cli carries named allows (2: large_enum_variant, result_large_err, justified in #378).

So #374 (the || true inert gate) and #378 (the allow-list paydown, 23 → 2) both closed correctly, and there is no clippy-gate debt outstanding.

⚠⚠⚠ CORRECTION — this table said the opposite earlier on 2026-10-07, and I was wrong.

It claimed "3 of 8 gate clippy; 5 cannot fail (server, worker, cli, tools, gateway)". I measured the local submodule checkouts, which are 7–23 commits behind origin/main, so I read the pre-#374 state and published it as current.

⚠ The irony is the point: this happened inside the sweep whose purpose was killing stale assertions, in the same edit where I told readers to "read the version from the pod digest, never from this table" — while reading a workflow file from a stale tree. The lesson generalises past versions: measure a repo claim against origin/<default>, never against a local checkout, which is already recorded as a false-zero source.

⚠ A second, independent error in the first pass: the probe took head -1 of the clippy lines, which for catalog is the cargo clippy --version reporting step and carries no -D warnings — so even against a fresh tree it would have misread catalog as ungated. Two defects stacked in one three-line probe.

22 PRs merged, 0 open, 142 tests, AC3 holds at exactly 4 Dataset impls. /api/catalog/* is the only interface — no SQL surface of any kind. Its wiki repo does not exist yet: catalog.wiki.git returns Repository not found because GitHub creates it only after a first page is saved through the web UI. Content is staged at docs/wiki/Home.md with publish instructions; nothing claims the wiki is live.

Opened 2026-10-09, advanced 2026-10-10 — the durability frontier: prod ran RF=1 and every signal read healthy (#460). A2 (off-box replica) is now armed on prod against a genuine second failure domain — domains=[local-device:66320, remote-gcs], with replica_set_size 1→2, survives_node_loss 0→1 and single_point_of_failure 1→0 — and parts sealed from then on do replicate, proven over a 1853s window (stated because a window shorter than the ~900s seal period would have proved nothing) with the objects counted externally by gcloud. ⚠⚠ The acceptance proof then refused to pass, which is the point of having one: read back from the remote's own manifest, 40 parts listed and 39 at replica_count 1. Uploads are enqueued on seal and nowhere else, so arming a replica replicates nothing that already exists, and the healthy gauges describe the declared replica set rather than the parts — a claim about configuration being read as a claim about data. ⚠⚠⚠ And the result is worse than an empty remote: the bucket holds a manifest naming all 40 parts and the bytes of one, so it reads as a complete store and is not a recoverable one. Fixed rather than redefined: ehdb#400 → v0.6.0 backfill_under_replicated, whose own trap was that durable_view nulls local_path, so a repair path that only knew fs::read would have compiled, run, reported success and copied nothing — the same defect one level down. ⭐ A second defect fell out of the worker pin bump: ehdb#402 → v0.7.0, because ClaimCoordinator could not report in-flight depth and a literal 0 there does not publish "unknown" — it publishes the stalled-consumer reading, every scrape, on a healthy bus. Honest framing throughout: bounded-loss consistent prefix of sealed history, not "no data loss." ⚠ C2+C3 (election + fencing) remain HELD; election-before-enforce is absolute.

Opened 2026-10-09 — two durability gauges contradicted each other on prod (#462, RESOLVED). Neither lied: unrep_age = unsealed_age − δ where δ is a 1 ms capture skew between writer.append() and on_append, measured in 4 of 8 runs. ⚠ The valuable by-product was catching that the A3 divergence check I had drafted was invalid: unrep_age at a seal boundary is the part's lifetime, not upload latency, so it diverges identically for a local-only replica and would have "passed" with no remote at all. The A3 check is now defined on instruments that were validated (ehdb#399).

Opened 2026-10-09, CLOSED 2026-10-10 — vendor/ was 19,670 tracked files consumed by nothing and already stale for 57 of 344 crates (#463, landed via server#528). Removed: 7,331,526 deletions, 572 MB. Build, 1510 tests and clippy verified clean without it, and the repo's one --offline call (ci/stamp-version.sh:47) still succeeds. .gitignore carries the reasoning inline so it cannot silently return.

Opened 2026-10-09 — a retention + object-store archival tier, so the hot EHDB volume is a window rather than an accumulator (#459, closing #457). ⭐⭐ Reclaim is banked and measured, not estimated: 2.30 GiB growth-corrected — df_used 3,283,072 → 885,496 KiB, manifest parts 329 → 26, PVC 32.1% → 8.7%, 306 parts pruned. The constraint that shaped the design: partition = shard_for_execution with shard_count=1, so a part holds many executions interleaved and dropping one would destroy the early events of a still-running execution — hence archive per-execution (exact), prune per-part (coarse), with a floor as the bridge.

Opened 2026-10-07, advanced 2026-10-08 — the EHDB north star: a distributed registry and control plane (#455, see Umbrella: EHDB north-star registry): EHDB as "like etcd but truly distributed" — append log, catalog relations, topology/service registry with discovery including ephemeral runtime units, secret references (references, never material), and vectors. All internal EHDB, cleanly separate from the external business catalog. P1–P5 are now REACHED in prod (Server, Gateway, Worker, ephemeral Execution, and the secret-material guard); Ehdb is shipped and awaiting its first command dispatch. ehdb v0.5.0 is cut and live, both server pins moved, and the server has shipped v3.124.0 → v3.131.0. ⭐⭐ ehdb#391 CLOSED — Dataset::supersede_key lets a merge collapse superseded keys, taking a vector collection from 4,608 stored records / 65.02 ms to 500 / 7.13 ms, converging to exactly the live set. ⚠⚠ But only one of the three named consumers may opt in: RuntimeDataset is disqualified by watch_since — the op-log reader added in P2 — because one history reader disqualifies a dataset. ✅ ehdb is now releasable (it had no automation at all; every v0.4.x tag was hand-pushed), with GH006 and the bot-tag trap avoided by construction — and ⚠⚠ gated OFF behind EHDB_RELEASE_ENABLED, verified by merging it: the job ran on main and was skipped, tag still v0.4.5. ✅ RESOLVED — ehdb v0.5.0 was cut, both server pins moved, and auto-release is now ON. ⚠ Auto-release is not auto-deploy: a cut tag does not roll prod; every deploy still takes the full baseline / by-digest / rollback discipline. ⭐⭐ P6 reframed its own deliverable: top_k is an exact brute-force scan, so recall@k is 1.0 by construction and becomes falsifiable only once an ANN index exists — while the number that actually bounds the design is cost, which tracks op-log depth, not live points (500 points re-embedded 10x costs the same as 5,000 written once, within 2%). ⚠⚠ Compaction cannot fix it — merge combines parts, not keys, and the Dataset trait has no supersede hook at all, so the prerequisite is a key-level compaction primitive (ehdb#391), shared by the D8 registry folds and the catalog folds. ⚠ P7 (unsealed-tail replication) and P8 (writer election + fencing) are specced and deliberately not started — both change durability or safety semantics, and a lease without fencing is not safe under partition. ⚠ election.rs/fencing.rs live in ehdb-reference, which the server does not link: existence is not reachability. Two findings filed: ehdb#388 (lazy-ack O(n²)) and ehdb#389 (drain attribution + the #298 cap tradeoff). Still open on the acceptance list: A5, fold determinism has no engine-side enforcement point.

Opened 2026-10-06 — the catalog model on EHDB (#427, see Umbrella: Catalog model on EHDB): a generalized, event-sourced catalog of NoETL internal resources, playbooks first, in the new noetl/catalog repo. Storage is EHDB and only EHDB. ⭐ It finishes a named predecessor — catalog_log.rs already writes CatalogRecord in shadow and CatalogRelation is a fold nothing calls on a read path. P0 (spec + repo + 4-of-4 RED-proven CI) and P1 (model types, 30 tests) have landed; P2 is the four Dataset impls, where ⚠ the attribute and relation folds must not copy ProjectionStore's latest-wins shape. Spin-off: #429, the kind filter returning a partial set.

Umbrellas open in the ai-task queue: the chain-store repoint blocker (#357 — the store cannot tell a genuine chain root from a post-restart one; populator built + kind-proven, repoint ON HOLD), now ARMED ON PROD with the comparator at 100.00% over 772 comparisons after #360 fixed the two defects that failed the first ramp at 6.3% — a snapshot behind the store is staleness, not conflict, and coverage is monotonic; the #358 second-writer alarm moved into chain_for, which re-reads the log to tell the two apart, and two server tests that never ran are tracked as #361), request latency on POST /api/execute (#319 — the seconds decomposed into layers: the ~4 ms per-append fsync (P1, parked behind ehdb#322), four redundant catalog reads (fixed, server#394), and the mirror's inline fallback (fixed, server#395, v3.100.4 live). See Umbrella: Request latency), the system worker pool ceiling (#318 — a fixed 2-shard capacity with no autoscaler; the ClaimCoordinator route is built, the KEDA ScaledObject was deployed, regressed c=6 and was rolled back, and the autoscaler stays held — see Umbrella: System Pool Design), the EHDB projection tier (tier 2) (#265 — mirror the incumbent snapshot store and make it serve-capable; phase A proven in kind 2026-08-14, 9 arms / 0 failures, NOT flipped and NOT deployed. ⚠ the charter's noetl.projection is a dead table — 0 rows, no Rust writer — so the real target is noetl.projection_snapshot, which has exactly one INSERT site. ⚠⚠ a 13-event execution completes and gets no snapshot row at all, so the tier can only hold what the incumbent writes and the prod soak must measure coverage, not only divergence), the EHDB primary serve path (#257 — a writer-fronted tier service; wiring + reachability slices shipped and kind-gated, no tier is primary anywhere), EHDB-only IaC — declared, kind-validated, prod converge pending (#222 — automation/ehdb/ via ops#245; the post-NATS topology as noetl playbooks in local mode), the T3 events-bus migration (#212 — noetl.events.> off NATS; shadow LIVE on prod, publish + projection parity PROVEN, consume path merged but unproven), the Domain-Specific SLM platform RFC (#139 — build + continuously improve org SLMs via NoETL playbooks; travel#63 is the reference implementation), the Rust server parity-port umbrella ([#49][i49]), the distributed-OS program (#107), the event-WAL + derivable-storage model (#104), the decouple-result-data RFC (#115), the worker-driven orchestrate E2E coverage (#111), the Rust regression baseline migration (#98), the Python-era deploy legacy cleanup (#97), the provider-code duplication (#295 — after #286 the HotelBeds dispatch code exists in the planner AND the MCP playbooks, with repos/ops a stale third copy; which one wins is an owner decision), and the flights roadmap (#291 — multi-city, baggage-aware filtering and fare-rule semantics, none expressible on Duffel offer_requests).

Opened 2026-08-27/28 — from the IaC-completeness and secret-migration work: #303 the credential API answers 200 with decrypted data and no Authorization header, reachable from the user worker pool (not internet-exposed; a router-level gate now ships in shadow, rejecting nothing); #306 rbac.yaml declares an unapplied ClusterRole binding the worker SA to cluster-wide delete on namespaces and secrets — so the DR procedure itself is the escalation vector (scoped-down proposal in ops#281, left as a draft); #307 projection recovery coverage was ~0 by construction — the comparator runs against completed executions and the spine route only holds in-flight ones — fixed in server v3.92.0, prod live in verify (folds from the durable tier, digests compared, served_tier still 0; the flip to serve is a separate decision); #305 noetl-cmdbus-writer-1 is 0-replica debris with no IaC holding a bound 20Gi PVC; #304 the prod Postgres password wants rotating; #300 noetl-secret carries an orphan POSTGRES_PASSWORD key nothing binds.

Rule 0a asks that every open ai-task issue appear here, so that comparing this table against gh issue list --label ai-task is a real check rather than a partial one. These were tracked in their own issues but had no row on this page — listed 2026-08-28 after the mismatch was found by running exactly that comparison.

Reconciled 2026-10-02 by running that comparison again: 12 rows were closed issues still listed as open, and 19 open issues appeared nowhere in the live sections. Both fixed — the table is now 54 rows and matches gh issue list --label ai-task exactly (0 absent, 0 closed-listed-as-open). ⚠ Measuring this needs the Recently closed / Sessions log / Releases sections excluded and the prose links in Active umbrellas counted: a naive table-row grep reported 84 stale rows and 67 missing, and both numbers were wrong.

Reconciled again 2026-10-04: 60 rows, 0 closed-listed-as-open, 0 open issues absent — against 101 open ai-task issues. Two closed rows removed (#385, #211) and two open issues added (#380, #204). ⚠ The slice boundary bit again: cutting the table at the next ## heading swallows Recently closed (it is ###), which read as 139 rows, 81 closed. Bound it on a line-start ### Recently closed — ⚠ this very sentence put that string in backticks mid-line, so a plain str.index now truncates the slice to 1,485 bytes and the table reads empty. A note describing a trap can instantiate it. ⚠ And removing the #211 row made #204 absent — its only live citation was inside that row's title, so a deletion can silently orphan an unrelated issue. Re-run the comparison after editing, not before.

Three cutovers were refused this session rather than forced. Each is blocked on something an agent should not decide or cannot safely do:

Also open and deliberately unresolved: #246 — the measurement half shipped (worker v5.107.0), but whether a payload-level provider error should terminate the step is a semantic decision still to be made; and #169 — Auth0 JWT verification stays in shadow until one real login supplies the aud value.

Opened 2026-08-13 — #264: the EHDB parity endpoint writes crossstore_divergence_total, the counter its own paging alert reads, so investigating a divergence alert by querying executions inflates it with nothing in the response saying so.

Opened 2026-08 — standing platform gaps rather than umbrellas, listed here because each is something an operator would want to know before trusting a dashboard or a manifest: #238 (no alerting exists on prod — zero rule objects, zero notification channels, and almost nothing scraped), #242 (four metric recorders with no caller, plus the two production flags that do nothing), #240 (three MCP agents still leak raw provider bodies in prod — the source is merged, the catalog is not re-registered), #239 (git worktree on repos/noetl silently reports the wrong directory), #237, #236, #235, and #234 (the pre-migration project is a live dependency of 36 registered playbooks, not manifest drift). #241 (env vars absent from the deployment-spec pages) closed 2026-08-05 — all 225 are now documented.

#101 (orchestrator scaling — subsumed by the #115 RFC) and #95 (postgres timestamptz NaiveDateTime bug) both closed 2026-06-22. #100 (cursor/claim loop mode) closed 2026-06-15 — server v3.8.0 + tools v3.10.1; test_pft_flow_v2 all_passed:true on kind (see Recently closed). #99 (transfer-tool credential aliases) closed 2026-06-14 via tools#65 + worker#87 + e2e#58 (see Recently closed). The subscription / listener tool RFC (#90) closed 2026-06-12 with all 7 phases shipped + live-proven (see Recently closed); refinement follow-ups #91–#94

• tools#57 are tracked separately.

See the Repo Map page for the full submodule inventory. Quick view of the production repos and their current versions (2026-07-30 — prod runs on GKE Autopilot noetl-prod-autopilot, us-central1, project shastaratech-noetl-prod, migrated off noetl-demo-19700101 on 2026-07-26; since 2026-07-27 the command bus is EHDB, not NATS — NATS remains installed only as the rollback path; since 2026-07-30 the bus is also ~2.4× faster than NATS on server v3.58.3 / worker v5.81.3):

The 2026-06-04 session retired all Python deployments + their services + configmaps from the kind cluster (per the user directive "delete all legacy stuff"). Local validation runs against the Rust topology by default:

              ┌──────────────────┐
              │  Gateway         │  noetl/gateway  v3.2.0
              │  (Rust)          │  auth · SSE · subscriptions · shard routing
              └────────┬─────────┘
                       │ HTTPS
              ┌────────▼──────────────────────────────┐
              │  noetl-server-rust  (v2.19.7)         │
              │  catalog · execute · events ·         │
              │  /api/internal · DbPoolMap sharding   │
              │  orchestrator engine · SSE            │
              │  workbook resolution · pipeline parse │
              └────────┬──────────────────────────────┘
                       │
              ┌────────▼─────────┐
              │   NATS JetStream │  NOETL_COMMANDS stream
              │   + Postgres     │  noetl.event + noetl.command
              └────────┬─────────┘
                       │
        ┌──────────────┴──────────────┐
        ▼                             ▼
  ┌──────────────────┐         ┌──────────────────┐
  │ noetl-worker-    │         │ worker-system-   │
  │ rust v5.11.3     │         │ pool             │
  │ (shared pool)    │         │ (Rust, runs      │
  │                  │         │  system          │
  │ noetl-tools      │         │  playbooks)      │
  │ v2.18.1:         │         │                  │
  │ python · shell · │         │ Consumer:        │
  │ http · postgres  │         │  ..._pool_system │
  │ duckdb · rhai ·  │         │ Filter:          │
  │ task_sequence ·  │         │  noetl.commands. │
  │ playbook · noop  │         │  system.>        │
  └──────────────────┘         └──────────────────┘
   Consumer:                     [Outbox publisher +
    ..._pool_shared               projector migrated
   Filter:                        to system playbooks
    noetl.commands.               via Phase 2.a]
    shared.>

Retired this session (2026-06-04): noetl-server (Python deploy), noetl-worker (Python deploy), noetl-outbox-publisher (Python deploy), noetl-projector (Python statefulset), and the noetl/noetl-ext/noetl-projector/noetl-worker-metrics services + 4 legacy configmaps + noetl-worker SA. The kind cluster is the regression-test topology for Rust-only e2e.

[Status legend: ✅ = shipped + kind-validated]

Six interlocking gaps fixed in one iteration brought control_flow_workbook end-to-end on the Rust-only stack — exercising the complete v10 control-flow surface:

playbook YAML  ──►  noetl-server-rust orchestrator                 noetl-worker-rust
                                                                   + noetl-tools v2.18
─────────────────  ──────────────────────────────────────────────  ──────────────────
workload: {...}   ► #56 workload + input alias decode               PythonTool wrapper
tool.kind: python ► (existing dispatch)                             globals().update(args)
tool.kind:                                                          ► #17 capture
  workbook        ► #59 parser substitutes inline action            `result = {...}`
                                                                    global → data
tool: [{...}]     ► #57 ToolDefinition::Pipeline accepts both       ► #18 TaskSequenceTool
  (pipeline)        flat (name-as-field) + nested (label-as-key)     runtime
                                                                    
{{ step.field }}  ► #60 build_context exposes step data at top
  in next.arcs      level (not just steps.<name>); apply_event
                    captures call.done before command.completed
                    overwrites
                    
command.failed    ► #58 trigger_orchestrator on command.failed
                    + dedicated short-circuit in process_in_progress
                    emits playbook.failed terminal
                    
worker → server   ► #55 EventEmitRequest accepts i64 wire shape
  event emission    (was rejecting integer execution_id)

Validated end-to-end:

noetl exec tests/fixtures/playbooks/control_flow_workbook
→ playbook_started
→ start (python)
→ eval_flag (workbook→python; is_hot=true captured via marker)
→ hot_path (next.arc when="{{ eval_flag.is_hot == true }}" matched)
→ parallel hot_task_a + hot_task_b
→ playbook.completed ✅

Per the Rust-only direction, Python pieces stay only as:

1. Container payloads — runtime stays Rust; user code that wants Python ships in a container dispatched by the container tool kind (#43, in design).
2. Back-compat GKE deployments — existing Python pods on the production cluster aren't removed (yet) since GKE traffic still uses them; no new feature work goes there.

The kind cluster is the canary for the Rust-only topology. When the operator runs the validation rigs end-to-end against their live cluster, R5 cutover decision will move the production topology to match.

Chronological notes on what each session accomplished — see Sessions Log.

Most recent (top of log):

• 2026-10-08 — ⭐ The registry became discoverable, and five measurement gaps closed. ehdb #385 (P2–P5: discover(kind), watch_since, ephemeral units, SecretRef), #386, #387. Memory is O(parts) not O(records); ⚠⚠ a shared sequencer minted before the append lock reorders 34.2% of appends losing nothing (#362's mechanism in 20 ms); ⚠⚠ a part with 1 of 2 copies is reported durable and done; 530x for batching, and the #298 cap makes a large drain quadratic. Benches now run in CI. 1,188 tests green. → full entry
• 2026-10-02 (later) — 🔒 Closing out the CI arc — the gate had to be fixed twice, once by itself. ⚠⚠ CHECK 3 forbade correcting a CHECK 2 violation: moving a pointer off a side branch onto the tracked branch is never a fast-forward, so the gate reported a defect and refused the fix (#379). Private submodules removed → coverage 31 of 31, zero unverifiable (#377); #372 CLOSED; clippy now gates on server + worker — measuring first explained the || true (#374, sweep #378); docs + doctor PR gates merged. → full entry
• 2026-10-02 — 🔒 ai-meta had NO CI at all; it now has a gate proven to bite — and the gate's first sweep caught the gate. #371 + #373 merged. Two of three first-sweep findings were its OWN false alarms: it cried "dangling, unpushed, or wrong SHA" at two pointers whose repos are simply private. Coverage on main is 31 of 33, now printed. Real finding: #372. Fleet audit — ⚠⚠ clippy is || true on 5 Rust repos, so it cannot fail (#374); 6 repos with no PR trigger and wikis that cannot host Actions (#375). → full entry
• 2026-09-30 (later) — ⚙️ #362: the chain is ordered by following LINKS, not by event_id — a snowflake minted before insert means an event can commit into the MIDDLE of an id-ordered read. Prod armed on server v3.118.0 + ehdb v0.4.5. → full entry
• 2026-09-30 — ↩️ The chain-store ramp was ROLLED BACK: a third mechanism, and my verification window was too short — size a prod ramp by DURATION, not denominator. → full entry
• 2026-09-04 — ⭐ The tier deficit is 318 events, not the ~12k being quoted. Structural mirror lag measured at 0, surplus 0, 52/100 executions affected, 303 inside the #320 window. The ~12k was failed batch deliveries, not a completeness measurement. Repair endpoint (server#399, v3.101.0) merged inert — not run against prod. → full entry
• 2026-09-03 — ⭐ The p95 tail was the event-log mirror, not the publish retry — and my published diagnosis was wrong. server v3.100.4 live; inline fallback 24.6% → 8.3%. ⚠ the tail is not collapsed and the bistability is not gone. → full entry
• 2026-09-01 — 🔥 An 8-hour silent dispatch outage: the cmdbus manifest grew quadratically and filled the PVC while the writer reported Ready with zero logs. Resolved on prod via worker v5.129.1 — snapshots 172→32, manifest 871→169 MB. → full entry
• 2026-08-31 — 🐍 The Python platform is gone; #307's real blocker turns out to be a mirror gap. → full entry
• 2026-08-29 — 📐 #307 — the projection fold stops being unmeasurable; prod live in verify (server v3.92.0). The catalog gets a log, a relation, and full coverage. → full entry
• 2026-08-27/28 — 🔐 Secret Manager is the sole source. All four secretKeyRefs removed (server v3.90.0 / worker v5.123.0), each removal digest-checked against the file replacing it first, so the change was plumbing rather than content. Then the #267 sweep: the gateway manifest would have caused a login outage on apply, ServiceAccount/noetl-server-rust was declared nowhere, and the watchdog script existed only in the cluster. ⚠⚠ #307 projection recovery has ~zero coverage; ⚠⚠ #306 the DR apply would escalate the worker pool. Session details
• 2026-08-11 — ⏱️ A cooldown that gated belief instead of retry. EHDB tier is_reachable() read its own cached-negative deadline as a verdict, so five seconds after a tier service died — with nothing having contacted it — the tier decided it was authoritative again. Belief now lives in a flag only a successful op sets. worker v5.115.3, inert, not deployed.
• 2026-08-05 (part 9) — 📊 Observability arc closed. P1+P2 blind spots shut across server, worker and gateway; 38 alert rules proposed (none applied). Two new defects filed: an unbacked-off claim spin (#244) and amd64-only gateway images (#245). Session details
• 2026-08-05 (part 8) — 🔎 149 warn/error sites with no metric, triaged by consequence. P1+P2 closed; P3 (82) deliberately left as logs. The materializer's only data-losing outcome had no counter. Session details
• 2026-08-05 (part 7) — 📊 Absence is the default state of a metric. The prod gateway's /metrics served zero bytes; no binary carried a version; *_build_info now ships in all three. #242 fully dispositioned — 4 dead recorders → 1 deliberate hold. Session details
• 2026-08-03 — 🛡️ P0 prod discovery ran (read-only), and the EHDB-only IaC turned out to be wrong about prod six ways. state_builder defaulted to server while prod runs offserver on the server and both system pools — a converge would have turned a running subsystem off; the three claim_* PVC names matched no PVC in prod; a wrong user_pool_container is rejected, not ignored, so the converge would have aborted half-applied; the gateway lives in ns gateway, not noetl; and the writer's memory limit would have gone 4Gi → 2Gi. All six are now defaults carrying prod's measured value (ops#245 @ 5e28543, not merged). Two premises were inverted — prod runs all three materializer groups, so narrowing verify_groups would have failed on a live one. A seventh defect appeared only by running the adoption path on a kind rig rebuilt into prod's shape: kubectl apply's three-way merge keeps the old app: Service selector, so the writer Service resolves to zero endpoints behind a 1/1 Running pod with all nine faces up — a total silent outage cutting off server, both pools, gateway and KEDA. Kind: plan → converge (VERDICT: PASS, PVCs adopted verbatim) → clean idempotent no-op. ⚠️ Prod NOT converged; P1–P5 unrun. Session details
• 2026-08-02 — 🧪 First sustained-concurrency soak of the EHDB-only platform (local kind): PASS on every paired-evidence gate — 360 execs, published 6840 == projected 6840 == all three group cursors, lag 0, 0 dup / 0 publish errors / cursor_errors=0 / out_of_order_appends=0; 3 SSE subscribers saw an identical 7864 frames each; KV 880 ops / 0 mismatches. Three fixes in worker#211 (open, no release cut): H5 fail-loud bus source (#221), the #209 graceful seal (tip 7781 / lag 570 → reopened 7781, all 570 survived), face supervision. ⚠️ The hard-kill number could not be measured in kind — masked by consumption lag, so prod is more exposed, not less. ⚠️ The bus was never the bottleneck (shard lag 0 while the 8-slot pool saturated) → throughput/latency remain prod-only. Defects: ehdb#311 one bad connection silently kills the WAL face; NOETL_STATE_BUILDER=shadow silently means Off; no EHDB-only IaC exists at all. ops#241 reviewed, not merged (superseded).
• 2026-07-31 — 🟢 The EHDB-lag autoscaler is LIVE on prod — the T5-readiness sequence executed end to end; NATS untouched, T5 still the human's call. Merged + deployed: ehdb#303 (ehdb_feed_subject_lag{subject}, with the label set seeded at startup so a drained pool reports 0 rather than vanishing — a vanished series is a KEDA scaler error, not a scale-to-min) + ehdb#304 → ehdb e457249; server#291 publish retry 0.5 s → 10 s + #292 → v3.58.4; worker#197 + #198 (ehdb_l0_out_of_order_appends, closes #206) → v5.82.0 → v5.83.0; ops#242/#243/#244. Merging #303 and #304 surfaced a silent-failure seam: the pre-existing serve_metrics_with_resume renders lag through render_prometheus, which predates #303 and omits the per-subject family — a writer binding it would have served the resume gauges while dropping the autoscaler's trigger series, with nothing erroring; fixed with serve_snapshot_metrics_with_resume + a test on each side. Autoscaling proven on prod, on the per-pool metric: the user pool — which had had no autoscaling of any kind since 2026-07-26, because KEDA 2.15 deletes the HPA while paused is set — went 2 → 4 → 8 → 16 → 20 under 200 real executions and back to 2 on drain, every rescale attributed to s0-metric-api-ehdb_feed_subject_lag{subject="commands-shared-shard-0"} by name; 200/200 COMPLETED, 0 dup, 0 publish errors. The split earned its keep on the first scrape — shard lag 4 was entirely the system pool's (shared=0, system=2). #208 re-verified live under load: writer restarted mid-burst → 60/60 COMPLETED, 0 publish errors, claimers redialing unaided, the restart legible from one scrape (origin=persisted, clamped=false, replay_records=67 = the unacked tail, not a from-zero replay — precisely the distinction #304 was built to make). Soak 6 × 40, out_of_order_appends=0 throughout. ⚠️ Two corrections the plan had wrong. (1) The nats-jetstream trigger was not a harmless no-op: noetl_worker_rust_shared still holds the 149 messages frozen at the T4 cutover, KEDA takes MAX across triggers, and it drove the pool to 15 replicas within 40 s of unpausing (every one of those rescales attributed to the NATS metric). Removed in ops#244. At the scaler interface a stale queue is indistinguishable from a busy one — the same failure the per-subject split exists to prevent. (2) T5 would break the SPA today. NOETL_COMMANDS_RUST is genuinely idle (last message 2d23h) but noetl_events had traffic 3 minutes before the audit, with four live consumers — three worker materializers plus the gateway's SSE subscriptions. That is T3, still unchecked; the gateway fails soft but loses live SPA updates and async callbacks. Inventory: playbooks/210-t5-readiness/pre-t5-checklist.md. Images reached AR by crane copy from GHCR — Cloud Build's publish-ar fails on every release (build SA cannot stage source in the _cloudbuild bucket) → #211, needs a human IAM grant. ai-meta gitlinks ehdb→e457249, server→91a4a66, worker→4c83117, ops→6469482.
• 2026-07-30 (docs sweep) — 🧭 Issue + wiki reconciliation after the prod deploy. #205 and #208 closed prod-validated; #210 filed for the user pool having no autoscaling since 2026-07-26; #209 annotated with what the seal race actually cost in prod (a latency artefact, not loss — the crash path is the real exposure); #194 given a single consolidated T5 status board; #204–#210 added to the roadmap board. Dashboard, ecosystem map, releases and the ehdb / ops / server / worker wikis brought to server v3.58.3 / worker v5.81.3, and the VictoriaMetrics → Google Managed Prometheus correction applied. No code, no prod action.
• 2026-07-30 (later) — 📈 The last T5 prerequisite — an EHDB-lag KEDA autoscaler for the user pool, so deleting NATS cannot strand it; plus the two #205/#208 prod-rollout cleanups. 4 PRs held (ops#242, ehdb#303, ehdb#304, server#291). Corrected two assumptions — prod monitoring is GMP, not VictoriaMetrics (no in-cluster PromQL endpoint → KEDA metrics-api scraping the writer :9102 directly, no new IAM), and the exposed lag is whole-shard, not per-pool (a stuck system-pool command would pin the user pool at max → ehdb_feed_subject_lag). Scale-up 3 → 6 → 12 → 20 proven in kind on the real metric, then back to min on drain; prod applied paused, unpause staged. T5 HELD.
• 2026-07-30 (prod deploy) — 🏁 The #301+#302 bundle lands on shastaratech prod: EHDB is now ~2.4× faster than NATS, and the bus survives a writer restart — and it turns out it had not been surviving one for 2.4 days. server v3.58.3 / worker v5.81.3 / ehdb 86a24f9 in one rollout. Dispatch p50 138.5 / p95 156.4 / p99 181.0 ms (n=60, unsaturated) vs the 338 ms NATS baseline (#205 CLOSED — T5 latency gate GO). Writer pod deleted mid-load → 38/38 COMPLETED, 114 = 38×3 commands, 0 dup / 0 loss, ~2.7 s redial, cursor-resume not replay (#208 CLOSED). ⚠️ Found on arrival — user-playbook dispatch had been silently dead since 2026-07-28 07:49Z (~2.4 days): the writer restarted, every user-pool claimer wedged on a half-open socket, and noetl-worker-system-pool-shard1 restarting later kept system commands flowing and masked it completely. The T4 sign-off had not survived its first writer restart; this deploy remediated it. Two rough edges left open: the resume log line cannot be told apart from a replay-from-0 (ehdb#304) and 2 × HTTP 500 during the writer gap (server#291). NATS untouched; T5 HELD.*
• 2026-07-30 — 🔁 EHDB command bus survives a writer restart — ai-meta#208 both defects fixed + kind-validated, and a third found; 3 PRs held for review. Branched off ehdb main at 03e94be (#301). Defect 1 (a writer-only restart wedged dispatch silently — claim_next parks on a half-open socket, so the redial path hangs off an Err that never comes): TCP keepalive on every ehdb-feed socket (~11 s detection even with no FIN/RST) + a negotiated coordinator heartbeat (one beat up front arms a 3-missed-beat read deadline for the connection; catches a writer that is alive but stuck, which keepalive cannot) — heartbeats only go to a client that asked, so the wire stays backward compatible. Defect 2 (a restarted writer replayed its whole shard from cursor 0 — kind lag 2738 draining at ~1/s): CursorStore persists committed_cursor() atomically on the writer's volume + ClaimCoordinator::resume starts there (NOETL_COMMAND_BUS_CURSOR_PERSIST_MS, NOETL_COMMAND_BUS_CURSOR_FALLBACK=tail|beginning), the cursor is clamped to the reopened log's tip (a log that lost an unsealed part reopens behind the cursor → without the clamp the bus goes permanently dark), and the writer seals its log on SIGTERM (the engine reopens from its durable manifest, so an unsealed part is invisible after a restart). ehdb_feed_shard_committed{shard} added to the writer :9102 — lag reads 0 both when caught up and when a replay just finished, so the cursor is the series to watch. Third defect, found while validating: the server dropped one command per writer restart (EHDB publish failed: early eof + HTTP 500; command.issued durable, nothing on the bus) → redial-and-retry, 3 attempts 250 ms apart. Kind before/after, same cluster: pre-fix a writer-only rollout restart gave 21 issued / 0 claimed with lag frozen at 3094 for 100 s; post-fix all 3 claimers redialed in ~2 s, the writer resumed from_cursor=3246 origin="persisted" at lag 0, and a writer restarted mid-burst left 30/30 executions completed, 0 dupes, lag → 0. No-regression re-proved: pod-kill redelivery 39/39, pool isolation 0 cross-pool claims (84 + 83 across the two #166 system shards), 0 spurious redials on a 10-min idle bus. ehdb#302 + worker#196 (on #195) + server#290 (on #289) — all held for review; prod bundles #301 + #208 in one deploy. Kind restored to NOETL_COMMAND_BUS=nats; nothing in prod. Follow-up: the SIGTERM seal races in-flight ingest (one command per restart recovered by the orchestrator re-issue ~30 s later — a latency artifact, not loss).
• 2026-07-28 — 🚀 EHDB is the production command bus — L1 T4 full flip SUCCEEDED (2026-07-27) · 🐛 #203 silent feed-delivery loss fixed · 🐍 Python cleanup. The staged Option-2 (single command shard) cutover completed on shastaratech prod with server v3.58.1 / worker v5.81.1: shadow 0 divergence → canary 30/30 (0 system claims, pod-kill redelivery 12/12) → full flip 30/30 + 50/50 soak = 80/80, 0 dup, 0 loss, server NATS_pub=0, feed lag→0, #166 state sharding intact. NATS stays installed as the one-command rollback; T5 (delete NATS) held on the human. ⚠️ Dispatch latency p50 285–557 ms vs ~200 ms NATS (~1.5–2.8×, sub-second) is the T5 go/no-go item → #205; out_of_order_appends not on writer :9102 → #206. The flip needed a real bug fixed first: producer-assigned sparse snowflake ids arriving out of order past the follower cursor were filtered out of the feed forever — silently, lag=0 (~10% loss); fix = the writer assigns the ordering key (ehdb#300 → ehdb d4b6235), kind 17/40 → 41/41, then prod-validated → #203 CLOSED; #202 CLOSED (Option 2 executed; the per-shard user-pool split was never needed). Releases: worker v5.79.0/v5.80.0/v5.81.0/v5.81.1, server v3.58.1. Python: 5.0.0 on PyPI, shims removed (noetl#693), legacy PyPI landmine retired (noetl#694 — pypi.yml/pypi-v* is now the sole publisher); the big server/worker/outbox retirement is BLOCKED on noetl/core's imports → #207. Infra: Cloud Build → Artifact Registry (#204, worker#194 + server#288 OPEN, amd64-only by design). ai-meta gitlink bumps + RFC write-ups left PENDING (dirty shared tree); wikis + issues updated directly.*
• 2026-07-26 — 🚚 Prod migrated to the shastaratech org + cut over · 🐍 noetl 5.0.0 on PyPI · 🔀 EHDB L1 T4 executed on prod (held at canary). Prod stack moved to GKE Autopilot noetl-prod-autopilot (us-central1, shastaratech-noetl-prod) off noetl-demo-19700101; billing unblocked (Free-Trial→paid + noetl-dev→noetl-prod slot swap); images/IAM-WI/10 secrets/NOETL_ENCRYPTION_KEY(19/19 decrypt)/DBs migrated; cloudflared connector flip = 25/25 external /health 200 zero-blip, old prod left warm (shastaratech/gcp-org-playbooks#13, billing #9). noetl 5.0.0 LIVE on PyPI — Rust CLI via PyO3 (abi3-py39), 4 artifacts (macOS arm64 + Linux x86_64/aarch64 + sdist; Windows/Intel-Mac dropped); cli v4.20.0 (lib extraction) + v4.21.0 (DuckDB feature-gate); Stage-4 retirement evidence-only (#201). EHDB L1 T4 command-bus cutover on prod: Step 0 + Shadow PASS (0 divergence, 121→NATS==121→EHDB, 2 shards), HELD at canary on the per-shard user-pool blocker → bus back on NATS, prod clean; server v3.58.0 / worker v5.81.0 + both writers deployed (inert); two paths in #202; IaC playbooks/194-l1-t4-prod-iac/ (#194). Write-behind sink slices A/B/C merged default-off (worker#190/#191/#192, #199); release-red #200 CLOSED. ai-meta gitlink pointer bumps + RFC write-ups left PENDING (busy shared tree; wikis pushed directly).
• 2026-07-14 — 🧩 Provider tool round 5 — noetl provider {plan,drift,orphans,adopt} CLI verbs + query-tier fetch adapter + stack-scoped ownership, merged + released. tools#90 → noetl-tools v3.25.0 (5a06c7e): extract_facts_from_tier_records/fold_tier_records (wire-shape eventlog-tier fold — unwrap the payload string + recursive fact search; fixes the round-3/4 fixed-path bug — extract_facts_from_eventlog never matched the real context.result.context.data.provider_fact nesting, so a fold over real data on 3.23/3.24 silently returned nothing) + OwnershipModel::for_stack/detect_orphans_scoped/detect_stack_conflicts. cli#68 → noetl-cli v4.17.0 (a2e5b4d): the noetl provider verbs (Terraform-style CLI over the tool, local mode; glue over ProviderTool::execute+provider_state+to_tools_config); re-pin noetl-tools 3.24→3.25; noetl-executor unchanged at 0.7 (zero executor/ files touched — version-bump discipline held). crates.io tarball-verified + released-binary proof (noetl-v4.17.0-darwin-arm64, all 4 verbs GET-only zero-writes against the mock; drift over HTTP --server; adopt dry-run→digest→apply→stale-refusal). Follow-up 3 (first-class event type) DEFERRED (projection column rewrites the secret-free invariant tests + rebuilds server/worker). ai-meta gitlinks tools→5a06c7e + cli→a2e5b4d. Umbrella #189 OPEN. No prod/kind/live-cloud — mocks + unit tests only.
• 2026-07-13 — 🧩 Provider tool round 4 — drift adoption (reconcile policy report/enforce/adopt + confirm-gated adopt), merged + released. tools#89 → noetl-tools v3.24.0 (c172efb); cli#66 → noetl-cli v4.16.0 + noetl-executor 0.7.0 (7557dde). report new-default (converge needs reconcile:enforce); adopt accepts live actual as new desired, read/GET-only, confirm-gated like destroy (blind+stale REFUSED). All three crates crates.io-verified. ai-meta gitlinks tools→c172efb + cli→7557dde. Umbrella #189 OPEN.
• 2026-07-13 — 🚀 Provider tool round 3 — confirm-gated destroy + EHDB ownership projection + endpoint override, merged + released. tools#88 → noetl-tools v3.23.0 (aa32938): destroy verbs (*.delete/services.disable/*.iam.remove_binding) gated by a live-resolved SHA-256 plan_digest (dry-run emits plan + digest; apply needs confirm:<digest>; blind apply + stale digest REFUSED, nothing deleted; no wildcard/bulk; needs auth: even to plan; real destroys human-run); secret-free provider_fact per converge + pure provider_state fold (fold_facts/compute_drift/detect_orphans) over the raw eventlog tier (not the secret-free /api/ehdb/events read-model — #178 collision resolved); config-level endpoint: override for offline mock/emulator runs. cli#65 → noetl-cli v4.15.0 (94cc72a): local-mode endpoint/stack/confirm passthrough + re-pin to noetl-tools 3.23.0 (no fresh libduckdb-sys compile). Local-mode PROVEN on 3.23.0 (mock, no live cloud): endpoint-override apply → no-op; destroy dry-run → plan + plan_digest 6b5efdbb…, no DELETE. 39 provider tests + 464 lib + clippy clean. ai-meta gitlinks tools→aa32938 (f6eeb479) + cli→94cc72a (cbc7be99). Umbrella #189 OPEN — round-4 = drift adoption. No prod/kind/live-cloud, no real destroys — mocks + unit tests only.
• 2026-07-13 — 🚀 Provider tool round 2 — apply-mode converge + bounded LRO + local-mode CLI dispatch, merged + released. tools#87 → noetl-tools v3.22.0 (eb3d8c7): idempotent GET-first converge for the 4 ensure/ensure_binding actions (+ services.enable/projects.link, re-run = no-op) + bounded inline LRO polling (descriptive timeout + documented callback overflow). cli#64 → noetl-cli v4.14.0 (2d46013): kind: provider dispatches through noetl exec --runtime local (same tool the worker runs), re-pinned to noetl-tools 3.22.0 (libduckdb-sys unchanged → no fresh compile), supersedes cli#63 (closed). Local-mode PROVEN (dry-run would_call + apply-mode converge entry; mocks not live cloud). ai-meta gitlinks tools→eb3d8c7 (6cd1ba38) + cli→2d46013 (32a8fb86). Both design forks DECIDED for round 3 (EHDB-backed state; confirm-gated destroy). Umbrella #189 OPEN. No prod/kind/live-cloud — mocks + unit tests only.
• 2026-07-13 — 🆕 NoETL cloud provider tool (kind: provider) — Google REST MVP designed → implemented → merged + released noetl-tools v3.21.0 (tools#86 5e427fb, release cd9d71c); ai-meta pointer e9b3e14e; umbrella #189 OPEN (round-1). No prod/kind/GCP — impl + unit tests only.
• 2026-07-10 — 🗑️ EHDB segment-GC limits-based retention — a shadow store can self-bound without a consumer. Merged noetl/ehdb#271 → ehdb main 49db503 (engine) + noetl/worker#178 (knob) + noetl/worker#179 → worker main 521fd10 (re-pin ehdb to merged #271; #178 pinned the pre-merge branch commit). Keep-last-N (count-based) retention reclaims independent of consumer interest; plan_reclaim composes it as min(retention_target, interest_ceiling) — never reclaims ahead of a real consumer, drives when there is none, min_retained_segments is the hard floor; the only engine change is inside plan_reclaim (watermark-first commit / shared-tier coherence / per-shard lock all inherit unchanged). Keep-last-N not max-age (mtime resets on hydrate/cold-load). Config NOETL_EHDB_EVENTLOG_GC_MAX_RETAINED_SEGMENTS + worker NOETL_EHDB_EVENTLOG_SEGMENT_MAX_BYTES. 251 tests (+3 retention: self-bounds a consumer-less store / never reclaims ahead of a lagging consumer / respects the min floor); clippy+fmt clean; bench variant. Organic in-kind soak PROVEN: live store 19→2 segments / 2,147 B (from ~22 MB) under retention with no consumer, held flat; reclaimed=6, 0 restarts, replay intact. LOCAL kind only — no GKE/prod; noetl.event never purged. ai-meta gitlink bump repos/ehdb→49db503 + repos/worker→521fd10.
• 2026-07-10 — 🗑️ EHDB segment-GC worker wiring + in-kind soak — D11 FULLY CLOSED (mechanism + deployed exercise). Merged noetl/worker#177 → worker main b25263f: a periodic per-owned-shard eventlog_gc task (local + shared reclaim_shard) on spawn_blocking, gated off on every axis; a process-global per-shard advisory lock serializes GC against the per-op append path (resolves the two-writers-per-shard fork + closes a latent append↔append race). noetl_ehdb_eventlog_gc_* metrics; ehdb-selfcheck durable-eventlog-gc [--drive 1]. In-kind soak (v5.72.3-ehdbgc254, user pool, /ehdb-durable PVC, GC + interval=30s): periodic task spawned + healthy (eventlog_gc_ops_total{outcome="noop"}, last_ok=1 — correct noop, shadow store has no consumer); --drive 1 on the real PVC reclaimed shared 40→10 at watermark 30 (gc_holds, replay/cold-load/new-owner-hydrate coherent); 0 restarts; user+system pools uniform. Fork surfaced: the deployed shadow event-log has no durable consumer → interest-based GC reclaims nothing organically (correct); a limits-based retention mode (keep-last-N/max-age) for shadow, or a primary-consumer, is the one remaining optional follow-up. LOCAL kind only — no GKE/prod; noetl.event never purged. ehdb-wiki d68b988; ai-meta gitlink bump repos/worker→b25263f.
• 2026-07-09 — 🗑️ EHDB shared-tier segment GC — coherent reclamation across the shared medium (D11 done for both topologies). Merged noetl/ehdb#270 → ehdb main adca378: a monotonic per-shard cross-replica reclaim watermark on the shared medium; SharedTierEventLog::reclaim_shard reclaims local + shared watermark-first (reconcile → plan → commit watermark → reclaim local → delete shared), so cold_load/hydrate_owned_shard skip <= watermark and never re-pull a reclaimed segment (no un-reclaim on transfer); reconcile_owned_shard realigns a crashed owner. Store refactored into plan_reclaim (decision) + apply_reclaim/reclaim_to_segment (mutation). 248 tests (+5 shared-GC incl. crash-window reconcile); CLI durable-eventlog-shared-gc pruned shared 40→10 objects at watermark 30, cold-load + new-owner hydrate coherent; bench ~25 ms@50 / ~38 ms@150. Off by default + not auto-invoked. D11: GAP → delivered + drive-tested for BOTH single-writer local (PVC) AND shared-medium — remaining is operational (worker wiring + in-cluster soak). LOCAL/engine only — no GKE/prod; noetl.event never purged. ehdb-wiki 94afe29; ehdb#254 comment; ai-meta gitlink bump repos/ehdb→adca378 + repos/ehdb-wiki→94afe29.
• 2026-07-09 — 🗑️ EHDB durable event-log segment GC — the R1 / D11 gap (Stage-C blocker) closed for single-writer local. Merged noetl/ehdb#269 → ehdb main 93f3060: interest-based (consumer-ack watermark) reclamation of consumed sealed segments behind NOETL_EHDB_EVENTLOG_GC=consumer_ack (off by default). Base-offset (retained gapless-from-reclaimed+1; reclaimed_through==0 ⇒ byte-identical to pre-GC), durable fsync'd reclaim.json manifest = crash-atomic commit point (crash mid-GC re-deletes below-watermark leftovers on next open), write-forward of consumer Ack frames before unlink. Fixed tail() pending to the absolute tip. 242 tests (+10 GC incl. crash-mid-GC-via-manifest), clippy/fmt clean; CLI drive 41→11 segments at watermark 30. D11: GAP → delivered + proven for single-writer local (PVC), unblocking the Stage-C durable-primary consideration for that topology; shared-tier object reclamation is the one remaining follow-up (started this session). LOCAL/engine only — no GKE/prod; noetl.event never purged. ehdb-wiki e830957; ehdb#254 comment; ai-meta gitlink bump repos/ehdb→93f3060 + repos/ehdb-wiki→e830957.
• 2026-07-04 — 🔐 #169 Auth0 JWT signature verification — SHADOW canary LIVE on prod (server v3.50.0→v3.52.0, single replica, NOETL_AUTH_VERIFY_SIGNATURE=shadow); focused finish handover Handover-169-JWT published (docs/state-sync only — no code/prod/flag change).
• 2026-06-29 — 🎟️ #164 Muno ACTIVITIES render fixed on prod — planner v59 (cid 660442656010404734, rollback v58 660417115383399292). The activities turn rendered blank (render: null) though the provider returned 10 real activities: render_widget_chat crossed the ~100 KB result-tier offload threshold → noetl:// ref that 404s on prod (#154). The 10 activity cards were duplicated ~6× (widgets/first_widget/final_slot_state/post_docs/two post_events), each ~1.2 KB dominated by a raw [:600] HTML description (75 KB render on a fresh thread). Fix travel#88: _activity_card strips HTML → short plain-text (160 ch) + modalities 5→2 → card ~0.57 KB, render 75.3→39.5 KB. No widget-schema/FE change (CLOSED activity_list.schema.json still validates). Prod-verified: real activity_list (10 activities, no synthetic, no offload, schema PASS); places/hotels/flights/login unaffected; 0 restarts. #164 open until travel#88 merges + bump; refs #155.
• 2026-06-29 — 🏨 #162 Muno HOTELS bookable end-to-end on prod — planner v58 (cid 660417115383399292) + HotelBeds MCP v4 (cid 660409660586853242, book_hotel checkRate→bookings, TEST/SANDBOX-guarded). Verified: real TEST booking refs 197-14375473/74/76 (CONFIRMED, env=test) → hotel_confirmation; search + Trip-to-Paris + login healthy; 0 restarts. Rollback = planner v57 cid 660221263901623160. Closes #162; refs #155.
• 2026-06-28 — 🚑 #157 Muno hotels-turn drive wedge fixed forward + DEPLOYED — worker v5.47.3. A hotels follow-up ("What hotels are in Paris?") wedged in RUNNING because the hotelbeds render-hop __orchestrate__ drive result (~138KB) crossed the 100KB inline budget and offloaded to a result-store ref the server can't resolve under NOETL_RESULT_STORE_DUAL_WRITE=false (#104) — a #154-class wedge. worker#141 (squash f8f4d12, released v5.47.3 9f83ab5) keeps the __orchestrate__ result inline regardless of size. Bumped ai-meta repos/worker 5.47.2→5.47.3 (60938f4), rolled both pools new-pod-first (0 restarts, materializer balanced/lag-0, Fix A NATS reconnect 0/30s intact). Prod-verified: hotels exec 329775581438681088 COMPLETES + hotel_list widget; places no-regression exec 329775627576025088 COMPLETES + place_list. Worker-behavior-only — no OQ5/result_store/IAM/shadow change; rollback :5.47.2. Closed #157, referenced #155.
• 2026-06-28 — 📱 Muno mobile responsive layout SHIPPED to prod — the live travel SPA was still serving the broken overlapping 3-column shell on mobile because the fix had been built but never deployed. Merged travel#80 (squash → a8e78f1): reflows the 3-panel shell to a single column at the md breakpoint, sidebar + Trip-state become temporary MUI drawers (hamburger ☰ + luggage 🧳 buttons). The merge triggered the repo's Cloudflare Pages workflow (run 28329391574) → prod deploy https://417ab9ee.travel-bgo.pages.dev (bundle index-0naZf1z6.js). Verified live signed-in at 380px: full-width single-column chat, sidebar/Trip-state as left/right drawers with backdrop, composer full-width + Send reachable; desktop still 3-column. Minor nit: 16px composer-<form> overflow (cosmetic, follow-up). ai-meta repos/travel 79be476→a8e78f1; SPA + Cloudflare Pages only (no backend/OQ5/IAM/GKE).
• 2026-06-27 — 🚀 #104 result_store dual-write RETIRED on prod — server v3.47.0 (server#267, bd831e6, image @sha256:140066d9…) deployed behavior-neutral (flag default-on), then NOETL_RESULT_STORE_DUAL_WRITE=false flipped. Over-budget result_store writes stopped (dual_write_skipped_total climbs, put_total/dual_write_total flat); the #104 tier serves over-budget results by URN with 0 resolve errors; execs COMPLETE; sole-writer balanced 142==142==142 / dup-0 / lag-0; server + system-pool 0 restarts. Existing result_store data frozen (not deleted); revert = flag→true (deeper: RESULT_MINT_AUTHORITATIVE=false; image rollback v3.46.0 @sha256:6254838…). Probe muno/oq5-retire/test/overbudget. ai-meta server pointer → v3.47.0.
• 2026-06-26 — 🧱 SLM platform foundation G1/G2/G3 merged + released (review-only): tools v3.19.0 (container GPU placement + poll-to-terminal helper, #82), worker v5.47.0 (watcher-free poll completion fallback, flag NOETL_CONTAINER_COMPLETION_POLL default-off, #139), server v3.46.0 (versioned model/dataset/eval registry, flag NOETL_REGISTRY_ENABLED default-off, #266); ai-meta pointer bump 2e79af6. Plus the SLM teacher swapped OpenAI→Vertex Gemini (gemini-2.5-pro→flash fallback, Workload-Identity metadata token mint, no API key/secret) — ops #216 + travel #75 open on the Phase-A track. ops#214 held (stacked on the ops SLM branch). No prod deploy — both runtime flags ship default-off.
• 2026-06-25 — 🧠 Domain-Specific SLM platform Phase A (#140) landed — dataset_build + eval shipped as generic, config-driven NoETL playbooks (MLOps-as-playbooks). ops PR #212 (generic automation/mlops/slm pack + lib/ engines + toy 2nd domain) + travel PR noetl/travel#73 (travel instance: deterministic oracle as zero-cost label source, 45-turn seed corpus, contract schemas). Config-only proven (RFC §2.2 extraction test: a support-triage toy domain runs the same playbooks zero-edits). Baseline (45 turns → 29/16): 100% schema validity, eval gate passed, floor latency sub-ms, 9 intents/4 tools/10 widget types. Validated via noetl exec -r local. OpenAI ceiling deferred (teacher budget); finetune/serve gated on G1/G2/G3; model-choice deferred. Review-only, no prod change.
• 2026-06-25 — 🧠 Domain-Specific SLM platform RFC + umbrella (#139) + 11 children filed (design only).
• 2026-06-23 — 🪶 #104 OQ5 Option A SHIPPED — producer-staged result tier (worker v5.46.0); decouples the over-budget tier write from result_store (flag default-off). worker#132 + e2e#81 + docs#186 merged, ai-meta pointer #128; ops#206 soak-gate rules open. result_store-retirement soak defined but NOT started (gated on a prod producer-stage flip).
• 2026-06-23 — 🚀 #104 — WI/ADC GCS auth MERGED (server#265, fad5d8a → noetl-server v3.45.0) + ROLLED TO PROD GKE: server runs as the WI-bound KSA noetl-server-rust (@sha256:d3cbf1ad…) with the result-tier GCS ENV applied (OBJECT_STORE_BACKEND=gcs, bucket noetl-demo-19700101-results, auth=adc, cell usc1-a/256); /api/internal/cells reads it back; all tier-enable flags stay OFF → tier inert. Server healthy auth=adc (no token minted — lazy), off-server cutover sole-writer/lag-0/never-scan (smoke + system/scheduled_cleanup COMPLETED, published==acked=13, chain clean), 0 restarts. Prod configured + WI-live, ready for staged B→C→D enablement.
• 2026-06-23 — 🪶 #104 Phases E + F MERGED — the FINAL build phases (side-effect durability barrier + result-tier GC + DR), repo-only, flags default-off → inert in prod. Phase E: tools#78 → noetl-tools v3.17.0 (registry::kind_is_side_effecting, members noetl-directives + noetl-locator re-published first), worker#130 barrier → v5.44.0 (repointed onto published 3.17, no patch), e2e#79. Phase F: server#264 GC sweeper → noetl-server v3.44.0, worker#131 DR re-derive → noetl-worker v5.45.0, ops#205 + e2e#80. ai-meta pointers + deployment-spec wiki rows bumped. A–F build phases complete; #104 stays OPEN (operational items remain).
• 2026-06-23 — 🪶 #104 Phase D MERGED — the minting flip (dual-write window). All 4 PRs squash-merged in dependency order (server → worker → ops → e2e), flags default-off → inert in prod. Repo-only; prod minting cutover is a separate next task. NOETL_RESULT_MINT_AUTHORITATIVE makes the URN → Feather/GCS tier the authoritative result store with noetl.result_store kept as the reversible dual-write fallback. No new crate publish — no Cargo.toml dep change in either PR; server resolves noetl-locator 0.1.1, worker resolves from the registry. semantic-release cut noetl-server v3.43.0 + noetl-worker v5.43.0: server#263 → v3.43.0 6f6b9ef, worker#129 → v5.43.0 be6863a, ops#204 b19b759, e2e#78 07e85aa. ai-meta pointers bumped. OQ5 — result_store retirement — DECIDED metric-gated (drop dual-write once mint_authoritative_total{path=legacy_fallback} holds 0 across a staging soak + retention time floor), gated on a not-yet-done byte-source re-plumbing prerequisite (materializer fetches the payload from result_store today). #104 stays OPEN (Phase E/F + minting prod-cutover remain).
• 2026-06-23 — 🪶 #104 Phase C MERGED — resolve-by-URN read path (server GCS object backend + cell registry + GET /api/internal/cells; worker resolve-by-URN + fixes B/B1, closes OQ6). All flags default-off → inert in prod. Merged in dependency order (server → worker → e2e): server#262 → noetl-server v3.42.0 c2d5ca9, worker#128 → noetl-worker v5.42.0 7971041 (38 unit tests), e2e#77 39dc880 (3-pass rig + fake-gcs). No new crate publish — worker resolves noetl-tools 3.16.0 + adds published arrow=53; server resolves noetl-locator 0.1.1, both from the registry (no repoint). ai-meta pointers bumped; #104 stays OPEN (Phase D minting flip remains).
• 2026-06-22 — 🪶 #104 Phase B MERGED — shadow Feather result tier (5 PRs, flags default-off → inert in prod). Merged in dependency order: tools#77 (→ noetl-tools v3.16.0 + noetl-locator 0.1.1 published; member bumped 0.1.0→0.1.1 pre-merge so the additive parse/from_locator API publishes) · server#261 (→ v3.41.0, sibling noetl_result_materializer consumer) · worker#127 (→ v5.41.0, the consume-loop) · ops#203 (flag + cell seed) · e2e#76 (kind rig). No downstream repoint needed (both resolve from the registry, no git/branch dep). ai-meta pointers bumped; #104 stays OPEN (C–F remain). Session details
• 2026-06-22 — 📦 #104 Phase A MERGED — slim noetl-locator 0.1.0 extracted + published to crates.io; server accepts the canonical result URI behind NOETL_RESULT_URI_ACCEPT (default off). Repo-only, no prod deploy. Squash-merged tools#76 with a feat(locator): subject → noetl-tools v3.15.0 (dc0c5d8) and the release CI published the new workspace member noetl-locator 0.1.0 (member-publish order: locator before the root crate). Repointed server#260's temporary git-dep to noetl-locator = "0.1.0", confirmed cargo build/cargo tree resolve from the registry with the heavy graph absent (duckdb/kube/arrow/tonic/rhai/gcp_auth = 0), 623 tests green; squash-merged → noetl-server v3.40.0 (c89d078) + the e2e#75 kind rig (eeca8b7). Resolves the umbrella's OQ7. Flag default-off → inert in prod until a later server rollout. ai-meta pointers bumped (repos/tools→dc0c5d8, repos/server→c89d078, repos/e2e→eeca8b7); #104 stays OPEN.
• 2026-06-22 — 🧹 CQRS / off-server program close-out + tracking reconciliation (repo-only, no prod changes). Merged e2e#74 (squash 1deadf1, the prod-scoped Rust regression validator, 28/30 against live prod) + bumped repos/e2e. Closed #103 (CQRS event log — prod cutover live + validated; e2e#74 was its last piece), #102 (server#198/#199 landed; worker-side superseded by #103), #101 (acceptance met + kind-validated; consume side moved to #115 Phase 1). Closed superseded Python ops PRs #189/#190/#192 (replaced by the Rust in-process worker materializer noetl-worker/src/materializer.rs). Archived the completed 2026-06-18-orchestrate-plugin-dissolution + stale 2026-06-09-rust-stack-session-snapshot handoffs. Reconciled roadmap board 3 + this Active-umbrellas table. Fixed the failing release-server CI (cargo publish — added a version to the noetl-orchestrate-core path dep + publish the member first, mirroring noetl/tools).
• 2026-06-22 — 🐘 #95 CLOSED — postgres pg_value_to_json temporal/identity serialization shipped to prod end-to-end. The Rust postgres tool returned null for a tz-naive timestamp (NaiveDateTime) column plus date/time/uuid/numeric/bytea (the auth0_login expires_at: null repro). Merged tools#75 (squash 06302ac → semantic-release noetl-tools v3.14.2 6d9b674, published to crates.io) + worker#126 pin bump 3.14.1→3.14.2 (squash 60a849d → noetl-worker v5.40.5 da24952). Built the prod image via Cloud Build us-central1 (noetl-worker-rust:v5.40.5 @sha256:45212dbe…) and rolled it by digest onto prod noetl-worker-rust + noetl-worker-system-pool (server stays v3.39.5, CPU 250m/2 kept) — rolling restart clean (2/2 + 1/1, 0 crashloop), off-server CQRS cutover stayed healthy (materializer sole-writer lag 0, command lag 0, project_errors 0, WAL index rehydrated 5 execs/198 events per #119). ai-meta repos/tools→6d9b674 + repos/worker→da24952.
• 2026-06-22 — ✅ #127 CLOSED — the task_sequence per-sub-task context optimization merged, released, and shipped to prod; the code-opt + CPU-limit bump now compound on the live off-server cutover. Merged tools#74 (squash 9dd9aa6 → semantic-release noetl-tools v3.14.1 c8656c1, published to crates.io) — render_value builds the proxied minijinja context ONCE + threads it through the recursion (render_value_with/render_with), and new build_context_with_overlay skips the redundant to_template_context() HashMap clone + per-block ExecutionContext clones; micro-bench −61.6%/2.6× per-sub-task context cost (407 tests + 2 equivalence pins + clippy). Bumped the worker noetl-tools pin 3.14→3.14.1 (worker#125, squash 1a10a73 → noetl-worker v5.40.4 0afbf5c; deps-only, no worker source change, clippy clean). Built the prod image via Cloud Build (us-central1-docker.pkg.dev/noetl-demo-19700101/noetl/noetl-worker-rust:0afbf5c) and rolled it onto prod noetl-worker-rust + noetl-worker-system-pool (server stays v3.39.5/.6, worker CPU limits kept req 250m/limit 2). Rolling restart clean (pods Ready, 0 crashloop), and the off-server CQRS cutover stayed healthy throughout — materializer sole-writer projected==acked, lag ~0, command lag 0, executions completing, system-pool rehydrated its WAL index (#119). ai-meta repos/tools→c8656c1 + repos/worker→0afbf5c pointers bumped + wiki. #127 closed + board 3 → Done.
• 2026-06-22 — ⚡ #127 PROD WORKER CPU LIMIT RAISED 1→2 + APPLIED LIVE — the ~20% batch-throughput win materialized on prod. ops#202 (85e3c23) raised request 100m→250m / limit 1→2 on both noetl-worker-rust + noetl-worker-system-pool; ai-meta repos/ops pointer bumped. Kind profiling found the step workers pegged one CPU and spent 38–47% throttled at the 1-CPU limit (per-sub-task running_ctx.clone() + to_template_context() hot path); 2 CPU cut a 10k batch 166–172s→137.6s (~20%, zero patient loss). Live prod baseline confirmed 1 CPU (kind had 0.5); nodes 4× ek-standard-16 (~63.5 CPU) at ~2% → headroom. Applied via kubectl set resources: rolling restart clean (all pods Ready, 0 restarts), off-server CQRS cutover stayed healthy (materializer sole-writer lag 0, WAL index rehydrated). PROD gate vars / image / DB untouched; the deeper context-clone hot path is the structural follow-up. #127 stays OPEN.
• 2026-06-22 — ✅ #123 SHIPPED + CLOSED — a non-iterable loop in: no longer silently wedges (commands=0, RUNNING forever); the off-server drive now decodes the {"error":…} envelope and emits a terminal playbook.failed (server v3.39.6). server#258 (squash 275b914 → v3.39.6 7f109a9): apply_worker_orchestration decodes the drive ERROR envelope (decode_orchestrate_error) and emits the terminal failure (metric noetl_orchestrate_drive_total{stage="drive_error"}, structured execution_id), matching the in-process drive; orchestrate-core prefixes the offending step name onto the existing evaluate_loop error. An empty iterable ([]/{}) still short-circuits to next. 600 server + 135 orchestrate-core tests + clippy clean; kind-validated prod-exact (absent workload.batch_slots → FAILED with a clear message, valid [1,2,3] still COMPLETED 3-way, #120/#124 unaffected, 0 restarts). Code-only fix (worker stays v5.40.3); ai-meta pointer bumped → v3.39.6; #123 closed + board 3 → Done. PROD stays healthy on v3.39.5 + the live off-server cutover (NOT redeployed — ships to prod on a future server rollout). #127 stays OPEN.
• 2026-06-22 — ✅ #121 SECOND-HALF SHIPPED + CLOSED — off-server system/* WAL-chain wedge fully fixed (server v3.39.5). server#257 (squash 54ac277 → v3.39.5 c421273) gates BOTH off-server-drive sites in trigger_orchestrator_inner on should_publish(catalog_id) so system/* execs drive server-built run_state; regular execs keep the off-server path. #256 was only the first half (#121 reopened after the v3.39.4 prod re-cutover still wedged system/scheduled_cleanup on the system-pool worker). Pure is_system_path helper + unit test; 612 tests + clippy; kind full-gate BEFORE 9× WAL chain incomplete → AFTER COMPLETED 0 loops + regular loop still off-server. Server-only (worker stays v5.40.3); ai-meta pointer bumped; #121 closed + board 3 → Done. PROD re-cutover = Phase-B follow-up this session.*
• 2026-06-21 — ✅ #121 first-half SHIPPED (partial) — off-server WAL-chain-incomplete loop on system/ execs (link gate-off command.claimed through ChainHeads + don't route system/ execs to the off-server WAL drive); server v3.39.4; server-only; live prod off-server cutover wedge is the real-world repro.
• 2026-06-21 — ✅ #125 + #126 SHIPPED + CLOSED — task_sequence do: jump/break/retry control flow + http tool body data-shape fixed (noetl-tools v3.14.0 tools#73 + v3.13.1 tools#72); worker v5.40.3 adopts the fixes (worker#124). Full 10×1000 batch pft_flow_test now passes end-to-end on kind, correctness-clean (zero patient loss). Perf follow-up #127 filed. #121/#123 stay OPEN.
• 2026-06-21 — ✅ #124 SHIPPED + CLOSED — distributed task_sequence forward set:/sibling bindings no longer render empty; defer sub-task templates with unresolved vars to the worker re-render (render_value_deferring_unresolved); server v3.39.3 (server#255). #121/#123/#125/#126 stay OPEN.
• 2026-06-21 — ✅ #120 SHIPPED + CLOSED — reduce barrier no longer deadlocks (commands=0) on open/asymmetric loop joins (orchestrate-core); runtime liveness filter in the barrier (server v3.39.2 28e8950). server#254 (squash fbb855f); test test_open_loop_back_edge_does_not_block_dispatch, 133/133 + clippy, kind-validated 2×2 matrix COMPLETE; PROD/defaults untouched. #121/#123 stay OPEN.
• 2026-06-20 — ✅ #116 program-scale step 2 SHIPPED + multi-replica gate-ON validated — execution-affinity single-owner WRITE ORDERING. server#252 v3.39.0 + e2e#71. Step 1 (KV coherence) was necessary-not-sufficient — the command.issued prev-read + head CAS-advance are two non-atomic steps, so concurrent cross-replica emits forked the chain. Affinity routes every trigger (POST /api/events, which fires the drive) to the single replica that ShardConfig::owns(execution_id) owns; a non-owner forwards a reverse-proxy POST (one-hop loop guard, degrade-to-local). On the owner the single-process drive lock + in-memory ChainHeads make the read→advance atomic, no distributed lock; KV is the genesis/handoff vehicle (owner resolves LOCAL → kv_remote_hit→0). server#252 (5e00d0a, v3.39.0): src/affinity.rs, flags NOETL_EXECUTION_AFFINITY/NOETL_PEER_URL_TEMPLATE/NOETL_SHARD_INDEX_FROM_HOSTNAME (all default off), metric noetl_execution_affinity_total{outcome}. e2e#71 (66b6e1b): 2-replica StatefulSet topology + rig HARD gate. Multi-replica gate-ON kind PASS: linear/loop/fanout COMPLETE, every chain roots=1/dangling=0/walk==total (NO fork), forwarded_ok +9, never-scan (scans+0) + sole-writer across replicas; single-replica unchanged; 595 server tests + clippy green; baseline restored. Follow-up #117: off-server from_events spine ordered by event_id wedges fan-in under a chain-order≠id-order inversion (affinity + high-concurrency fanout) — fix = order spine by prev_event_id walk; linear/loop already reliable. Prod multi-replica verdict: write-ordering COMPLETE (no fork) — prod can horizontally scale the off-server stack for linear/loop; high-concurrency fan-out needs #117 first. All affinity flags default off; PROD GKE untouched.
• 2026-06-20 — ✅ #115 program-scale step 1 SHIPPED — multi-replica coherence DATA LAYER (NATS-KV-backed ChainHeads + ExecDescriptor); execution-affinity STAGED. server#251 v3.38.0 + e2e#70. NOETL_REPLICA_COHERENCE=nats_kv (default local, prod unchanged) backs the off-server drive's watermark + descriptor with JetStream KV buckets so 2+ replicas resolve the same value — head advance = CAS (one chain), descriptor = CAS merge; in-process maps = write-through cache / degraded fallback (local → bit-identical). server#251 (8f39a79): src/coherence.rs (CoherenceKv + lazy buckets), ChainHeads/ExecDescriptors async, ExecDescriptor serde, metric noetl_replica_coherence_total{structure,op,outcome} (proof kv_remote_hit). e2e#70 (e222877): kind_validate_replica_coherence.sh + un-staled the #113/#114 offload asserts behind NOETL_RIG_EXPECT_OFFLOAD (default false; under refs_in_state=true the offload paths legitimately stay flat). Kind-validated: single-replica nats_kv is bit-for-bit parity with local (linear/loop/fan-out ×2 all COMPLETE, roots=1/dangling=0/walk==total, state_build_event_scans +0, hotpath scan +0, sole-writer intact); 2-replica proved cross-replica resolves work (kv_remote_hit advanced for head + descriptor, no kv_unavailable). Necessary but NOT sufficient: on 2+ replicas concurrent cross-replica emits still fork the chain (the issuing_event head-read vs head-advance is non-atomic across replicas → observed forked chains + a cross-execution prev), so executions don't reliably COMPLETE on 2+ replicas yet — the remaining piece is execution-affinity (one replica owns an execution's drive + chain write; substrate present in src/sharding.rs shard_for/owns), STAGED as program-scale step 2. 588 server tests + clippy green; baseline restored. ai-meta pointers → server 8f39a79 (v3.38.0) + e2e e222877. PROD GKE untouched; default local; no gate/mode/builder default changed. Off-server architecture is multi-replica-COHERENT (data) but not yet multi-replica-COMPLETE (write-ordering) — prod cutover stays single-replica until affinity lands.
• 2026-06-20 — ✅ #115 Phase 5 SHIPPED + gate-ON validated — atomic-working-item context (tenet 6): the drive hands a worker only its minimal declared slice; NOETL_ATOMIC_ITEM_CONTEXT (default off). #77 (Explicit Input Binding) resolved. server#250 v3.37.0 + worker#121 v5.40.0 + e2e#69.
• 2026-06-20 — ✅ #115 Phase 6 SHIPPED + gate-ON literal-zero validated — hot-path noetl.event read class RETIRED; the table is AUDIT-ONLY (server v3.36.0). NOETL_EVENT_READ_PATH=event_scan|audit_only (default event_scan, prod unchanged) retires the remaining lifecycle readers of noetl.event (the WHERE execution_id replay class outside the drive). server#249 (b71ca1d): under audit_only get_catalog_id (per-ingest) + inherit_parent_trace + subscription dedup-audit + container-callback catalog/existence serve from the in-memory execute-time ExecDescriptor; a cold descriptor (post-terminal straggler after eviction / restart) resolves catalog_id from noetl.command (synchronous queue) — never a noetl.event scan. Proof metric noetl_event_hotpath_reads_total{site,outcome}. ops#199 (e5b0737) pins event_scan on the prod server manifest (operator-gated flip). e2e#67+#68 (0ab3c0a) kind_validate_event_read_path_phase6.sh. Gate-ON kind-validated (PUBLISH_ONLY + offserver + materializer + audit_only): hot-path scan Δ0 (served_descriptor +96 + served_command +3), drive state_build_total Δ0 + event_scans Δ0 ⇒ ZERO noetl.event scans anywhere on the hot path, end-to-end; linear/loop/fan-out/output_select COMPLETE; sole-writer + lag-0; audit still works (direct SELECT + status COMPLETED + replay event_count=25); committed gate rig PASS with audit_only on (no regression); 585 server tests + clippy green; baseline restored. Completes the RFC never-scan end state (tenet 3) under the flag. ai-meta pointers → server b71ca1d (v3.36.0) + ops e5b0737 + e2e 0ab3c0a. PROD GKE untouched; default event_scan; no gate/mode/builder default changed. Remainder = Phase 5 (atomic-item, needs #77) + program-scale (per-shard WAL, multi-replica descriptor coherence).
• 2026-06-19 — ✅ #115 Phase 4 KERNEL + FLAG SHIPPED + shadow kind-validated — off-server state builder (worker v5.37.0 + server v3.33.0); drive cutover staged. The pool-side state_builder (worker#118, fef961c) reconstructs WorkflowState from the noetl_events WAL — a per-execution chain index walks prev_event_id head→root, caches the spine keyed by the immutable chain head, advances only the new tail. A live WAL shadow loop (NOETL_STATE_BUILDER_SHADOW, default off) + the server NOETL_STATE_BUILDER=offserver|server flag (server#246, 3e6006d, default server). Gate-ON kind-validated on live cluster: shadow replayed the WAL → chain-walked spines whose indexed==spine sizes match Phase-3 topologies (linear 13, loop 62, fan-out 25, output_select 31, storage_tiers 55) = parity by construction; WAL-read wal_events_total=993 with event_scans_total=0; cache cold_rebuild=28 (replay/restart) + incremental=21 (live tail-advance — fresh fan-out indexed Incremental(5), indexed==spine==25==DB event_rows 25); fresh fan-out COMPLETED gate-ON, event_rows==distinct, 0 __orchestrate__ event rows, materializer pending=0/project_errors=0; 8 worker unit tests + 2 server config tests + clippy green; baseline restored. ai-meta pointers → worker fef961c + server 3e6006d. PROD untouched; no default changed. The offserver drive cutover (drive consumes the builder's state) + Phase 5 (atomic-item context, #77) / Phase 6 (retire event read path) remain.
• 2026-06-19 — ✅ #115 Phase 3 MERGED — chain-walk state builder (server v3.32.0); Phase 4 (off-server state builder + WAL cache) started. server#245 self-merged (no classifier block) → server v3.32.0 (8338417); ai-meta repos/server pointer bumped. Behind NOETL_STATE_BUILD_MODE=chain_walk (default event_scan, prod unchanged) the drive reconstructs WorkflowState by walking prev_event_id head→root (in-memory ChainHeads head + (execution_id,event_id) PK lookups — never a WHERE execution_id scan) → same from_events (orchestrate-core unchanged; parity by construction); falls back to event-scan on cold-head / lag / non-genesis. Gate-ON kind-validated (prior session): parity 41/41 MATCH, event_scans_total=0 / 1064 PK hops / 0 fallbacks, all topologies COMPLETE, sole-writer + lag-0 + gate rig PASS, 577 tests + clippy green. Phase 4 now in progress — move the chain-walk state construction OFF the server onto the system worker pool reading the WAL/NATS stream, pool-side cache keyed by the immutable chain head + incremental tail-advance (server chain-walk + event-scan remain fallbacks). PROD GKE untouched; no gate default changed.
• 2026-06-19 — ✅ #115 Phase 2 implemented + kind-validated — one-level prev_event_id event chain (server#244 + noetl#667 merged). Each noetl.event carries prev_event_id (the immediately-previous event in causal order) + each noetl.command the issuing-event link, so per-execution events form a walkable singly-linked list followable pointer-by-pointer without scanning noetl.event (additive; no reader yet — Phase 3). The emit chokepoint emit_events stamps the event link from a per-execution chain-head watermark (ChainHeads) — one path covers drive events + command.issued + worker-lifecycle on both the gate-off INSERT and the gate-on publish, the materializer persisting it; the command link = the real step.enter/unblocking completion so cursor-fan-out bodies share their branch origin (§4.4). Server-only (no orchestrate-core change). Chain-correctness proven gate-ON across 6 executions (linear 13/13, loop 62/62, fan-out 25/25 with a real shared branch origin, sub-playbook 46/46, + Phase-1 output_select 31/31 & storage_tiers 55/55 bounded): each has 1 root, 0 dangling / 0 duplicate event prev, 1 head, pointer-walk == full sequence (no gaps), real-step command dangling=0; kind_validate_orchestrate_gate.sh PASS (sole-writer 25==25, 0 dup cycles, catalog0=0, lag 0); 573 lib tests + clippy green. PROD untouched; no gate default changed. Awaiting merge → ai-meta pointer bump; Phase 3 (chain-walk state builder) next.
• 2026-06-19 — 🐞 #114 oversized-command.issued offload shipped (server v3.29.5); refs_in_state consume side (#101) is the remaining off-server-drive cutover blocker. With #113's decode fix in place, 4 large-context fixtures wedged at a distinct second stall: under refs_in_state=false the off-server drive embeds the full resolved upstream context into the next command, so its command.issued event (~1.32MB) exceeded NATS max_payload → publish ack-timeout → wedge. Fix: a command context over NOETL_COMMAND_CONTEXT_MAX_BYTES (512KB) is offloaded to noetl.result_store with a {__context_ref__} marker; get_command/claim_command resolve it before the worker sees it (metrics context_offloaded/context_ref_resolved). Server #242 → v3.29.5 + rig phase 8 e2e#64. Kind gate-ON: off-server rig PASS (new test_oversize_command_context COMPLETED, max command.issued ctx 585B, offload+resolve fired, 0 __orchestrate__ event rows, materializer lag 0); every command.issued event <1MB across all fixtures; 6 of #113's 9 now COMPLETE. Chose ref-on-oversize over refs_in_state=true (candidate #1): a kind experiment proved refs_in_state=true fixes the state-bloat (kind_playbook_lease_expiry completes — drive-state 29KB vs ~1MB loop) but breaks the bulk-consuming fixtures (test_storage_tiers/test_output_select fail at the bulk step) because the worker render-time ref-resolution isn't implemented — so the default stays false. The remaining 3 fixtures + the cutover (#107/#111) now hinge on the refs_in_state consume side (#101) — __orchestrate__ drive-state bloat (17.4MB for storage_tiers) + the _ref/bulk-resolve gap. ai-meta → server 385d21f (v3.29.5) + e2e 9919392. No prod default flipped; prod is pre-#108 in-server drive (unaffected).
• 2026-06-19 — 🐞 #113 off-server drive — recover offloaded drive result + stop drive on cancel (server v3.29.4); #114 opened. Fixed the worker-driven drive stall when an __orchestrate__ result exceeds the 100KB inline budget (worker offloads it with only a reference.ref → server now resolves+decodes it via result_store.resolve, metric ref_resolved, instead of dropping → non-convergent re-loop) + the cancel non-stop facet (match underscore playbook_cancelled + ExecutionState::is_terminal terminal guard, no restart). Server #241 → v3.29.4 + rig e2e#63. Kind gate-ON proven (785KB result → ref_resolved→COMPLETED, 0 decode WARNs; cancel froze a drive-loop instantly; sole-writer lag 0). 5/9 #113 large-context fixtures COMPLETE; the other 4 hit a distinct oversized-command.issued (full upstream context embedded → >1MB NATS payload) stall → #114 (#113 stays open until all 9 close). ai-meta → server 1e844c1 + e2e 12b27e9. No prod default flipped.
• 2026-06-19 — 🚀 #103 GKE pre-flip PREP — prod images pushed, GMP monitoring live, manifests staged; NO traffic flip / NO PUBLISH_ONLY. Verified prod already-Rust (the #49 cutover is done; pre-#103 live images), both flip secrets present, monitoring = Google Managed Prometheus (not VM). Pushed server v3.29.3 + worker v5.35.0 to the prod AR (amd64); applied + verified GMP PodMonitoring (worker+server /metrics — the noetl ns had none) + materializer-lag Rules (up{namespace="noetl"}=4 live); staged the roll-forward manifests (not applied — they roll live workloads); runbook gained a "Production (GKE)" section + GMP managedAlertmanager pager stub. Operator-gated: roll images → materializer shadow → pager → flip. ai-meta e5b6d6c → ops 9edd9c4 (ops PR #197). No prod default changed.
• 2026-06-19 — 🛡️ #103 materializer-lag GUARDRAIL shipped — the pre-flip observability gate. The server was FLIP-READY; the remaining gate was a materializer-lag metric + alert. Worker #116 → v5.35.0 extends the JetStream lag poller to track the noetl_events/noetl_materializer consumer on an independent task → noetl_worker_nats_consumer_pending{consumer="noetl_materializer"} climbs even when the materializer loop is dead. Ops #195+#196: VMRule (backlog warning>200/critical>2000/growing + stall-under-gate + project-errors + absent-under-gate, stall guarded on backlog>0), worker /metrics VMServiceScrape (was unscraped), VMAlert enabled, Grafana dashboard, flip runbook noetl-cqrs-publish-only-flip.md. Kind-proven full cycle on the VM stack: green baseline (backlog 0) → induced lag (materializer fault-injected, events publishing under the gate) → gauge 0→684, alerts fire (backlog warning+critical + stall) → recover → drains→0 idempotently (0 dup/loss), alerts clear. ai-meta → worker b910341 (v5.35.0) + ops 2fcfa59 + worker-wiki 0030f30. PUBLISH_ONLY stays default-off.
• 2026-06-19 — 🎯 #103 server cutover COMPLETE — FLIP-READY. The 2 ExecutionService cancel/finalize sites now route through the emit_event chokepoint (server #240 → v3.29.3 + e2e rig #62); kind-proven both modes; no remaining synchronous server event writers under the gate. Flipping PUBLISH_ONLY on is now a staged operator decision.
• 2026-06-19 — 🎯 #104 off-server-drive × gate reconciliation PROVEN — the last real blocker before the PUBLISH_ONLY flip is operator-safe. The combination #103 left unproven (gate-on was only ever validated with the in-process drive) is green on kind: gate-ON (PUBLISH_ONLY=true) with the off-server drive (PLUGIN_DRIVE=true) + materializer sole writer → fresh exec + cursor fan-out → COMPLETED; server wrote 0 noetl.event rows (all 25 PUBLISHED — event_ingest_published_total=25), materializer materialized all 25 exactly once (25 rows == 25 distinct ids, 0 catalog_id=0, 0 dup cycles), drive dispatched=applied, read-your-writes held (the relocated trigger fires post-materialize → server rebuilds state from committed log before bounding the off-server drive input). Server #238 → v3.29.2 (76d29bb): cold-cache apply now rebuilds WorkflowState from the durable log (the #104 WAL-rebuild principle) instead of dropping the in-flight result on a server restart mid-drive — kind crash-recovery proof: hard-kill mid-drive → cold_rebuild metric+log fires → that exec COMPLETES with full event integrity. Committed e2e rig kind_validate_orchestrate_gate.sh (e2e#61). Regression green: gate-off + in-process (prod default) and gate-off + off-server. ai-meta → server 76d29bb + e2e 61f7a5c. Remaining before a safe flip: only the 2 ExecutionService cancel/finalize sites. No prod default changed.
• 2026-06-18 — 🎯 #108 (c) — the worker-driven orchestrator drive is now the DEFAULT; #108 CLOSED. Flipped NOETL_ORCHESTRATE_PLUGIN_DRIVE to default true (server#233, v3.28.0 → server@80cc0e6). Gated on a scale soak on kind (images built from the released tips, server v3.27.0 / worker v5.33.0): a single 694-drive cursor+fan-out run (test_pft_flow_v2 3×40) COMPLETED with __orchestrate__ rows in noetl.event = 0 (event_suppressed +2082) and all 694 drives claimed on the system pool (shared pool got only the 671 real-step commands), 0 errors; 5× concurrent self-contained cursor = 5/5 COMPLETED. Then deployed the flipped image with no env var — the default-on path reproduced the identical shape (361 drives, system-isolated, 0 burst); 15/15 regression fixtures green; the revert (=false) verified to fall back to the in-process drive (system delta 0). In-process trigger_orchestrator_inner kept as the fallback. ai-meta → server 80cc0e6 + worker 437b0be + server-wiki 0210012.
• 2026-06-18 — orchestrate drive isolated on the SYSTEM pool via pool affinity (#108 follow-up b, kind-validated). Server stamps execution_pool on the command notification (server#232 → server@846166b); worker declines (ACK+skip) notifications not for its pool segment (worker#114 → worker@e2162b7), so the drive runs on the dedicated system pool even under JetStream consumer-filter drift. No worker HTTP pending-poll — the NATS consumer is the only claim vector. Validated: __orchestrate__ claimed+executed on the system pool (3), zero on the default pool, simple_python COMPLETED; 553+196 tests green. Only (c) the deliberate default-flip remains.
• 2026-06-18 — the orchestrate meta-command touches noetl.event ZERO times (#108 follow-up a, kind-validated). dispatch_orchestrate_command stops writing command.issued to noetl.event; the command lives only in noetl.command, and claim_command/get_command fall back to it on a miss (noetl.event stays authoritative for normal commands) (server#231 → server@9438f3b). So __orchestrate__ writes 0 of its former 5 rows per drive — the directive that system playbooks keep only their own state is met. Validated: cursor+fan-out COMPLETED via the noetl.command fallback, 0 event rows, 20 real steps normal, 0 errors. Remaining: NATS affinity (ops) + default-flip.
• 2026-06-17 — system playbook events no longer burst Postgres + system-pool routing (#108 slice 4b, kind-validated). The __orchestrate__ meta-command is infrastructure, not a workflow step, so the server now skips persisting its lifecycle events to noetl.event (handle_event_inner + claim_command) (server#230 → server@6aef3a6). At scale they'd burst noetl.event/Postgres for no benefit. Validated: __orchestrate__ now writes only the lone command.issued (1 of 5 — 80% fewer rows); cursor+fan-out flow still COMPLETED. Drive routes to the system segment (true isolation pending a NATS-affinity ops fix; resilient via the pending-poll meanwhile). Follow-ups: eliminate the last command.issued (claim from noetl.command) + NATS affinity.
• 2026-06-17 — 🎯 the orchestrator drive runs OFF-SERVER on the worker pool (#108 slice 3, kind-validated). With NOETL_ORCHESTRATE_PLUGIN_DRIVE=on the server issues system/orchestrate (entry: run_state, args = the bounded WorkflowState) to the worker pool instead of evaluating in-process; the worker runs the drive, the server applies the result on the command's call.done (server#229 → server@465cdbb v3.23.0). Kind: test/simple_python drove start→end→COMPLETED through the round-trip (dispatched=2, applied=2, 0 decode_error, __orchestrate__ didn't leak as a step, playbook.completed). Default off, in-process fallback. Bug caught+fixed: output_b64 rides call.done not command.completed. Next: shadow→flip at scale, make drive the default, route to the system pool.
• 2026-06-17 — worker-driven cutover slice 2: apply_orchestration_result extracted + slice 3 designed (#108). The post-evaluate emission (events → commands → terminal) is extracted verbatim from trigger_orchestrator_inner into a reusable fn (server#228 → server@586aeae) so the worker-driven drive applies a worker-computed result identically. Behavior-preserving (553 tests green, clippy clean). Slice 3 (dispatch) designed + grounded: apply_event would phantom-step a meta-command, so the design uses a reserved __orchestrate__ step ignored in state, a flag-gated scheduler, apply-on-callback, and loop-prevention. Lands behind NOETL_ORCHESTRATE_PLUGIN_DRIVE (default off), kind-validated before the flip.
• 2026-06-17 — worker-driven cutover slice 1: configurable wasm guest entry (#108). The worker can now dispatch a named plug-in export (worker#113 → worker@04420d0): tool: {kind: wasm, plugin: {path, version, entry}} names the export (default run); the worker-driven orchestrator will use entry: run_state. invoke_bytes_with_entry + run_by_ref_entry/run_and_apply_by_ref_entry (originals delegate with run); test proves run→0xAA vs run_state→0xBB + missing-export error. Purely additive, no live change. Server scheduler+apply (the hot-path round-trip, default-off flag) next.
• 2026-06-17 — orchestrate plug-in drives the real workload identically, live (#108 slice 4). The orchestrator runs the plug-in alongside the in-process drive on every evaluation + diffs commands (server#227 → server@bd652ab) — a process-global wasmtime host (feature orchestrate-shadow) loaded from noetl.plugin_module at boot, gated NOETL_ORCHESTRATE_PLUGIN_SHADOW; in-process result authoritative. Kind-validated over the live 10×1000 PFT: noetl_orchestrate_shadow_total{result="match"} 529, ZERO mismatch/error, workers stable. Plug-in gains a state-input path (run_state); both build configs green (default no wasmtime). Slices 1-4 prove orchestrator-as-plug-in end to end; next is the worker-driven cutover.
• 2026-06-17 — system/orchestrate@1 registered + servable in a deployed server (#108 slice 3). The server bakes the orchestrate wasm into its image and seeds built-in system plug-ins into noetl.plugin_module on boot (server#226 → server@b21b589, kind-validated). New src/system_plugins.rs (pure dir-scan + sha256, unit-tested) + a wasmbuilder Docker stage + NOETL_SYSTEM_PLUGIN_DIR; in-process upsert (not the token-gated HTTP surface); digest-keyed hot-reload. Validated: GET /api/internal/plugins/system/orchestrate?version=1 → 200 application/wasm 1559093 bytes, ETag=digest, stale→409, baked sha256 == served digest. Next: kernel scheduler dispatches the now-registered plug-in.
• 2026-06-17 — system/orchestrate plug-in runs identically to native in wasmtime (#108 slice 2). A wasmtime shadow-diff (server#225 → server@ccec104) loads the built .wasm through a harness mirroring the worker host's invoke_bytes ABI byte-for-byte and asserts the wasm output equals the native drive over auth0 multi-arc when: routing (minijinja in wasm) + cold-start. Finding: command-set identity (parsed Value eq), not raw bytes — the context map serializes in insertion order (serde_json preserve_order ← upstream HashMap iteration, differs wasm32 vs host arch); the scheduler deserializes to Vec<Command>, so the command set is the bar. 2 unit + shadow-diff green; plug-in excluded; test-only. Next: catalog register/serve → kernel scheduler.
• 2026-06-17 — system/orchestrate WASM plug-in exists — drive core runs as a 0-import module (#108 slice 1). New standalone plugins/orchestrate/ crate (server#224 → server@10a629b) wraps the drive behind the worker plug-in ABI (input = JSON event-slice + playbook; output = JSON OrchestrationResult; data-plane = memory/alloc/run) and compiles to wasm32-unknown-unknown — the first non-trivial compiled system playbook. Feasibility risk retired: the .wasm has 0 imports (no WASI, no host render) — the whole drive incl. minijinja runs in-guest. Native parity test reproduces native evaluate byte-for-byte; 551 server tests green, crate excluded from the workspace. Next: worker-host shadow-diff → catalog register/serve → kernel scheduler (NOETL_ORCHESTRATE_PLUGIN, default off).
• 2026-06-17 — Orchestrator drive core fully wasm-resident — Event-ABI round #109 CLOSED. Slice 3 (server#223) moved orchestrator/evaluate from src/engine/ into noetl-orchestrate-core. All 6 drive modules (renderer, playbook model, commands, evaluator, state, orchestrator switch) now compile native + wasm32-unknown-unknown — the system/orchestrate plug-in seed (#108). evaluate reads the pure core::event::Event; server converts db::Event at the trigger_orchestrator boundary (slice-1 From). 122 core + 565 server tests green, 0 WASI imports on wasm32, clippy clean; cargo-chef image (v3.20.0) kind-deployed, PFT 10×1000 — full command lifecycle, 0 errors, 0 restarts. ai-meta → server bfd3f77 (internal refactor, stays v3.20.0).
• 2026-06-14 — Transfer tool: Snowflake↔Postgres both directions — #99 CLOSED. Both transfer arms implemented with full credential-alias resolution. tools v3.10.0 (tools#65) + worker v5.22.0 (worker#87) + e2e#58. SF→PG: $n::text::<udt> coercion + RFC3339 timestamp reformat; PG→SF: generated INSERTs. Full bidirectional data_transfer/snowflake_postgres fixture COMPLETED on kind against live sf_test account. tools → 4127b4b · worker → 6d97e7c · e2e → 94aa7f1.
• 2026-06-14 — Snowflake key-pair JWT validated end-to-end — #98 last external-tool gap closed; transfer step → #99. noetl-tools v3.9.0 / v3.9.1 / v3.9.2 (tools#62/#63/#64) — key-pair JWT auth (bypasses MFA) + User-Agent fix (code 391903) + SQL-API context-in-body + multi-statement split (codes 391911 + 000008). Worker bumped to v3.9.2 (worker#83–#86) + e2e#57 fixture cleanup. create_sf_database (CREATE DATABASE) + setup_sf_table (CREATE TABLE + INSERT) both COMPLETED via JWT on kind against the live sf_test account (NDCFGPC-MI21697). Transfer step fails (inline creds, no key-pair fields) — filed #99. ai-meta pointers: tools a216ab2 · worker 9d6b127 · e2e e191231.
• 2026-06-12 — #90 Phase 7 shipped — scale hardening; #90 CLOSED (all 7 phases complete), live proof green. Final phase: server v3.5.0 (server#189) POST /api/execute/batch (N→N, partial-failure contained) + opt-in exactly-once dedup window (noetl.subscription_dedup, bounded-by-age, race-safe, default off); worker v5.19.0 (worker#79) batch dispatch + dedup opt-in + per-subscription rate limits (deterministic token-bucket RateGovernor, fetch-side backpressure → source keeps backlog, no loss, subscription.rate_limited event); ops (ops#176) + e2e (e2e#48); no tools change → no crate cascade. Live on kind: batch 12→12 COMPLETED on the subscription pool + per-message traceparent; dedup duplicate→1 execution + subscription.message.deduplicated; rate-limit engaged + 10/10 → executions (no loss); direct-curl within/outside-window + dedup-off + batch partial-failure all green. ai-meta → server 7b217d8 + worker 7531f4a + ops 6db69b9 + e2e 203593b. #90 closed; follow-ups tracked: #91–#94 + tools#57.
• 2026-06-12 — #90 Phase 6 shipped — CLI local noetl subscribe + FileEventSink + local_disk spool (live local proof green). Added noetl subscribe <spec.yaml> (cli v4.11.0, cli#60, closes cli#59): a kind: Subscription listener run standalone in local mode — no k8s, no NATS-dispatch server for the listening itself — reusing the same noetl_tools source clients + directive engine + spool engine the in-cluster worker uses, emitting the same ExecutorEvent envelope to a local FileEventSink (one event/line JSONL → replayable trail). Local dispatch (RFC §5.3): in-process via PlaybookRunner (pure-local default) or POST /api/execute. local_disk spool (§8.6): circuit-breaker + buffer + ordered replay + idempotency + dead-letter against a local dir, circuit state in a local file. New src/subscribe/{mod,spec,sink,dispatch,runtime,spool}.rs + examples/subscribe/. cli-only — no tools change / crate cascade (the source+spool surface ships in noetl-tools v3.5.0; bumps the lock 3.0.0 → 3.5.0 via the executor's "3"). Tests: 12 subscribe + full bin suite (53) green, incl. a deterministic outage→spool→ordered-replay→idempotency proof on the real engine. Live (in-cluster NATS on kind): 5 msgs → received=5 dispatched=5 failed=0 (19-event JSONL trail); local_disk spool outage → 6 message.spooled (0 dispatched, no loss) → recovery → 6 message.replayed in order → drained to 0. Finding: the NATS source ignores URL-embedded user:pass (async-nats ConnectOptions) — specs use explicit user/password. ai-meta → cli 2fb3fb0 (v4.11.0); wiki cli subscribe. #90 stays open for Phase 7 (scale hardening, volume-gated).
• 2026-06-11 — #90 Pub/Sub + Kafka brought to live-E2E parity with NATS (validation gap closed). Stood up the two remaining subscription brokers in kind — Pub/Sub emulator (gcloud SDK image) + single-broker KRaft apache/kafka:3.9.1 — under noetl/ops (ops#170), and added bounded-drain fixtures + kind-validate runners under noetl/e2e (e2e#41). Both backends passed the same live bar as NATS: publish/produce 5 → bounded drain count=5 acked=true → execution COMPLETED → call.done/command.completed/playbook.completed event trail. No adapter code change needed — the pure-Rust kafka crate talks to Kafka 3.9 KRaft and the Pub/Sub REST backend works against the emulator as-is. The one fix: the <step>.output.<field> accessor never resolved (both when: arcs skipped → drain stalled); corrected to <step>.<field> in the fixtures + the latent ops subscription_drain.yaml example. Validated on server v3.1.0 + worker v5.15.2 + tools v3.2.0; cluster left on that clean released stack. ai-meta → ops 568a4ac + e2e 8d21e7a. #90 stays open (Phases 2–7 design-only).
• 2026-06-11 — #89 shipped — JSON null round-trips through {{ step }} (server fix, v3.0.6). #89 — null→undefined serialization — CLOSED. The #88 cursor fixture walked all 4 pages but its 4th check_pagination crashed: the terminal page's next_cursor: null, re-injected via the whole {{ fetch_page }} envelope, rendered as the JS token undefined (invalid JSON), so the consuming Python step received response as a str. Traced the corrupt command.issued args.response to the renderer that builds next-step inputs — the server orchestrator (src/template/jinja.rs::render_to_value), not the worker the issue blamed. json_value_to_minijinja maps JSON null→Value::UNDEFINED; minijinja's map repr emits undefined; render_to_value failed from_str and fell through to a raw string. The noetl-tools engine already had a | tojson retry for exactly this; the server's copy had diverged without it. Fix (server#177, v3.0.6) ports the retry. 5 new tests; 619 lib + 8 parity green; clippy clean. Kind-validated end to end on the live test-server (baseline 4th check_pagination error → fixed success; cursor collects 35, matching offset). ai-meta → server 8e17fbe. Standing direction honored — Claude wrote the Rust directly, no Codex.
• 2026-06-10 — #88 shipped — pagination fixtures read response.body.*; #89 filed. #88 — offset/cursor pagination fixture path — CLOSED. The Rust http tool nests the parsed JSON payload under body ({{ fetch_page }} → {body, headers, status_code}); the fixtures read response.get('data', {}), which resolved to {}, so has_more/next_cursor defaulted falsy and the loop exited after page 1 despite the correct post-#85 machinery. Confirmed the shape against a live http-tool result, then switched both check_pagination steps to response.get('body', {}) (e2e#40). Kind-validated: offset walks 0→10→20→30, users 10/10/10/5, validate_results success 35, playbook.completed COMPLETED; cursor path-fixed + walks all 4 pages (Mg==→Mw==→NA==→null, 35 events fetched) but the terminal page surfaced a distinct worker bug → #89 (worker serializes next_cursor: null as JS undefined when re-injecting {{ fetch_page }}, so the consuming Python step gets an unparseable str). Other pagination fixtures (retry/max_iterations/pipeline*/loop_with_pagination) share the same envelope-key assumption over /api/v1/assessments|flaky ({data, paging}) — flagged, left for follow-up. ai-meta → e2e 72a7525.
• 2026-06-10 — #87 shipped, #85 deferred (e2e sweep follow-ups #85/#87). #87 — multi-tool sibling references — CLOSED. task_sequence (the tool: [list] pipeline runtime) stored each sub-tool's result for the aggregated output but never injected it into the running context, so a later sub-tool's {{ <label>.<field> }} rendered empty — masked in quoted positions, a syntax error at or near "," in unquoted numeric SQL (save_edge_cases test_large_payload). Fix (tools#48, v3.1.1) injects each sub-tool's result under its label (synthetic .data self-ref); worker adopts via worker#69. Kind-validated on a worker built from the fix: save_edge_cases test_large_payload → record_count = 100 (no syntax error), save_delegation_test clean. ai-meta → tools 76f942a + tools-wiki 4962f8b + worker b97f642. #85 — workflow-arc loop re-entry — DEFERRED (kept open). Implemented the dispatch-guard layer (draft server#176): a back-edge detector (cycle + recency) re-enters a completed loop head, so the loop no longer hangs (608 lib tests + 5 new pass). But kind validation surfaced a second blocker — set: ctx.X loop variables are recomputed per orchestrator pass and revert to the workload default when the producing step is re-dispatched (a minimal counter-loop thrashes 0,0,1,0,1,2,…). Full multi-page pagination needs durable event-sourced ctx propagation across iterations — larger than is safe to land well-tested in one session; held as a draft, not merged. Standing direction honored: Claude wrote all Rust directly (no Codex).
• 2026-06-10 — #80 closed — container_callback chain green end to end. Fixing the watcher's missing curl (the literal #80 goal) surfaced two more layered bugs beneath it. Watcher image (ops#168): the manifest used the retired bitnami/kubectl:1.30.3 (removed from Docker Hub; the live cluster was patched to the bitnamilegacy archive) with a runtime apt/apk install step that never put curl on PATH → callback POST returned HTTP 000. Switched to alpine/k8s:1.30.3 (kubectl + jq + curl baked in), dropped the install hack. Server insert (server#173, v3.0.3): once curl worked the POST reached the server and 500'd — the container-callback handler inserted call.done via a stale query targeting an attempt column that doesn't exist on the deployed noetl.event; fixed to the working handlers::events column set. OOM path: the watcher only read Job-level conditions so failed_oom could never fire — added pod-level OOMKilled→failed_oom classification (ops#168); the completed_at fallback for failed Jobs used bare jq now (numeric epoch → HTTP 422), fixed to RFC3339 now | todate; and the e2e fixture's bytes(40MiB) was calloc-lazy (mapped to the zero page, never faulted in) so the container exited 0 — switched to a written-into bytearray that dirties pages and reliably OOM-kills (e2e#38). Verified the kind cluster actually enforces memory limits (120 MiB in a 32Mi pod → OOMKilled exit 137). Rebuilt the server image + reloaded into kind; kind_validate_container_callback.sh both probes GREEN — happy_path → succeeded (delta 1), oom → failed_oom (delta 1). This is the last blocker on the #43 container-callback chain. ai-meta → ops cacc513 + server 5d2cf58 (v3.0.3) + e2e 6aaf06e.
• 2026-06-10 — #79 closed — e2e kind-val runners back on the current noetl CLI surface. Both scripts/kind_validate_*.sh runners aborted immediately on error: unrecognized subcommand 'playbook' — they targeted the retired noetl playbook register/execute + noetl execution status/events verbs. The validation logic and the event taxonomy (step.enter / command.completed / node_name / the fan-in barrier) were intact; only the invocation layer had drifted. Fix (e2e#37): noetl register playbook --file, noetl exec <catalog-path> --runtime distributed --json (exec by metadata.path, not the bare name), noetl status <id> --json, and the event log over noetl query (no events verb today — rows wrap under .result, order by event_id since noetl.event has no timestamp column). Added a fail-fast CLI-surface guard to each runner. Validated on kind (server-rust v3.0.1 + worker-rust, :8082): fanout_reduce PASS start-to-finish with no manual workaround; container_callback drives register→exec→COMPLETED cleanly and stops at the metric-delta assertion because the deployed noetl-k8s-watcher image lacks curl (watcher.sh: curl: not found → HTTP 000) — a cluster-side watcher gap tracked on #80. Version-skew note: PATH binary is noetl 2.17.0, repos/cli submodule is v4.10.0; the targeted surface is identical across both, so the runners work on either (the binary lags the submodule by a major line — worth refreshing for parity, not required here). Pointer: e2e → a3594b3; e2e wiki: new Kind-Val Runners page.
• 2026-06-10 — #82 closed — GUI credential View/Edit recovered for pre-wallet records. The Secrets Wallet (#61) moved credential storage to forward-only envelope encryption; pre-wallet records now 500 on GET /api/credentials/{id}?include_data=true (Decryption failed: aead::Error), so the GUI View/Edit flow dead-ended on a generic toast (response shape unchanged). Fix (gui#36): View surfaces the real reason + points to Edit; Edit still opens with the list-row metadata (name/type/description/tags) + a warning banner and an empty-but-required data field, so re-entering the secret and saving re-seals the record under the current wallet — recovering it. Validated live against kind + the dev:kind UI on :3001. Also landed e2e#36 (duplicate workload probe-flag keys removed from tooling_non_blocking) and gui#35 (dev:kind convenience script). Pointers: gui → 8cacc9e (v1.11.1), e2e → 4a9ffbc.
• 2026-06-10 — #81 closed — noetl-server v3.0.2 fixes the container-tool command type contradiction. ToolSpec.command was Option<String> (scalar) but the container tool kind writes a K8s-Job-style array — an array failed the server's ToolDefinition untagged-enum match (400), a scalar was rejected by the worker's ContainerConfig.command: Option<Vec<String>>. Typed command as Option<serde_json::Value> (same as args); ToolCall::from_spec forwards it verbatim. 2 regression tests; clippy clean (server#172, v3.0.2). Kind-val GREEN end-to-end: server accepts the array command, worker creates the K8s Job, Job reaches Complete 1/1. Server pointer bumped (ai-meta → server bd36672). Chain counter-bump validation stays gated on #79 (runner CLI) / #43.
• 2026-06-09 — E2E sweep cleanup — noetl-tools v3.1.0 + noetl-server v3.0.1. Stripped the diagnostic tracing::debug! scaffolding added during the e2e triage, kept the production fixes: YAML when: true boolean + |tojson object-template fallback (tools#47), 64 MB result-store body limit + pipeline command/spec stash (server#171). Pointers bumped (ai-meta@316048c tools, @6590bd6 server); tracks #49. All 7 sweep playbooks PASS on Rust-only kind. Worker crates.io dep-revert deferred — v3.1.0 not yet on crates.io ([skip ci] release commit).
• 2026-06-08 — noetl-tools v2.24.2 clippy cleanup + noetl/server#22 closed. Cleared the clippy -D warnings CI gate on noetl-tools (15 warnings across 7 files; all mechanical lint fixes). Closed stale noetl/server#22 (Phase D orchestrator engine port — complete). noetl/server PR #167 (same clippy shape) opened, awaiting merge.
• 2026-06-05 — Rust-only regression rig — canonical v10 SQL + http config shapes. Swept ~30 self-contained e2e fixtures against the Rust-only kind stack and fixed three config-shape classes in noetl-tools: postgres command: alias + multi-statement SQL (tools#24, v2.18.3), a task_sequence→duckdb regression test (tools#25), and the duckdb command: alias + http params/headers/form non-string coercion (tools#26, v2.18.4). Worker adopted both (worker#50, worker#51). Newly GREEN: duckdb_test, json_serialization_save, duckdb_retry_query, pagination/{offset,cursor,max_iterations,pipeline}, retry_simple_config. Recovered the cluster first (server had latched into NATS not configured after a podman restart). Server-side follow-up noted: loop_with_pagination renders {{ execution_id }} empty in a multi-statement postgres command.
• 2026-06-05 — postgres-tool observability — real SQLSTATE errors. noetl-tools 2.18.2 (tools#21) + worker dep bump (worker#49): the postgres tool surfaces the real SQLSTATE + message instead of the opaque db error. Validated end-to-end — a bad query reports ERROR: relation "..." does not exist (SQLSTATE 42P01) in the call.error event. Closes the last follow-up from the credential/iterator saga.
• 2026-06-05 — iterator_save_test GREEN — full v10 + credential + iterator-pipeline surface validated. server#73 (v2.19.7) defers task_sequence _prev/_results refs at command-build so nested-pipeline templates render at runtime. iterator_save_test reaches playbook.completed and writes 3 rows to the real demo_noetl DB — the deepest v10 path (iterator → pipeline → _prev chaining → nested credential → postgres write). Closes the credential + iterator + pipeline chain (server#71, worker#46, worker#48, server#73).
• 2026-06-05 — Nested-pipeline credentials + template-timing finding. worker#48 (v5.11.3) — the worker now pre-resolves keychain aliases on task_sequence SUB-tasks; iterator_save_test's nested save_item postgres step connects to demo_noetl. Closes the credential-path chain (store → alias-key → nested resolution, all validated). Last iterator_save_test blocker found + filed: server#72 — the server pre-renders task_sequence {{ _prev.* }} refs (runtime-only) to empty → malformed SQL (a symptom the v2.19.5 Chainable change surfaced).
• 2026-06-05 — Keychain-credential path validated on Rust-only. Continuing R5 Tier 4, registered the pg_k8s postgres credential and probed the DB-backed fixtures. Surfaced + fixed a 3-bug chain in the keychain subsystem: credential store bound AES-GCM Vec<u8> to a TEXT column (server#71, v2.19.6); alias resolution read only the auth: key not v10's credential: (worker#46, v5.11.2). Proven: iterator_save_test's create_table connects + runs DDL against the real demo_noetl DB. Third bug — nested-pipeline credentials (task_sequence sub-tasks bypass worker resolution) — filed as worker#47 for a follow-up round. Session details
• 2026-06-05 — v10 control-flow runs end-to-end on Rust-only. Phase F R5 Tier 4 re-probe found + fixed 7 more bugs across the Rust stack (server v2.19.5 server#69 6 commits, worker v5.11.1 worker#44, tools v2.18.1). Four v10 fixtures now reach playbook.completed — start_with_action, end_with_action, loop_test, control_flow_workbook; actions_test correct-fails on a missing TEST_SECRET env. Root-cause chain: catalog SQL type drift → ToolSpec null-serialization → worker array-config drop → orchestrator end-step skip + task_sequence label-wrap → minijinja Lenient-vs-Chainable undefined → end-step trigger gate. Also: rust-analyzer workspace setup + rule (ai-meta@38287b7). Session details
• 2026-06-04 (late evening) — Rust-only e2e complete + legacy cleanup. Six interlocking server gaps closed in one iteration (#55–#60)
  • two noetl-tools fixes (#15, #16); worker dep bump; kind cluster legacy Python deployments retired. control_flow_workbook runs fully end-to-end on the Rust-only stack. Standing direction pinned: Rust-only focus, ignore Python tasks. Session details
• 2026-06-04 (afternoon) — Pipeline + failure termination + workbook resolution. Three server PRs landed together as v2.19.3 (#61, #63, #65).
• 2026-06-04 (morning) — EE-5 lax decode + workload + input alias. v2.19.1 + v2.19.2 — unblocked Rust worker → Rust server emission + canonical v10 playbook compatibility.
• 2026-06-04 (early morning) — Phase F R4-5 + R4 complete. N=2 shard kind validation script + ExecutionService refactor.
• 2026-06-03 — Phase F R4 series — DbPoolMap N+1 pool layer, AppState wiring, per-execution handler cutover, cluster-wide list fan-out.
• 2026-06-02 (afternoon) — Architecture pivot: rest of migration moves to system playbooks. Closed #30, #45; promoted #46.

See Releases for the per-repo release log with links to GitHub Releases pages.

Recent:

• 2026-10-04 — noetl/noetl pypi-v5.1.1 — 🔒 the shipped wheel's RustSec count goes 5 → 3. h2 0.4.19 and rustls 0.23.45 clear two advisories Dependabot reported as zero; event-listener 5.4.2 and rustls-webpki 0.103.15 ride along. Verified on the published artifact (4 files sha256-matched, sdist Cargo.lock read, 5.1.0's lock scanned first as a positive control). Remaining 3 are documented deferrals (#402).
• 2026-10-04 — noetl/noetl pypi-v5.1.0 — 📦 the artifact finally matches the lock — the 7 cleared Rust advisories reach the wheel users install (#395). ⚠⚠ Verified by upload counts, because twine --skip-existing makes a publish-nothing run go green.
• 2026-10-04 — noetl/cli v5.0.3 — 📦 arm64 debs are built and indexed; the apt Release file gains its 8 identifying fields and stops listing itself (#390).
• 2026-09-04 — noetl/server v3.101.0 — 🔧 scoped, inspect-only tier repair endpoint (server#399). Re-mirrors only the event IDs the comparator reports absent; both comparisons under ParityRecording::Inspect so a repair does not inflate the counter its own alert reads. Inert, not run against prod (prod stays v3.100.4).
• 2026-09-01 — noetl/worker v5.129.1 — 🔥 bounded manifest retention + ingest failure counters, the two fixes for the 8-hour silent dispatch outage. Deployed: manifest 871 → 169 MB.
• 2026-08-29 — noetl/server v3.92.0 → v3.94.0, noetl/worker v5.124.0, noetl/cli v4.22.0 — 📐 the catalog gets its own log tier, relation and bulk loader; #307's projection fold becomes measurable.
• 2026-08-11 — noetl/worker v5.115.3 — ⏱️ the cooldown gates RETRY, never BELIEF. is_reachable() consulted the cached-negative deadline, so a lapsed window re-promoted a dead tier with no evidence; belief moved to a flag only a real successful op sets. Mutation-checked twice (worker#262, #257). Inert, not deployed.
• 2026-08-05 — noetl/server v3.64.0 → v3.69.0, noetl/worker v5.98.0 → v5.99.1, noetl/gateway v3.8.0 — 📊 the metrics surface becomes readable when nothing has happened: *_build_info{version} in all three binaries, publish-skip reasons and all seven sweep outcomes pinned at 0, and the dead NATS + JetStream metrics deleted. Prod untouched.
• 2026-08-03 — noetl/worker v5.92.0 — 🛡️ the writer's SIGTERM seal now covers the events log, and ingest is actually stopped first. worker#211 (ddb4de8): fail-loud internal bus source (closes #221), sequenced seal + confirmed stop-ingest (#209 graceful half), face supervision. The release gate caught that the first seal attempt never actually closed the listener — Notify::notify_waiters() stores no permit and the acceptor registers on its first poll, so an early shutdown lost the signal (24/30 runs under contention; lsof showed the writer's own acceptor still LISTENing). Fixed with a watch + an ack after the fd is dropped. Deployed to shastaratech prod by digest @sha256:ad1e96063bf5…81725; a controlled SIGTERM sealed both hosts in 568 ms with every cursor resuming clamped=false. ⚠️ semantic-release then pushed 5.91.3 over it on main (#224).
• 2026-07-30 — noetl/server v3.58.3 + noetl/worker v5.81.3 — 🏁 the EHDB command bus goes faster than NATS, and learns to survive a writer restart. #205: group commit + off-lock fsync in the writer (ehdb#301 03e94be) + pipelined publish (server#289, worker#195) — prod dispatch p50 138.5 / p99 181.0 ms vs the 338 ms NATS baseline (~2.4×). #208: keepalive + negotiated coordinator heartbeat, durable committed cursor clamped to the reopened tip, seal on SIGTERM (ehdb#302 86a24f9, worker#196, server#290) — writer deleted mid-load → 38/38, 0 dup / 0 loss, ~2.7 s redial. These are the images now running the production command bus.
• 2026-07-28 — noetl/worker v5.81.1 + noetl/server v3.58.1 — 🐛 the #203 feed-delivery fix: the command-bus writer now assigns the feed ordering key (append_writer_assigned) instead of trusting the producer's sparse snowflake id, so an out-of-order append can no longer land below the follower cursor and be silently dropped (ehdb#300 → ehdb d4b6235; worker#193 + server#287). Kind before/after on an identical 40-concurrent burst: 17/40 → 41/41. These are the images now running the EHDB command bus in production.
• 2026-07-26 — noetl/worker v5.81.0 — 🪝 #199 Slice C auto-sink observe-only (worker#192): the auto-sink loop observes and writes nothing; the write slice is held for design sign-off. Default-off.
• 2026-07-26 — noetl/worker v5.80.0 — 🪝 #199 Slice A sink-confirmation signal (worker#190): an author-declared sink: true connector step marks its execution pending-sink on dispatch and confirm_sunks on success. Author-declared, never inferred (a postgres/http/transfer step can be a read — classifying by kind would confirm-sink on reads and release un-sunk context); fail-safe is to under-confirm. Default-off (NOETL_SINK_GATE_EVICTION).
• 2026-07-25 — noetl/worker v5.79.0 — 🪝 #199 Slice B durable-segment GC gate (worker#191): the reclaim pass consults SharedWalIndex::any_sink_pending() and defers while any execution is pending-sink. Behavior-neutral, default-off.
• 2026-07-24 — noetl/cli v4.21.0 — 🐍 Stage 2 of the PyPI wheel: the duckdb crate (features = ["bundled"]) gated behind the duckdb-integration feature so the wheel matrix builds without compiling the DuckDB C++ amalgamation (cli#74). Distributed artifacts (tarballs, deb, Homebrew, Dockerfile) keep the engine explicitly. This is the CLI crate compiled into the published noetl 5.0.0 wheel (noetl --version → 4.21.0).
• 2026-07-24 — noetl/cli v4.20.0 — 🐍 Stage 1 library extraction (cli#73): src/main.rs becomes a shim over src/lib.rs exposing cli_main(argv) -> i32 (the full 24-subcommand tree) + re-exported PlaybookRunner/RunOutcome. Verified byte-identical to 4.19.0. This is what the wheel links, not a reimplementation.
• 2026-07-26 — noetl/noetl 5.0.0 on PyPI — 🐍 the NoETL CLI shipped as a Python wheel, the Rust CLI bound via PyO3 (abi3-py39): noetl console script + import noetl over the real PlaybookRunner. 4 artifacts (macOS arm64, Linux x86_64/aarch64, sdist); Windows dropped (the CLI depends on nix/POSIX), macOS-Intel covered by the sdist. https://pypi.org/project/noetl/5.0.0/ (noetl#692, tag pypi-v5.0.0).
• 2026-07-23 — noetl/server v3.58.0 + v3.57.0 — 🧱 EHDB write-behind-cache boundary: command-input tiering (server#284) and permanent-log-lean (server#283) — an over-floor inline step result / command input is stripped out of the permanent log at the events_project materializer into a byte-source reference. Both default-off (NOETL_PERMANENT_LOG_LEAN).
• 2026-07-14 — noetl/tools v3.25.0 + noetl/cli v4.17.0 — 🧩 provider tool round 5: the noetl provider {plan,drift,orphans,adopt} CLI verbs (Terraform-style CLI over the tool, local mode; glue over ProviderTool::execute+provider_state+to_tools_config) (cli#68, a2e5b4d) + the query-tier fetch adapter extract_facts_from_tier_records (unwrap the payload string + recursive fact search — fixes the round-3/4 fixed-path bug where the fold silently returned nothing against real event data on 3.23/3.24) + stack-scoped detect_orphans_scoped/detect_stack_conflicts (tools#90, 5a06c7e). noetl-executor unchanged at 0.7 (zero executor/ files touched). crates.io tarball-verified + released-binary proof (noetl-v4.17.0-darwin-arm64, all 4 verbs GET-only zero-writes). Follow-up 3 (first-class event type) DEFERRED. ai-meta tools→5a06c7e + cli→a2e5b4d; #189 OPEN. Mocks only — no prod/kind/live-cloud.
• 2026-07-13 — noetl/tools v3.24.0 + noetl/cli v4.16.0 (+ noetl-executor 0.7.0) — 🧩 provider tool round 4: reconcile policy (report new-default / enforce / adopt) + confirm-gated drift adoption (adopt = accept live actual as new desired, read/GET-only, blind+stale REFUSED; covers update-in-place + import) (tools#89, c172efb), AND local-mode reconcile/known_desired passthrough + executor 0.6.0→0.7.0 (cli#66, 7557dde). All three crates crates.io-verified. Behavior change (user-confirmed): converging needs reconcile:enforce. ai-meta tools→c172efb (c6790983) + cli→7557dde (e4d3a32c); #189 OPEN.
• 2026-07-13 — noetl/cli v4.15.1 — 🛠️ release-cli crate-publish FIXED: bumped noetl-executor 0.5.0→0.6.0 (+ bin dep "0.5"→"0.6") so the Tool::Provider variant ships to crates.io. The publish-crate job had failed on v4.14.0 + v4.15.0 (crate channel stuck at 4.13.0); full release run 29290603328 GREEN — executor 0.6.0 + noetl 4.15.1 now live on crates.io, binaries + .deb on the v4.15.1 release. apt + homebrew still SKIP (missing APT_REPO_PUSH_TOKEN / HOMEBREW_TAP_PUSH_TOKEN) but now emit a ::warning::. cli#67, ai-meta cli→5bb1705 (7d58e7d); #190 OPEN pending the two repo secrets.
• 2026-07-13 — noetl/tools v3.23.0 + noetl/cli v4.15.0 — 🚀 provider tool round 3: confirm-gated destroy (*.delete/services.disable/*.iam.remove_binding — live-resolved plan + SHA-256 plan_digest; blind apply + stale digest REFUSED; no wildcard) + EHDB ownership projection (secret-free provider_fact + pure provider_state fold over the raw eventlog tier, not the secret-free /api/ehdb/events read-model) + config-level endpoint override (tools#88, aa32938), AND local-mode endpoint/stack/confirm passthrough + re-pin to noetl-tools 3.23.0 (cli#65, 94cc72a; local-mode proof green on 3.23.0). ai-meta tools→aa32938 (f6eeb479) + cli→94cc72a (cbc7be99); #189 OPEN, round-4 = drift adoption. Mocks/wiremock only — no prod/kind/live-cloud, no real destroys.
• 2026-07-13 — noetl/tools v3.22.0 + noetl/cli v4.14.0 — 🚀 provider tool round 2: apply-mode idempotent GET-first converge for the 4 ensure/ensure_binding actions (+ services.enable/projects.link, re-run = no-op) + bounded inline LRO polling (tools#87, eb3d8c7), AND kind: provider dispatches through noetl exec --runtime local — same tool the worker runs, re-pinned to noetl-tools 3.22.0 (cli#64, 2d46013; supersedes cli#63, closed). ai-meta tools→eb3d8c7 (6cd1ba38) + cli→2d46013 (32a8fb86); #189 OPEN, both design forks DECIDED for round 3.
• 2026-07-13 — noetl/tools v3.21.0 — 🆕 new kind: provider cloud provider tool (Google REST MVP; Cloud Resource Manager v3 / Cloud Billing v1 / Service Usage v1 over REST + GcpAuth ADC, zero new heavy deps; plan/apply + explicit-auth-for-mutations) (tools#86, cd9d71c; ai-meta e9b3e14e; #189 OPEN)
• 2026-07-04 — noetl/server v3.52.0 — 🔐 #169 Auth0 JWT RS256/JWKS signature verification (server#276) shipped + deployed to prod in shadow (NOETL_AUTH_VERIFY_SIGNATURE, default off; also carries #166 Phase 5 dark). Prod v3.50.0→v3.52.0.
• 2026-06-29 — noetl/server v3.48.0 + noetl/worker v5.48.0 — ⚡ #156 off-server drive tail-attach (server#268 ddca5c4 + worker#142 12710b6): server attaches the per-execution event tail to the __offserver_build__ dispatch, worker applies it to its WAL index before building → per-hop O(execution) not O(global log), removing the drain-lag 8s reconcile cliff. Flag NOETL_OFFSERVER_ATTACH_TAIL (default off). DEPLOYED on prod flag-off (behavior-neutral) then flipped on; 5 Muno turns COMPLETED identical correct widgets, 90 drives attached O(tail), 0 restarts, reconnect 0/30s; 1-step rollback. ai-meta pointer f2b5666
• 2026-06-28 — noetl/worker v5.47.3 — 🚑 #157 hotels-turn drive wedge fix (worker#141): __orchestrate__ drive result stays inline regardless of size, so a render-hop result over the 100KB budget no longer offloads to a result-store ref the server can't resolve under NOETL_RESULT_STORE_DUAL_WRITE=false (#104) — Muno hotels turn now COMPLETES + returns a hotel_list widget; deployed both pools (0 restarts), ai-meta pointer 60938f4
• 2026-06-27 — noetl/server v3.47.0 — 🚀 #104 result_store dual-write retirement gate (NOETL_RESULT_STORE_DUAL_WRITE, default on = behavior-neutral) (server#267); flipped OFF on prod — result_store writes retired, #104 tier authoritative, data frozen
• 2026-06-26 — noetl/tools v3.19.0 — 🆕 G1 container GPU placement (node_selector/tolerations/volumes) + G2 poll-to-terminal helper (tools#82); SLM platform foundation #144/#145, review-only
• 2026-06-26 — noetl/worker v5.47.0 — 🆕 G2 watcher-free poll completion fallback (NOETL_CONTAINER_COMPLETION_POLL, default off) (worker#139); #145, review-only
• 2026-06-26 — noetl/server v3.46.0 — 🆕 G3 versioned model/dataset/eval registry (NOETL_REGISTRY_ENABLED, default off) (server#266); #146, review-only
• 2026-06-23 — noetl/worker v5.46.0 — 🪶 #104 OQ5 Option A: producer-staged result tier (NOETL_RESULT_PRODUCER_STAGE, default off) — decouples the over-budget tier write from result_store; materializer skip-on-exists.
• 2026-06-23 — noetl/server v3.45.0 — 🚀 #104: WI/ADC GCS auth (3-mode none/static/adc) MERGED + ROLLED TO PROD as WI KSA noetl-server-rust (@sha256:d3cbf1ad…); result-tier GCS ENV applied, all tier-enable flags OFF → tier inert; off-server cutover stayed healthy, 0 restarts (server#265, fad5d8a).
• 2026-06-23 — noetl/server v3.44.0 — 🪶 #104 Phase F: result-tier GC sweeper (NOETL_RESULT_TIER_GC, default off) — conservative dry-run-first, never deletes a live-referenced object (server#264, 341b614).
• 2026-06-23 — noetl/worker v5.45.0 — 🪶 #104 Phase F: result-tier DR re-derive (NOETL_RESULT_TIER_DR, default off) — materializer verify-and-repair, byte-identical rebuild of a missing/corrupt tier object (worker#131, dd07016).
• 2026-06-23 — noetl/worker v5.44.0 — 🪶 #104 Phase E: side-effect durability barrier (NOETL_SIDE_EFFECT_BARRIER, default off) — adopt-only, side effects fire exactly once across re-drive; depends on published noetl-tools 3.17 (worker#130, d696f7e).
• 2026-06-23 — noetl/tools v3.17.0 — 🪶 #104 Phase E: registry::kind_is_side_effecting side-effect classifier (conservative default true; only noop/rhai false) (tools#78, 1d49dd5; members noetl-directives + noetl-locator re-published first).
• 2026-06-23 — noetl/server v3.43.0 — 🪶 #104 Phase D: mint-authoritative flag (NOETL_RESULT_MINT_AUTHORITATIVE, default off) + result_store dual-write counter (noetl_result_store_dual_write_total) — the reversible dual-write fallback leg of the minting flip; no Cargo.toml change (resolves noetl-locator 0.1.1 from the registry) (server#263, 6f6b9ef).
• 2026-06-23 — noetl/worker v5.43.0 — 🪶 #104 Phase D: the minting flip — NOETL_RESULT_MINT_AUTHORITATIVE (default off) makes the materializer the authoritative tier writer + resolve-by-URN the primary consume path, with the dual-written result_store as the fail-safe fallback; new noetl_worker_result_mint_authoritative_total{path} (tier | legacy_fallback) (worker#129, be6863a).
• 2026-06-23 — noetl/server v3.42.0 — 🪶 #104 Phase C: GCS object backend + cell-endpoint registry + GET /api/internal/cells (resolve-by-URN read side; default-off) (server#262, c2d5ca9).
• 2026-06-23 — noetl/worker v5.42.0 — 🪶 #104 Phase C: resolve-by-URN read path + fixes B/B1 (references-in-state behavior, flatten_single_tool_result, closes OQ6; adds published arrow=53, resolves noetl-tools 3.16.0; 38 unit tests) (worker#128, 7971041).
• 2026-06-22 — noetl/tools v3.16.0 + noetl-locator v0.1.1 — 🪶 #104 Phase B: ResultCoordinates::parse/from_locator (the inverse of logical_uri); additive → member bumped 0.1.0→0.1.1, published (tools#77).
• 2026-06-22 — noetl/server v3.41.0 — 🪶 #104 Phase B: ensure the sibling noetl_result_materializer durable consumer at stream-birth (own ack cursor) (server#261).
• 2026-06-22 — noetl/worker v5.41.0 — 🪶 #104 Phase B: shadow Feather result tier — separate noetl_events consume-loop writes over-budget results (tabular → Feather, non-tabular → JSON) at the derived §7 key; gated NOETL_RESULT_MATERIALIZER_ENABLED (default off) (worker#127).
• 2026-06-22 — noetl/server v3.40.0 — 📦 #104 Phase A: accept the canonical result URI behind NOETL_RESULT_URI_ACCEPT (default off), parsed via the slim noetl-locator 0.1.0 (heavy graph absent from the control plane) (server#260, c89d078).
• 2026-06-22 — noetl/tools v3.15.0 + new crate noetl-locator v0.1.0 — 📦 #104 Phase A: extract the slim dependency-free noetl-locator crate (pure std; ResourceLocator/ResultCoordinates/shard_key/CellPlacement/legacy parse) so the control-plane server parses the result URI without noetl-tools' heavy graph; noetl-tools re-exports it as noetl_tools::locator (tools#76, dc0c5d8; noetl-locator 0.1.0 published to crates.io).
• 2026-06-22 — noetl/worker v5.40.5 — ✅ #95: adopt noetl-tools 3.14.2 (postgres temporal/identity serialization); Cargo.lock pin 3.14.1→3.14.2, no worker source change; built noetl-worker-rust:v5.40.5 @sha256:45212dbe… + rolled by digest to prod under the live off-server cutover (worker#126, da24952).
• 2026-06-22 — noetl/worker v5.40.4 — ✅ #127: adopt noetl-tools 3.14.1 (task_sequence per-sub-task context optimization); Cargo.lock pin 3.14→3.14.1, no worker source change; built noetl-worker-rust:0afbf5c + rolled to prod under the live off-server cutover (worker#125, 0afbf5c).
• 2026-06-22 — noetl/tools v3.14.2 — ✅ #95: postgres pg_value_to_json now serializes tz-naive timestamp (NaiveDateTime) + date/time/uuid/numeric (lossless binary-wire decode)/bytea instead of returning null; the auth0_login expires_at: null repro (tools#75, 6d9b674).
• 2026-06-22 — noetl/tools v3.14.1 — ✅ #127: behavior-preserving task_sequence per-sub-task context reuse — render_value builds the proxied minijinja context once + build_context_with_overlay skips the redundant to_template_context() clone; micro-bench −61.6%/2.6× per-sub-task context cost (tools#74, c8656c1).
• 2026-06-22 — noetl/server v3.39.6 — ✅ #123: surface a non-iterable loop in: as a terminal playbook.failed — the off-server drive decodes the {"error":…} envelope (decode_orchestrate_error, metric noetl_orchestrate_drive_total{stage="drive_error"}) instead of silently wedging (commands=0); orchestrate-core prefixes the offending step name onto the evaluate_loop error (server#258, 7f109a9, squash 275b914). Code-only (worker stays v5.40.3); 600 server + 135 orchestrate-core tests + clippy; kind-validated prod-exact (absent iterable → FAILED with a clear message, valid [1,2,3] still COMPLETED).
• 2026-06-22 — noetl/server v3.39.5 — ✅ #121 second-half: gate BOTH off-server-drive sites in trigger_orchestrator_inner on should_publish so system/* execs drive server-built run_state (they INSERT to noetl.event, never enter the WAL); regular execs keep the off-server path (server#257, c421273, squash 54ac277). Server-only (worker stays v5.40.3); 612 tests + clippy; kind full-gate system/scheduled_cleanup BEFORE 9× WAL chain incomplete → AFTER COMPLETED 0 loops.
• 2026-06-21 — noetl/server v3.39.4 — ✅ #121 first-half (partial): link gate-off command.claimed through ChainHeads + don't route system/ execs to the off-server WAL drive (server#256, 77aaa06)
• 2026-06-21 — noetl/worker v5.40.3 — ✅ #125 + #126 adopt noetl-tools 3.14.0 (task_sequence control flow + http body data-shape fixes); 10×1000 batch pft_flow_test clean (worker#124, 6dd3449).
• 2026-06-21 — noetl/tools v3.14.0 — ✅ #125 task_sequence honours do: jump/break/retry (tools#73, 638c3c6).
• 2026-06-21 — noetl/tools v3.13.1 — ✅ #126 http tool body under data (Python-era contract restored; back-compat body alias) (tools#72, 8dd0e1f).
• 2026-06-21 — noetl/server v3.39.3 — ✅ #124 distributed task_sequence forward set:/sibling bindings no longer render empty; defer sub-task templates with unresolved vars to the worker re-render (server#255, 365d3be).
• 2026-06-21 — noetl/server v3.39.2 — ✅ #120 reduce barrier no longer deadlocks on open/asymmetric loop joins; runtime liveness filter in the orchestrate-core barrier (server#254 28e8950)
• 2026-06-20 — noetl/server v3.39.0 — ✅ #116 program-scale step 2: execution-affinity single-owner WRITE ORDERING (multi-replica gate-ON validated) — closes the chain-fork race step 1 left open. Affinity routes every trigger for an execution to the replica that ShardConfig::owns it (non-owner forwards a reverse-proxy POST); owner's single-process drive lock + in-memory ChainHeads make the read→advance atomic, no distributed lock; KV = genesis/handoff vehicle (kv_remote_hit→0). src/affinity.rs; NOETL_EXECUTION_AFFINITY/NOETL_PEER_URL_TEMPLATE/NOETL_SHARD_INDEX_FROM_HOSTNAME (all default off); metric noetl_execution_affinity_total{outcome} (server#252 5e00d0a). 2-replica gate-ON kind PASS: chains roots=1/dangling=0/walk==total (NO fork), forwarded_ok +9, never-scan + sole-writer across replicas. Follow-up #117 (off-server from_events spine event_id-order wedges fan-in under inversion).
• 2026-06-20 — noetl/server v3.38.0 — ✅ #115 program-scale step 1: multi-replica coherence DATA LAYER — NOETL_REPLICA_COHERENCE=nats_kv (default local, prod unchanged) backs ChainHeads + ExecDescriptor with JetStream KV buckets (head CAS + descriptor CAS merge); src/coherence.rs; metric noetl_replica_coherence_total{structure,op,outcome} (server#251 8f39a79). Kind: single-replica parity with local; 2-replica cross-replica resolves proven. Necessary-not-sufficient → execution-affinity STAGED (2+ replicas still fork the chain).
• 2026-06-20 — noetl/server v3.37.0 — ✅ #115 Phase 5 atomic-working-item context (tenet 6): input_binding + NOETL_ATOMIC_ITEM_CONTEXT (default off) (server#250 a96ade8)
• 2026-06-20 — noetl/worker v5.40.0 — ✅ #115 Phase 5 forward the atomic-item-context flag onto the off-server from_events drive (worker#121 2484d17)
• 2026-06-20 — noetl/server v3.36.0 — ✅ #115 Phase 6 retire the hot-path noetl.event read class; the table is AUDIT-ONLY (server#249, b71ca1d): NOETL_EVENT_READ_PATH=event_scan|audit_only (default event_scan, prod unchanged). Under audit_only the remaining lifecycle readers (get_catalog_id, inherit_parent_trace, dedup-audit + container-callback catalog/existence) serve from the in-memory ExecDescriptor; cold → noetl.command (synchronous queue) — never a noetl.event scan. Proof metric noetl_event_hotpath_reads_total{site,outcome}. Gate-ON kind-validated: hot-path scan Δ0 + drive state_build_total/event_scans Δ0 ⇒ ZERO noetl.event scans on the hot path, end-to-end; audit/replay still work; 585 tests + clippy green. RFC never-scan end state (tenet 3) reached under the flag.
• 2026-06-19 — noetl/worker v5.37.0 — ✅ #115 Phase 4 off-server state-builder kernel + WAL shadow loop (worker#118, fef961c): pool-side per-execution chain index sourced from the noetl_events WAL; chain_walk() head→root spine in event_id order (parity by construction); cache keyed by the immutable chain head (CacheHit / Incremental tail-advance / ColdRebuild). Live WAL shadow loop (NOETL_STATE_BUILDER_SHADOW, default off) + metrics. Gate-ON kind-validated (993 WAL events, 0 noetl.event scans, 28 cold + 21 incremental); 8 unit tests + clippy green; default off. Drive cutover staged.
• 2026-06-19 — noetl/server v3.33.0 — ✅ #115 Phase 4 NOETL_STATE_BUILDER=offserver|server flag scaffold (server#246, 3e6006d, default server): the server-side flag for the off-server state-builder drive cutover (staged). 2 config tests; no prod default changed.
• 2026-06-19 — noetl/server v3.32.0 — ✅ #115 Phase 3 chain-walk state builder (flagged, default-off) (server#245, ai-meta pointer bumped): behind NOETL_STATE_BUILD_MODE=chain_walk the drive reconstructs WorkflowState by following the one-level prev_event_id chain head→root (in-memory ChainHeads head + (execution_id,event_id) PK lookups — never a WHERE execution_id scan) → same from_events (parity by construction); event-scan kept as the default + fallback (cold-head / lag / non-genesis). NOETL_STATE_BUILD_PARITY_CHECK shadow-builds both ways in one REPEATABLE READ snapshot. New metrics noetl_state_build_total{mode,outcome} / _event_scans_total (no-scan proof) / _chain_hops / _parity_total{result}. Gate-ON kind-validated: parity 41/41 MATCH, scans=0 / 1064 hops / 0 fallbacks, all topologies COMPLETE, sole-writer + lag-0, 577 tests + clippy green. No prod default changed.
• 2026-06-19 — noetl/server v3.31.0 — ✅ #115 Phase 2 one-level prev_event_id event chain (server#244, ai-meta afdb365): each noetl.event/noetl.command carries the chain link, stamped at the emit chokepoint from a per-execution chain-head watermark (ChainHeads), covering both gate paths + the materializer. Additive; nothing reads it yet (Phase 3). Kind-proven walkable/1-root/no-gap/no-scan across 6 gate-ON topologies; 573 tests + clippy green. Companion DDL noetl/noetl ecd16a2 (noetl#667). No prod default changed.
• 2026-06-19 — noetl/server v3.30.0 — ✅ #115 Phase 1 surface _ref/_store on kept refs + refs_in_state default true (server#243): consume-side accessors for {{ step._ref }} lazy-load + storage-tier predicates; references stay out of state/commands by default (worker selective-resolve landed in worker#117 v5.36.0). Closed #113 + #114; kind gate-ON all 9 stalls COMPLETE.
• 2026-06-19 — noetl/server v3.29.4 — 🐞 #113 off-server drive: recover offloaded drive result + stop drive on cancel (server#241, rig e2e#63). apply_worker_orchestration resolves+decodes an offloaded __orchestrate__ result (over the 100KB inline budget → durable reference.ref) instead of dropping it → non-convergent re-loop (metric ref_resolved); cancel now matches underscore playbook_cancelled + a terminal guard evicts the orch-cache (no restart). Kind gate-ON proven; 5/9 #113 fixtures COMPLETE, the other 4 hit a distinct oversized-command.issued stall → #114. No prod default flipped.
• 2026-06-19 — noetl/server v3.29.5 — 🐞 #114 offload oversized command context: a command.issued context over NOETL_COMMAND_CONTEXT_MAX_BYTES (512KB) is offloaded to noetl.result_store with a {__context_ref__} marker, resolved in get_command/claim_command (server#242, rig e2e#64) — published events stay under NATS max_payload, no publish-wall wedge. Kind gate-ON: rig PASS, all command.issued <1MB, 6 of #113's 9 fixtures COMPLETE; remaining 3 + cutover gated on the refs_in_state consume side (#101). No prod default changed.
• 2026-06-19 — noetl/server v3.29.4 — 🐞 #113 recover offloaded drive result + stop drive on cancel: apply_worker_orchestration resolves+decodes an over-budget __orchestrate__ drive result via result_store.resolve (metric ref_resolved) instead of dropping it → non-convergent re-loop; cancel matches underscore playbook_cancelled + a terminal guard evicts the orch-cache (server#241, rig e2e#63). No prod default changed.
• 2026-06-19 — noetl/server v3.29.3 — 🎯 #103 cutover COMPLETE, FLIP-READY: the 2 ExecutionService cancel/finalize writers route through the emit_event chokepoint (server#240, e2e rig e2e#62) — the last synchronous server noetl.event writer under the gate is closed. Kind-proven both modes (gate-off byte-identical INSERT; gate-on PUBLISHED + materializer sole writer + terminal state + 0 loss/dup). All three flip blockers closed → PUBLISH_ONLY flip is a staged operator decision. Default-off; no prod default changed.
• 2026-06-19 — noetl/server v3.29.2 — off-server-drive × gate crash-recovery: cold-cache apply rebuilds WorkflowState from the durable log instead of dropping the in-flight drive result (server#238, refs #104/#103). Unblocks the PUBLISH_ONLY flip (off-server drive × gate now kind-proven). Confined to the cold branch; no prod default changed.
• 2026-06-19 — noetl/tools v3.13.0 + noetl/worker v5.34.0 — #103 ack-after-materialize durability: deferred ack-after-processing capability (tools#71: AckMode::Defer + $JS.ACK durable handles + ack/nack/term) + in-process CQRS materializer consume-loop (worker#115: drain→project→ack-only-on-success, redeliver on failure) + system-pool wiring (ops#194). Kind fault-injection: gate-on sole-writer loss=0 across a mid-drain failure. Default-off.
• 2026-06-18 — noetl/server v3.28.0 — worker-driven orchestrator drive now default ON (NOETL_ORCHESTRATE_PLUGIN_DRIVE defaults true) (server#233, closes #108; scale-soak-gated, revert = =false).
• 2026-06-18 — noetl/worker v5.33.0 — pool-affinity decline (drive isolated on the system pool) (worker#114, refs #108 (b)).
• 2026-06-14 — noetl/worker v5.22.0 — transfer endpoint credential-alias resolution, both Snowflake↔Postgres directions (worker#87, closes #99).
• 2026-06-14 — noetl/tools v3.10.0 — Snowflake↔Postgres transfer arms + flatten credential config (tools#65, closes #99).
• 2026-06-14 — noetl/tools v3.9.2 — Snowflake SQL-API context in request body + multi-statement split (tools#64, refs #98).
• 2026-06-14 — noetl/tools v3.9.1 — set User-Agent on the Snowflake HTTP client (tools#63).
• 2026-06-14 — noetl/tools v3.9.0 — Snowflake key-pair JWT authentication (tools#62; kind-validated on live sf_test account).
• 2026-06-12 — noetl/server v3.5.0 — POST /api/execute/batch
  • opt-in exactly-once dedup window (server#189, RFC #90 Phase 7 — scale hardening).
• 2026-06-12 — noetl/worker v5.19.0 — batch dispatch + dedup opt-in + per-subscription rate limits (worker#79, RFC #90 Phase 7 — scale hardening, closes #90).
• 2026-06-12 — noetl/cli v4.11.0 — noetl subscribe, local-mode subscription listener (cli#60, closes cli#59, RFC #90 Phase 6). Standalone kind: Subscription listener + FileEventSink JSONL trail + local_disk store-and-forward spool; cli-only (reuses noetl-tools v3.5.0 source+spool). ai-meta → cli 2fb3fb0.
• 2026-06-11 — noetl/server v3.0.6 — round-trip JSON null in whole-object {{ step }} references (server#177, closes noetl/ai-meta#89). A null field in a {{ step }} envelope rendered as the JS token undefined (invalid JSON), so the consuming step received an unparseable str; render_to_value now retries with | tojson (undefined/none → JSON null) — the server renderer had diverged from the noetl-tools engine that already did this. Kind-validated: cursor pagination collects all 35 events through the terminal next_cursor: null page. ai-meta pointer → server 8e17fbe.
• 2026-06-10 — noetl/tools v3.1.1 — multi-tool sibling references (tools#48, closes noetl/ai-meta#87). TaskSequenceTool now injects each sub-tool's result under its label so a later sub-tool resolves {{ <label>.<field> }} (was rendering empty — a syntax error at or near "," in unquoted numeric SQL positions). Worker adopts via worker#69. Kind-validated (save_edge_cases test_large_payload → record_count = 100). ai-meta pointer → tools 76f942a + worker b97f642.
• 2026-06-10 — noetl/server v3.0.3 — container-callback insert matches the deployed noetl.event schema (server#173, tracks noetl/ai-meta#43). The handler's call.done insert targeted a non-existent attempt column → HTTP 500 on every watcher callback; replaced with an inline INSERT matching the working ingestion path. Unblocked the container-callback chain (kind-val GREEN both probes). ai-meta pointer → 5d2cf58.
• 2026-06-10 — noetl/gui v1.11.0 + v1.11.1 — credential View/Edit recovery for pre-wallet records (gui#36, closes noetl/ai-meta#82) + dev:kind convenience script (gui#35). ai-meta pointer → 8cacc9e.
• 2026-06-10 — noetl/server v3.0.2 — container-tool command type contradiction fix (server#172, closes noetl/ai-meta#81). ToolSpec.command Option<String> → Option<serde_json::Value>: the container tool's array command now decodes server-side + passes through to the worker's Vec<String>; scalars stay JSON strings for shell/db tools. Kind-val GREEN (K8s Job reaches Complete 1/1).
• 2026-06-08 — noetl/tools v2.24.2 — clippy cleanup: 15 warnings resolved across 7 files (tools#44, closes tools#42). Mechanical lint fixes, zero behavioral changes.
• 2026-06-05 — noetl/tools v2.18.4 — duckdb command: alias (parity with postgres) + http params/headers/form non-string coercion (tools#26); worker adopts it (worker#51). Unblocks the pagination + http + duckdb-command fixtures.
• 2026-06-05 — noetl/tools v2.18.3 — postgres command: alias
  • multi-statement SQL on postgres + duckdb (tools#24, closes tools#23); worker adopts it (worker#50). duckdb_test + json_serialization_save GREEN.
• 2026-06-05 — noetl/tools v2.18.2 — postgres tool surfaces the real SQLSTATE + message instead of db error (tools#21); worker bumped to it (worker#49).
• 2026-06-05 — noetl/server v2.22.0 — Secrets Wallet Phase 2: GCP Cloud KMS KeyManager (Cloud KMS :encrypt/:decrypt + Workload Identity); runtime NOETL_KMS_PROVIDER (local/gcp-kms); KEK can leave the process (server#81, tracks #61). Kind-validated on local.
• 2026-06-05 — noetl/server v2.21.0 — Secrets Wallet Phase 1c/1d: credentials + keychain store envelope-encrypted (per-record DEK wrapped by the KEK); self-describing {"v":1,…} blob, forward-only (server#79, tracks #61). Kind-validated end-to-end.
• 2026-06-05 — noetl/server v2.20.0 — Secrets Wallet Phase 1b: envelope-encryption core — KeyManager/LocalDevKms/EnvelopeCipher (server#77).
• 2026-06-05 — noetl/server v2.19.8 — Secrets Wallet Phase 1a: remove the all-zeros default encryption key, fail closed (server#75, tracks #61). Kind-validated.
• 2026-06-05 — noetl/tools v2.18.5 — dollar-quote-aware statement splitter; the 2.18.3 splitter shredded plpgsql $$ … $$ blocks (tools#27).
• 2026-06-05 — noetl/server v2.19.7 — defer task_sequence _prev/_results refs at command-build (server#73); nested-pipeline templates render at runtime → iterator_save_test GREEN.
• 2026-06-05 — noetl/worker v5.11.3 — resolve keychain aliases on task_sequence sub-tasks (worker#48); nested postgres-in-pipeline steps connect.
• 2026-06-05 — noetl/server v2.19.6 — credential store base64-armors the AES-GCM blob for the TEXT data_encrypted column (server#71); keychain creds register + round-trip.
• 2026-06-05 — noetl/worker v5.11.2 — resolves keychain alias under the v10 credential: key (worker#46).
• 2026-06-05 — noetl/server v2.19.5 — v10 control-flow end-to-end (server#69, 6 commits): catalog INT4 + catalog_id alias, ToolSpec skip-null, orchestrator end-step-with-action + task_sequence flatten + intra-pass dedup, template Chainable- undefined, end-step trigger gate.
• 2026-06-05 — noetl/worker v5.11.1 — preserve array tool_config for task_sequence (worker#44).
• 2026-06-05 — noetl/tools v2.18.1 — task_sequence parse_tasks accepts worker-envelope shape.
• 2026-06-04 — noetl/server v2.19.4 — orchestrator template context: step data at top level + call.done capture.
• 2026-06-04 — noetl/tools v2.18.0 — TaskSequenceTool.
• 2026-06-04 — noetl/tools v2.17.1 — PythonTool result- global capture.
• 2026-06-04 — noetl/server v2.19.3 — three fixes shipped together: pipeline flat shape decode (#61), failure termination (#63), workbook resolution (#65).
• 2026-06-04 — noetl/server v2.19.2 — v10 workload + input alias (#59).
• 2026-06-04 — noetl/server v2.19.1 — EE-5 lax decode for integer execution_id (#57).
• 2026-06-04 — noetl/server v2.19.0 — Phase F R4-4b (ExecutionService refactor + cluster-wide list fan-out).
• 2026-06-04 — noetl/server v2.13.0 → v2.19.0 — Phase F R4 series (DbPoolMap N+1 pool layer through R4-5 kind validation).
• 2026-06-04 — noetl/gateway v3.2.0 — Phase F R3b-2 shard-info twin endpoint.

How agents (Claude / Codex / Cursor) operate across this ecosystem — pointers into the rule files in agents/rules/: