NATS is fully removed from the command path (2026-08-01). The worker claims commands from the EHDB bus (NOETL_COMMAND_BUS=ehdb + NOETL_COMMAND_BUS_CLAIM_ADDR) and its pool comes from NOETL_FEED_FILTER_SUBJECT — the legacy NATS_FILTER_SUBJECT name is no longer read. A worker that cannot resolve a pool now refuses to start rather than silently joining shared, which previously stalled every execution with nothing in the logs (noetl/ai-meta#218). NatsSubscriber, NatsCommandSource and the NATS lag poller are gone; CommandNotification, segment_from_filter and claim_outcome moved to src/dispatch.rs because they were NATS-named but on the live EHDB path. This is about the internal bus only. nats is also a user-facing tool kind: a playbook step can point it at the user's own broker to store or publish business data, the same way postgres, duckdb and the object-store kinds reach user-supplied services. That tool lives in the separate noetl-tools crate with its own async-nats, takes its endpoint from the step's url or a keychain credential, and is fully supported — verified on v5.91.2 by tests/user_nats_tool_dispatch.rs. See nats-mcp-tool-kinds. EHDB is internal transient infrastructure and is never exposed as a user store.

The worker's own async-nats dependency still ships for state_builder's WAL-rehydrate path (live under shard_read_verify, reads NATS_URL), the materializers' NATS arms and spool_runtime's nats_object backend. noetl/ai-meta#219 once proposed dropping the dependency wholesale; it is closed as won't-remove — doing so would have taken the user-facing tool and the rehydrate path with it. Recovery path: the DR IaC in playbooks/211-t3-events-migration/nats-dr/. See noetl/ai-meta#212 and #220.

This page is the durable reference for deploying noetl-worker into any environment. It covers the runtime contract the binary expects, the resources it consumes, the network surface it exposes, and — critically — every environment variable it reads, with the why behind each one.

This page is the single source of truth for the deployment shape. Any code change that adds, renames, removes, or shifts the meaning of an env var MUST update the Environment Variables section in the same change set. Same rule for ports, dependencies, and runtime requirements. See agents/rules/wiki-maintenance.md.

The matching deployment manifests live in noetl/ops (Helm chart + kind overlays). This wiki page describes what the manifests need to provide; the manifests are the implementation.

What the binary expects from its environment to start cleanly:

1. EHDB command-bus coordinator reachable at ${NOETL_COMMAND_BUS_CLAIM_ADDR} (the writer's claim face, :9101), with NOETL_COMMAND_BUS=ehdb. Hard requirement. 1b. NOETL_FEED_FILTER_SUBJECT set (e.g. noetl.commands.system.>). It is the input the worker's EHDB pool is derived from. A worker that cannot resolve a pool refuses to start rather than defaulting to shared — a crashloop an operator sees in seconds beats a cluster that looks healthy while nothing progresses (noetl/ai-meta#218).
2. noetl-server reachable at ${NOETL_SERVER_URL} (HTTP). The worker's per-command server_url override (set by the publishing server on every command notification per noetl/ai-meta#53) takes precedence at runtime; the env var is the initial default for one-server deployments.
3. Worker pool slot: WORKER_POOL_NAME identifies which pool this worker belongs to. Server-side runtime registration is keyed by (kind, name); multiple workers in one pool share work.
4. Worker identity: WORKER_ID distinguishes this individual worker. When unset, the worker generates a uuid (fine for ephemeral pods; for StatefulSets prefer setting it explicitly from the pod ordinal so logs + metrics stay readable across restarts).
5. KEDA-managed scaling: the worker is scaled by KEDA on EHDB per-subject lag, scraped from the writer's :9102 /metrics (ehdb_feed_subject_lag{subject="commands.<pool>.shard.<n>"}). The old nats-jetstream trigger is gone. See Scaling.

The scrape port also carries the EHDB Data Query Interface data-plane read handler (noetl/ai-meta#178): GET /ehdb/tiers/{tier} where tier ∈ eventlog|kv|object|vector. The noetl-server relays raw-tier queries here over a direct synchronous HTTP read (not the NATS drive) via NOETL_EHDB_WORKER_QUERY_URL. The handler is read-only, guarded (worker/src/ehdb/guard.rs refuses control-plane roles), disabled-by-default (a strict outcome=disabled no-op unless NOETL_EHDB_ENABLED is truthy — no new env var of its own), bounded (limit/top_k ≤ 1000), and secret-free (it relays the tier *Outcome view structs verbatim). It reads the same local_reference log / durable per-shard segment stores the tier mirrors write through; the raw eventlog scan k-way-merges across the durable shards this replica owns. Because the mirror is pod-local and this Service is headless, point the server's query URL at a single-replica pool for coherent reads. Apart from this read route the worker exposes no API — all other traffic flows out (NATS pull, HTTP to server).

The worker does NOT call Postgres directly. Per agents/rules/data-access-boundary.md, NoETL platform data is accessible via server API only.

Recommended starting point for production. Workers scale on EHDB per-subject backlog (KEDA), so each replica should be sized for one concurrent playbook dispatch.

WORKER_MAX_CONCURRENT (default 1) governs how many commands a single worker pod processes in parallel. Stay at 1 unless the pool's tools are I/O-bound (HTTP fetches dominated by external latency); CPU-bound tools should scale out via more replicas, not more concurrency per pod.

Required: a memory-backed /dev/shm sized to exceed the Arrow IPC cache budget.

The worker allocates one Arrow IPC shared-memory cache per process at init (ArrowIpcSharedMemoryCache::new(), budget NOETL_IPC_CACHE_BUDGET_BYTES, default 256 MB). The cache stages call.done results that exceed the broker's 100 KB inline budget into POSIX shared-memory regions backed by /dev/shm (via the shared_memory crate's shm_open+ftruncate+mmap).

The Kubernetes container-runtime default for /dev/shm is a 64 MiB tmpfs. When the cache writes past 64 MiB the store page-faults against the full tmpfs and the kernel delivers SIGBUS — the worker dies with exit code 135 and crash-loops. Mount a memory-backed emptyDir at /dev/shm sized above the cache budget:

spec:
  containers:
    - name: noetl-worker
      env:
        - name: NOETL_IPC_CACHE_BUDGET_BYTES
          value: "268435456"   # 256 MiB — pin next to sizeLimit below
      volumeMounts:
        - name: dshm
          mountPath: /dev/shm
      resources:
        limits:
          memory: "768Mi"      # >= sizeLimit + worker RSS
  volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 320Mi        # >= NOETL_IPC_CACHE_BUDGET_BYTES + headroom

Three values move together and must stay coherent:

If you raise the cache budget, raise the sizeLimit and the memory limit in lockstep. The committed manifests wiring this up live in noetl/ops ci/manifests/noetl/worker-*.yaml (fixed in ops#193).

/healthz returns 200 whenever the process is responding (it does not check upstream NATS / control plane — those have their own failure modes the heartbeat covers), so a slow warmup never trips a liveness restart.

/readyz (noetl/ai-meta#130) returns 503 warming during boot and 200 ready once boot warmup has completed — on the drive pool that means the system/orchestrate wasm module is Cranelift-compiled + cached (the noetl_worker_ready gauge flips 0→1). Gating readiness on warmup hides the one-time ~1.5s compile (the cold-start, ~99% Module::new) from the first real request: Kubernetes only routes / completes a rollout once worker_ready=1. The warmup also completes before the pull loop starts claiming, so the first orchestration is warm even where the probe is not wired (e.g. the metrics port is not fronted by a Service).

The table above describes the HTTP probes the binary serves on :9090. The prod noetl-cmdbus-writer StatefulSet does not use them — it probes cmdbus-claim (:9101) over TCP, for liveness as well as readiness:

livenessProbe:  { tcpSocket: { port: cmdbus-claim }, timeoutSeconds: 1,
                  periodSeconds: 20, failureThreshold: 3 }
readinessProbe: { tcpSocket: { port: cmdbus-claim }, timeoutSeconds: 1,
                  periodSeconds: 5,  failureThreshold: 3 }

A listening socket passes a TCP probe whatever the process is doing behind it, so the writer reports Ready with Restart Count: 0 through a stall of any length. On 2026-09-22 that hid an 18-hour degradation in which the tier service's read_execution ran to a ~36s tail and took the whole product with it (noetl/ai-meta#351). Same class as noetl/ai-meta#322, and worse: there only readiness was TCP.

Full detail: cmdbus-writer-tier-service.

• Stream: noetl_commands (default; override via NATS_STREAM). Subjects: noetl.commands.>. Historical — the NATS layout below is retained for the DR path only.
• Consumer: pull-based, durable. Default name worker-pool (override via NATS_CONSUMER). Filter subject: noetl.commands (override via NATS_FILTER_SUBJECT).
• Subjects the worker reads: per-pool filter shapes vary; the server publishes commands to noetl.commands.{system|shared}.<execution_id> per the Phase F sharding-design. A "shared" pool worker filters on noetl.commands.shared.*; a "system" pool worker filters on noetl.commands.system.*.

The worker is the scale-out unit per agents/rules/execution-model.md. KEDA reads NATS consumer lag and scales the Deployment between minReplicas and maxReplicas.

Standard KEDA ScaledObject for the worker pool:

triggers:
  - type: nats-jetstream
    metadata:
      account: $G
      natsServerMonitoringEndpoint: "nats.noetl.svc:8222"
      stream: noetl_commands
      consumer: worker-pool
      lagThreshold: "5"

Workers can be added, removed, or restarted freely — state lives in the server's event log + cache; the worker is stateless.

The worker mints its own snowflake IDs via src/snowflake.rs for client-side event IDs (per agents/rules/observability.md Principle 3). The 10-bit node id comes from either:

1. NOETL_SNOWFLAKE_NODE_ID (preferred — set per pod by the deployment manifest).
2. NOETL_SHARD_ID (back-compat alias for the same idea).
3. Derived from NOETL_NODE_ID / NODE_NAME / pod hostname via FNV-1a hash to 10 bits.

For multi-replica deployments, set NOETL_SNOWFLAKE_NODE_ID explicitly to avoid hash collisions producing duplicate IDs.

The epoch comes from NOETL_SNOWFLAKE_EPOCH_MS if set, otherwise defaults to a build-time constant. Match the server's epoch (2024-01-01T00:00:00Z UTC) to keep IDs mutually orderable across producers.

All env vars the binary reads at startup or runtime, with the why behind each one.

Secrets Wallet Phase 4b (noetl/ai-meta#61, noetl/worker#56) — the worker half of the mTLS transport the server opts into in Phase 4a (noetl/server#103, NOETL_TLS_*). The worker's control-plane HTTP client (credential fetch, command claims, event posts) speaks plain HTTP by default; the env below makes it present a client certificate so the worker→server credential channel is authenticated + encrypted (the resolved secret no longer travels plaintext on the wire).

Built on the rustls-tls reqwest backend the worker already uses (reqwest::Identity::from_pem + Certificate::from_pem_bundle). Set NOETL_SERVER_URL to the server's https:// URL when these are on. The cert/key/CA are mounted from a K8s Secret; the manifests live in noetl/ops.

Init-container caveat (Phase 4c): the wait-for-api init container curls /api/health over plain HTTP — against an mTLS server it can't complete and blocks the pod in Init. It needs the client cert (mTLS curl) or a non-mTLS health path (parallels the server's mTLS httpGet-probe caveat).

Secrets Wallet Phase 5c (noetl/worker#58, tracks noetl/ai-meta#61) — defense in depth on top of the Phase-4 mTLS transport. mTLS encrypts the wire; sealing encrypts the credential payload to a key only this worker holds. The cleartext exists only briefly inside the worker process after unseal; never on the wire, never in the server's HTTP response body.

Wire shape. The worker generates a long-lived X25519 keypair once at startup, includes the base64 public half in its register payload's runtime JSON blob:

{
  "name": "<WORKER_ID>",
  "component_type": "worker_pool",
  "runtime": { "kind": "rust", "worker_public_key": "<base64-32-byte-x25519-pub>" },
  "status": "ready",
  ...
}

The private half stays in-process for the worker's lifetime — only the public half ever leaves. The server-side deployment-specification § Sealed credential delivery describes the endpoint shape; the wire envelope is {alg, v, eph_pub (b64), ciphertext (b64)} carried by the same noetl_sealed algorithm constant on both sides (x25519-hkdf-sha256-chacha20-poly1305, v=1).

Zeroize. After the auth-alias resolver has injected the resolved fields into the tool config, the worker zeroizes the string values in the intermediate credential.data map so an OOM core dump or a debugger snoop on the worker pod can't recover the secret from the post-dispatch heap.

Sealing requires a worker name on the server side. The server looks up worker_public_key by the name (kind=worker_pool) of the runtime row. Workers identify themselves at register time with WORKER_ID, so that env var is required for sealing to work; absent → the wrapper logs a WARN and falls back to the plaintext path.

This is live in production. Since 2026-07-27 the shastaratech prod cluster runs NOETL_COMMAND_BUS=ehdb — commands reach workers over the EHDB feed, not NATS. NATS is still installed as the rollback path; deleting it (T5) is a separate, irreversible approval.

The worker plays two roles here, and they are configured separately.

As a consumer (every worker pool) it claims commands from the shard coordinator over the network instead of subscribing to NATS. As a writer host (one designated deployment per command shard) it owns the durable command log and serves the ingest / claim / metrics faces that the server and all other workers talk to.

Pool isolation is not a separate flag: each consumer subscribes with a subject filter (commands.<pool>.>) derived from its NATS_FILTER_SUBJECT segment, and the coordinator only ever hands a member a matching command. A user-pool worker therefore cannot claim a system command — proven on prod (0 system claims during the canary).

Rollback is a single env flip back to nats on the server and every worker pool, valid for as long as NATS remains installed.

Full cutover procedure, the three-round prod execution log, and the T5 checklist: ehdb wiki — Runbook: L1 Command-Bus Cutover.

The mechanism is a bridge for credentials that already exist as env vars in the worker's environment (because they come from GKE Workload Identity, an existing K8s Secret mount, or similar already-in-place trust per agents/rules/execution-model.md). Net-new business-logic credentials belong in the NoETL keychain, not in this allowlist.

System worker pool only. The off-server state builder reconstructs orchestrator WorkflowState from the noetl_events WAL (not the materialized noetl.event table), walking the one-level prev_event_id chain and caching the built spine keyed by the immutable chain head (noetl/ai-meta#115 Phase 4). The shadow loop is observation-only — it proves on the running cluster that the builder reads the WAL with zero noetl.event scans and that the chain-walk + pool-side cache (hit / incremental tail-advance / cold-rebuild) behave, without touching the drive. The drive cutover that makes the drive consume this builder's state is staged behind the server's NOETL_STATE_BUILDER=offserver.

Live in production on both system-pool shards and undocumented here until 2026-08-05. Found by diffing env::var("NOETL_*") call sites in worker@main against this page.

The chain index the builder caches is unbounded by default. That is what took the system pool to ~748 Mi steady and then OOM-killed it at a 768 Mi limit, wedging the off-server drive and stopping Muno turns and login (noetl/ai-meta#163). These three are the cure, and prod runs all of them.

The interaction is the part worth remembering: TTL and ceiling are safe only in combination with rehydrate-on-miss. Setting either bound alone trades an OOM for silent state loss.

NOETL_STATE_INDEX_MAX_EXECUTIONS and NOETL_STATE_INDEX_REHYDRATE_CONCURRENCY exist in the same code path and are not set in prod.

Documented 2026-08-05. These were invisible to three earlier sweeps because their names live in constants (pub const EHDB_KV_MODE_ENV: &str = "NOETL_EHDB_KV") and are read as env::var(CONST) — a pattern matching a literal inside the call sees nothing.

EHDB's storage tiers are selected per tier, not by one switch. Every one defaults to off.

Updated 2026-08-10 — the "none in prod" statement above was true until this date and is no longer. NOETL_EHDB_{ENABLED,MODE,CLIENT_ROLE,LOCAL_REFERENCE_LOG} plus EVENTLOG/PROJECTION/KV/OBJECT = shadow are now set on the three prod data-plane worker Deployments (noetl-worker-rust, noetl-worker-system-pool, noetl-worker-system-pool-shard1). NOETL_EHDB_VECTOR remains off — it has no live call site.

Shadow only. No tier serves anything: it is a dual-write plus parity compare and never touches the authoritative path. Prod parity is holding (mirrored / materialized, last_degraded 0, zero parity_mismatch).

Shadow runs on the worker Deployments, not the writer, because the mirror seams fire per-worker on that worker's own activity; the writer claims only commands.cmdbus.> and would mirror nothing. It needs no PVC — the live path passes authoritative_sequence: None, so parity rests on the engine's own count and monotonic-order invariant, and a fresh log after a restart is correct. Durable storage is a primary concern, and primary is where the writer's PVC comes in.

⚠ Do not set any tier to primary — but the reason changed on 2026-08-11, and the old reason is no longer true.

Superseded (was true until ai-meta#247): runtime_hook_env armed the mirror only in shadow, so selecting primary silently disarmed verification — measured in kind on one binary under identical load, shadow mirrored 30 events and primary mirrored 0.

Now: runtime_hook_env arms the mirror for both shadow and primary (arming is monotonic — it can only add mirroring), and the event-log tier has a real authoritative serve path gated by primary_serve::decide. Serving requires three conditions, not one: the tier is primary, a durable tier service is reachable, and parity held on this operation. Any missing condition demotes to the incumbent and records primary_unavailable — never a silent wrong answer, never a failed caller.

⚠ Which append path carries the decision depends on how the tier is configured — and the serve-ready configuration is not the default one. decide has three call sites, and no single one of them covers every configuration:

call site	reached when
eventlog::mirror_live_event (this worker's own hook)	NOETL_EHDB_EVENTLOG_MIRROR_SOURCE=worker — the default. server disarms the hook before the tier mode is consulted, so this site never runs there
metrics_server::ehdb_tier_append_handler, Resolution::Local branch	NOETL_EHDB_TIER_QUERY_SOURCE=local — the default
eventlog::serve_service_append, the Resolution::Service branch of the same handler	NOETL_EHDB_TIER_QUERY_SOURCE=service

Superseded (was true until worker 8d46b33, worker#273, 2026-08-12): earlier text here described the serve path as riding the append path without naming which one, which read as "any primary flip serves". On MIRROR_SOURCE=server + TIER_QUERY_SOURCE=service — the only configuration on which the tier is both complete (#258) and not per-pod fragments (#257 PR 4) — the flip reached neither of the first two sites. It produced served_primary = 0, in fact the whole noetl_ehdb_eventlog_ops_total family absent, and no log line, while events flowed and landed correctly. The third site closes that; a property test walks all eight input combinations and asserts its verdict equals decide's, so the sites cannot drift apart about what primary means.

🟢 FLIPPED 2026-08-13 — the event-log tier is primary and SERVING in prod on worker v5.118.1 / server v3.81.0. The paragraph that stood here said no tier was primary anywhere, the tier-service address was set nowhere and NOETL_EHDB_TIER_QUERY_SOURCE was unset everywhere; all three are now false. Live prod values are in the Live prod configuration table below. Serving evidence: served_primary=232 with every demote outcome 0, serve_state: served_primary in the endpoint reply, and the IS SERVING log line. The pre-flip gate ran 20 executions on a fresh store: 0 divergence, 20/20 full-set match, tier-service append errors 0, store parses 580/580, and all five replicas returned the same 29 records. The full precondition list is playbooks/257-serve-readiness/GO-LIVE.md. See Umbrella: EHDB Primary Serve Path (ai-meta#257).

Live prod configuration (2026-08-13, shastaratech-noetl-prod ns noetl):

where	variable	value
writer (noetl-cmdbus-writer)	NOETL_EHDB_TIER_SERVICE_BIND	0.0.0.0:9110
writer	NOETL_EHDB_TIER_SERVICE_DIR	/data/eventbus/ehdb-tier — on the eventbus RWO PVC, not pod-local
worker pools (×3)	NOETL_EHDB_EVENTLOG	primary
worker pools	NOETL_EHDB_TIER_QUERY_SOURCE	service
worker pools	NOETL_EHDB_TIER_SERVICE_ADDR	noetl-cmdbus-writer-0.noetl.svc.cluster.local:9110
worker pools	NOETL_EHDB_EVENTLOG_MIRROR_SOURCE	server (worker hook disarmed)
server	NOETL_EHDB_EVENTLOG_MIRROR_SOURCE	server
server	NOETL_EHDB_WORKER_QUERY_URL	http://noetl-worker-rust-metrics.noetl.svc.cluster.local:9090
server	NOETL_EHDB_CROSSSTORE_PARITY_ENABLED	true

⚠ served_primary is per-pod. The server's mirror relay targets the noetl-worker-rust-metrics Service, so appends land on whichever app=noetl-worker-rust pod it reaches, and the system pools receive none. Sum across replicas when alerting — a single pod reading 0 is not a demotion.

⚠ NOETL_EHDB_TIER_QUERY_SOURCE=service without NOETL_EHDB_TIER_SERVICE_ADDR falls back to local with only a WARN. Setting the query source alone gives downgraded_local — a read that looks correct and answers from a pod-local fragment. Always set the pair.

⚠ Only the event-log tier can be flipped today. kv, object, projection and vector have a serve_primary_cycle, but its only caller is bin/ehdb-selfcheck, so setting them primary cannot change what any caller receives — it is a no-op with a warning. The RFC's suggested cutover order (KV first, event-log last) describes a desired blast-radius order, not a runnable one: the only wired tier is the one it names last. Wiring the other four is unstarted work, tracked on ai-meta#257.

The flip-time signal says which case a tier is in — one line and one operation="runtime_hook" counter increment per tier per process (ai-meta#259 corrected these; the old text told operators a serving tier was inert, at the exact moment they were running a cutover):

⚠ Not yet released. The three-outcome table below is worker#263, which is open and unmerged. A binary built from main today records only the single primary_not_wired outcome. Check noetl_worker_build_info{version} against the release that carries #263 before reading these outcomes as current.

⚠ And it fires from runtime_hook_env, which NOETL_EHDB_EVENTLOG_MIRROR_SOURCE=server disarms. On the serve-ready configuration this signal is unreachable — so it cannot mislead there, but it also says nothing there, and its silence is not evidence of anything. It is the signal for the MIRROR_SOURCE=worker configuration. On the serve-ready one, read serve_state and the transition log line instead (below). Landing #263 would not have made a primary flip serve: the same disarm routes around both the warning and the first serve call site.

outcome	level	meaning	operator response
primary_not_wired	WARN	tier has no runtime serve path (kv / object / projection / vector)	nothing was promoted; flipping again will not help
primary_no_tier_service	WARN	serve path exists, NOETL_EHDB_TIER_SERVICE_ADDR unset	set the address; until then every op demotes (no_durable_service)
primary_armed	INFO	serve path armed and a tier service is configured	watch served_primary against parity_diverged / no_durable_service

primary_armed is not a claim that the tier is serving — reachability and per-op parity are only knowable per operation. It says the flip was accepted.

⚠ Reachability is measured, never assumed, and never inferred from a clock. It is derived from real tier-service operations — the append is the probe — and a demotion stays in force until a real operation succeeds. A lapsed cooldown permits a retry; it does not restore belief. (Before worker#262 it did, so five seconds after a tier service died the tier decided it was authoritative again with nothing having contacted it.) A service that answers but rejects a record is reachable — that is a data problem, and demoting there would let one poisoned payload disable authoritative serving platform-wide.

⚠ Prod workers run ephemeral-storage: 1Gi (request = limit) and local_reference is an unbounded pod-local JSONL log — the 8 MiB segment rotation belongs to durable_segment, not this backend. Watch df -h /tmp.

The bus has been live in prod since T5; the storage tiers are shadow-only (ehdb#241).

The projection tier's shadow means more since ai-meta#265, and prod already sets it.

⚠ Two corrections worth stating plainly, because both were recorded wrongly before:

1. Prod has set NOETL_EHDB_PROJECTION=shadow on all three worker deployments for some time. The 2026-08-14 kind-gate exit note claiming "no NOETL_EHDB_PROJECTION* variable exists on any prod workload" was wrong. A gate's exit assertion is itself a representation.
2. What that value reaches changed. Before #265 it selected a mode whose only caller was bin/ehdb-selfcheck. Since A1/A2 the same value additionally reaches the tier-generic store (projection.jsonl, its own file and its own lock) and the write-path serve decision through primary_serve::decide. shadow still resolves to "the incumbent is authoritative" and the tier receives nothing at all unless the server carries NOETL_EHDB_PROJECTION_MIRROR_SOURCE=server.

The worker's half of the projection tier is deliberately small:

• No pod-local write path, by design. A local resolution is refused 503 with a reason. The event log has one only because it predates the tier service; giving the projection tier one would build N disjoint per-pod fragments that a primary flip would promote while the incumbent holds everything.
• Therefore NOETL_EHDB_TIER_QUERY_SOURCE=service and a reachable NOETL_EHDB_TIER_SERVICE_ADDR are hard prerequisites for the projection tier, not tuning.
• The worker cannot read noetl.projection_snapshot at all (data-access-boundary.md), so it has no worker mirror mode — the mirror is server-authored or absent.
• worker/src/ehdb/projection.rs still carries shadow_project / fold_window_authoritative, which compare EHDB to a second copy of EHDB's own logic. That is a useful engine-conformance drive and it is not serve-readiness evidence; the evidence is the server-side cross-store comparator against the incumbent's rows.

Server-side variables (NOETL_EHDB_PROJECTION_MIRROR_SOURCE, ..._PARITY_ENABLED, ..._READ_SOURCE, ..._MIRROR_ASYNC, ..._PARITY_LAG_TOLERANCE_SECS) are documented on the noetl/server deployment-specification page. Cutover order and gating: playbooks/265-projection-tier/CUTOVER.md.

Writer-fronted tier service (ai-meta#257). Unset everywhere today; the client is a strict no-op without an address, which is what keeps primary demoting and the flag-off path byte-identical.

⚠ local at more than one replica is not one tier — it is N disjoint fragments, and this is the single most consequential thing on this page. The server's mirror hop resolves the pool's ClusterIP Service, so each event lands on whichever pod that hop reached. Measured at three replicas: one execution's 13 events read back as 0 / 13 / 0 records, and all three replicas labelled their answer local. The cross-store comparator, pinned at each pod in turn, returned divergent / match / divergent for the same execution. Two thirds of reads report the tier as empty while one third reports a clean full-set match — so a single-replica sample has a 1-in-3 chance of reading the full set and declaring success. Prod runs multiple replicas.

Setting service on every replica makes the tier one store by construction: the same three replicas then read 13 / 13 / 13 with match from each, and the pod-local stores stay empty because nothing is written to them.

Failure behaviour is deliberately asymmetric, so a misconfiguration cannot degrade quietly into the fragmented state:

Point NOETL_EHDB_TIER_SERVICE_ADDR at the writer's stable per-pod DNS (noetl-cmdbus-writer-0.noetl.svc.cluster.local:9110), never the headless Service — that round-robins, and the premise is one process.

Attribution rides noetl_worker_ehdb_query_ops_total{operation="tier_query_source.read"} with outcomes service / downgraded_local / misconfigured.

⚠ The label alone is not evidence. A defect can keep tier_query_source: service truthful-looking while serving the pod-local store — one mutation did exactly that, passing the label check and failing only the data checks. Compare counts, not labels.

Tier-route reply fields (tier_query_source from worker#272; serve_state from worker#273). GET|POST /ehdb/tiers/{tier} carries two fields that exist so an operator can read the serve decision from something that cannot be absent the way a metric can (Registry::gather prunes empty families, so a labelled series does not exist until it fires):

⚠ unknown is a real answer and is not not_primary. It means this pod has decided nothing — it received none of the execution's appends, because the server's mirror hop resolved to a different replica. Expect served_primary on the replicas that received appends and unknown on the rest; in the three-replica rehearsal that was 1–2 of 3 per execution. Requiring all replicas to report served_primary would be asserting a property of kube-proxy.

Do not read serve_state as parity. It is the tier's authority to answer; cross-store parity against noetl.event is the server comparator's job.

The matching transition log line is emitted once per transition, in both directions — not per append, because an append happens 13 times per execution and a line on each is a line nobody reads, which is operationally the same as the no-line state this was added to fix:

NOETL_EHDB_EVENTLOG=primary IS SERVING: the event-log tier answered
authoritatively through the writer-fronted tier service
    serve_state="served_primary" outcome="served_primary"

NOETL_EHDB_EVENTLOG=primary is NOT serving — the incumbent answers
(demoted: no_durable_service)
    serve_state="no_durable_service" outcome="unavailable"

The serve series is pinned at 0 at the bind site of the route carrying server-authored appends, gated on EHDB enabled + tier not off, on both shadow and primary — so the flip changes values, never which series exist, and a disabled build's /metrics stays byte-identical. Seven outcomes: mirrored / parity_mismatch / primary_divergence / primary_unavailable / rejected / served_primary / unavailable. served_primary 0 means "did not serve"; the family absent means "this binary predates the serve path" — and noetl_worker_build_info{version} answers which.

Authoritative event identity (ai-meta#258).

The verdict rides the tier's existing family as noetl_ehdb_eventlog_ops_total{operation="authoritative_id"}, with outcomes agreed / stamped / unstamped / disagreed / not_json / not_supplied. disagreed is a real cross-store identity divergence caught at the append site. Count, ordering and payload parity against the authoritative log are not checked here — they are compared server-side (noetl/server#343), because per data-access-boundary.md the worker may not read noetl.*.

Per-tier limits and tuning, same default-off caveat:

Flight SQL — NOETL_EHDB_FLIGHT_SQL, NOETL_EHDB_FLIGHT_SQL_BIND, NOETL_EHDB_FLIGHT_SQL_TOKEN_ALIAS, NOETL_EHDB_FLIGHT_SQL_TOKEN_FIELD, NOETL_EHDB_FLIGHT_SQL_ALLOW_LOOPBACK_UNAUTH — is documented in its own section below.

⚠ NOETL_EHDB_FLIGHT_SQL_ALLOW_LOOPBACK_UNAUTH disables authentication on the loopback interface. Development only.

Set only on the pod that hosts the writer. NOETL_EVENT_BUS_HOST is the master switch; the rest configure the faces it binds. None is set on the prod worker pools — prod runs the writer as its own StatefulSet (noetl-cmdbus-writer), so these are the in-process alternative.

Two independent drain loops with an identical knob set and identical defaults. Neither is set in prod, so both run on the defaults below.

Companions to the TTL / byte-ceiling bounds above — these govern the refill after an eviction, which is what makes bounding the index safe.

Found 2026-08-05 by a second pass. The first pass matched only env::var("NOETL_*") and missed everything read through the crate's env_bool / env_u32 / env_u64 / env_addr helpers — 55 variables in this binary. All eight below are set in prod.

Each consumer picks its transport independently. Post-T5 they are all ehdb, and there is nothing else to point them at.

⚠ They are four independent selectors, not one switch. A cutover that misses one leaves that single consumer pointed elsewhere while the other three move, and nothing cross-checks them.

Its only reader is sharding::AffinityConfig::from_env, which has no caller. AffinityConfig::decide() is exercised only by sharding.rs's own unit tests, and record_affinity_decision — the counter for those decisions — has no caller either, so the metric is absent from a live pod.

The steering block that used all three was deleted by 14a3f1f chore(nats): remove the NATS command path during T5 and was not reinstated on the EHDB path. The likely reading is that it is superseded rather than lost — Phase 5 put system workers on per-shard durable consumers and L1 T4 made pool isolation and shard routing subject dimensions, so the writer routes and worker-side steering is redundant. That is inference, not proof; the code cannot tell the two apart.

Until it is settled, read this variable as decorative: setting it true does not enable anything. Tracked on noetl/ai-meta#242 and #166.

Also live on both system-pool shards and previously undocumented. Both are read as trimmed strings and an empty value is filtered to None, so a blank setting is indistinguishable from unset.

State-builder metrics (:9090/metrics): noetl_worker_state_builder_wal_events_total (events consumed from the WAL — the WAL-read proof, RFC tenet 5) · noetl_worker_state_builder_event_scans_total (noetl.event scans the builder issued — the no-scan proof, RFC tenet 3; stays 0) · noetl_worker_state_builder_builds_total{outcome=cache_hit|incremental|cold_rebuild|incomplete} (cache effectiveness + correctness) · noetl_worker_state_builder_chain_hops (chain-walk depth per cold rebuild) · noetl_worker_state_builder_build_duration_seconds{outcome=…} — build latency by the same four outcomes, new in v5.96.0. Until it shipped the counter said which path a build took and nothing said what it cost, which is why #156 could quantify the per-hop floor in kind and not on prod.

The write-side sibling of the #104 result materializer: a third read-model projector off the noetl_events WAL (durable consumer noetl_state_materializer, self-ensured at startup) that projects each execution's slim event-chain (event_id, prev_event_id, event_type, node_name, status, the #104 result_ref URN, a bounded extracted block, and a payload column — the verbatim shard_chain_payload (slim projection + prev_event_id) the Phase-3 cold-load reader reconstructs a byte-equivalent chain from) into a per-execution Arrow Feather shard on object store at …/shard=sNNNN/.../execution=<eid>/state/<open|sealed>.feather (co-located with the same execution's result bytes via the shared FNV-1a shard_key). Append-while-live, seal-on-terminal. Written by the shadow writer; read by the Phase-3 cold-load path (below) when NOETL_STATE_SHARD_READ is on — otherwise the drive serves from the Phase-1 bounded in-memory index and the writer is output-neutral.

READ-ONLY w.r.t. noetl.* (object PUT only — preserves the #103 sole-writer); never fails an event (errors counted + WARN-logged with execution_id, batch acked, idempotent key → safe redelivery); bounded by terminal eviction + idle-TTL sweep + a max-open ceiling so it never reproduces the unbounded Phase-1 index it complements. Default off → not spawned (true no-op). Instant rollback = unset NOETL_STATE_SHARD_WRITE.

Cell placement reuses the result tier's NOETL_RESULT_CELL_ENV / _REGION / NOETL_RESULT_CELL / NOETL_RESULT_SHARD_COUNT so a state shard lands in the same cell + s{fnv:04} folder as that execution's result bytes.

State-materializer metrics (:9090/metrics): noetl_worker_state_materializer_drained_total · …_rows_total (slim rows projected) · …_shards_written_total{seal=open|sealed} · …_shard_bytes_total · …_errors_total (encode/write failures — never fails an event) · …_evicted_total{reason=idle|max_open} · …_open_shards (resident working set, O(live executions)) · …_cycle_duration_seconds. Writer lag rides ehdb_events_group_lag{group="noetl_state_materializer"} — it used to ride a worker-side JetStream gauge, which T5 left with nothing writing it (noetl/ai-meta#242).

The read side that makes the Phase-1 bounded index cheap: on an off-server drive cache miss (the execution's chain was evicted by TTL/byte pressure, or this pod never indexed it), the drive cold-loads the execution's state by reading the Phase-2 Feather state shard from object store — one keyed GET /api/internal/objects/{key} (~tens of ms) — instead of replaying the retained noetl_events WAL (a scan of up to the whole 24h window, bounded to the rehydrate deadline). The shard's payload column carries the verbatim shard_chain_payload per event, so the reconstructed WalEventIndex chain is byte-equivalent to the WAL-replay / in-memory path by construction (same payload → same apply → same spine → same drive commands). The drive makes routing decisions (next.arcs, when, loop counters, fan-in) from this spine, so byte-equivalence is the correctness bar.

Correctness never depends on the shard alone (belt-and-suspenders):

• No shard (never written / GC'd) → object_get 404 → fall back to the WAL replay (unchanged).
• Stale open shard (writer cadence lags the WAL tip) → the reconstructed chain can't reach expected_head → the advance_to staleness guard returns Incomplete → fall back to the WAL replay, which supplies the tail.
• NOETL_STATE_SHARD_READ_VERIFY dual-build guard → after a shard build serves, also run the WAL replay and byte-compare the two spines; on any divergence increment state_equivalence_mismatch_total and serve the WAL build. Never serves divergent state.

Read-only w.r.t. object store + noetl.*. Byte-equivalence assumes the slim index (NOETL_STATE_INDEX_SLIM=on, the prod default): with slim off the shard stores the slim projection while the WAL stores the full envelope — still output-equivalent to the drive, but not byte-identical. Default off → the miss path is byte-identical to today's WAL replay. Instant rollback = unset NOETL_STATE_SHARD_READ.

The reader resolves the shard object key deterministically — tenant/project default to default/default (the worker mints result URNs with no tenant/project), the folder shard + cell come from the same NOETL_RESULT_CELL_* env the writer read, and the date partition derives from execution_id — so no carried metadata is needed.

State-reader metrics (:9090/metrics): noetl_worker_state_shard_reads_total{outcome=hit|miss|fallback} (the payoff distribution — hit = shard served, one object read replacing a WAL scan) · noetl_worker_state_shard_read_duration_seconds (cold-load latency vs the WAL-replay miss cost) · noetl_worker_state_equivalence_mismatch_total (the VERIFY divergence tripwire — MUST stay 0). The served-outcome also shows on noetl_worker_state_builder_drive_builds_total{outcome=served_shard} (vs served_rehydrated for the WAL path).

The off-server orchestrate drive runs the system/orchestrate wasm module via wasmtime. On the first dispatch after a fresh pod start the module is fetched from the server catalog and Cranelift-JIT-compiled (Module::new) before it is cached by digest — a one-time ~1.5s cost (≈99% compile, ≈1% fetch of the ~1.6MB module) that the subsequent warm hops do not pay. Boot warmup compiles the module once at startup, behind the /readyz gate, so the first real drive is a cache hit.

Warmup metrics (:9090/metrics): noetl_worker_plugin_load_seconds{phase="fetch"|"compile"} (the cold-start attribution — compile is the dominant phase) · noetl_worker_plugin_warm_total{outcome="warmed"|"skipped"|"error"} (boot warmup outcome) · noetl_worker_ready (0 while warming, 1 once warm — the /readyz source of truth).

NOETL_COMMAND_BUS selects which transport carries commands: nats (default, unchanged), shadow (the writer exists and receives the server's publishes, but NATS stays authoritative and the worker keeps consuming NATS), or ehdb (the worker claims from the EHDB bus and creates no NATS subscriber — the shape T5 needs).

One worker per shard is the writer host (NOETL_COMMAND_BUS_HOST). It opens the shard's durable command log and serves three faces: ingest (the server publishes here), claim (worker replicas compete here), and a Prometheus lag endpoint (the KEDA signal). Every other worker is a claimer and needs only NOETL_COMMAND_BUS_CLAIM_ADDR.

Restart behaviour (#208). A writer restart is a routine event — node drain, image bump, OOM, HPA move — and both halves of surviving it are now in the binary, shipped in v5.81.3 and prod-validated 2026-07-30: the writer pod was deleted mid-stream under continuous load and 38 executions → 38 COMPLETED, 114 = 38 × 3 commands, 0 duplicates / 0 loss, redial in ~2.7 s, resuming from the committed cursor rather than replaying the shard.

⚠️ This mattered more than it looked. Before v5.81.3 the pre-fix bug had left prod's user-playbook dispatch silently dead for ~2.4 days (writer restarted 2026-07-28 07:49Z; every user-pool claimer wedged on a half-open socket with no error anywhere) — masked because the system pool happened to restart later and kept its own commands flowing. If you are debugging "executions stuck at command.issued", check the claimers' last log line against the writer's start time before anything else.

• Claimers detect a writer that stopped answering (TCP keepalive plus a coordinator heartbeat on the claim connection) and redial on their own, so restarting the writer alone no longer requires restarting the workers. A redial logs EHDB claim_next failed; reconnecting to the claim coordinator with the claim address, member and filter.
• The host resumes from its committed cursor instead of replaying, and logs the decision once at startup: EHDB command-bus claim coordinator resumed shard=0 from_cursor=N origin=persisted|fallback_tail|fallback_beginning. origin=fallback_* on a pod that should have had a cursor means the volume did not survive — check that the PVC is really mounted.
• On SIGTERM the host persists the cursor and seals the log. The seal matters: the storage engine reopens from its durable manifest, so records still in an unsealed part are invisible to the restarted process even though every append was fsynced. Give the pod enough terminationGracePeriodSeconds for the seal to finish — prod runs 90 s (raised from the 30 s default for exactly this). ⚠️ The seal covers graceful stops. An ungraceful stop (SIGKILL, node loss) can still lose the unsealed tail of the active part, and the seal itself races in-flight ingest — noetl/ai-meta#209, open. On a command bus that is a loss class, so it should land before T5 removes the NATS fallback.
• The persisted cursor is clamped to the reopened log's tip. This is not a micro-optimisation: the engine resumes from its durable manifest, so a log that lost an unsealed part can reopen behind the persisted cursor — and because the writer then re-issues keys from its recovered sequence, a cursor no future key can exceed would filter out every record and take the bus permanently dark. Clamping costs at most a small redelivery.

Shutdown sequence + its knobs (#209, #226). A process may host more than one writer — the command bus and the events feed — and the phases run across every host at once rather than finishing one host before starting the next: stop all ingest → quiesce once → persist all cursors → seal all engines. Host-at-a-time left the second host accepting and appending for the whole of the first host's sequence and then spent its own deadline from a standing start, out of the same shared budget.

Verify a graceful stop from the terminating pod's own log: it should end with EHDB writer hosts all sealed — shutdown complete and Worker stopped. An ERROR line saying the hosts did not all seal — or the absence of both lines — means an unsealed tail, and the next incarnation will resume clamped=true over a log that came back below its own persisted cursor. The same fact is on /metrics as noetl_worker_shutdown_hosts_sealed vs _total; see Observability.

Metrics (:9102, host only): ehdb_feed_shard_lag{shard} + ehdb_feed_total_lag (the KEDA trigger — undelivered + delivered-but-unacked) · ehdb_feed_shard_committed{shard} (the acked-through cursor, i.e. what a restart would resume from). · ehdb_feed_shard_inflight{shard} + ehdb_feed_total_inflight (delivered-but-unacked depth, from ehdb v0.7.0). Across a writer restart the committed cursor should pick up roughly where it left off; a jump back toward 0 with a lag spike to the size of the log is the replay regression #208 fixed.

⚠️ Do not read the resume log line as proof on its own. On the prod validation the resume took the durable path (origin=persisted) yet logged from_cursor=0, and ehdb_feed_shard_committed reset 408 → 165 across the restart — the counter is segment-relative, not a global monotonic offset. No replay happened (proven by exact command accounting), but the line cannot distinguish "resumed" from "replayed from 0". Until noetl/ehdb#304 lands, confirm a resume by counting delivered commands, not by reading the cursor.

⚠️⚠️ inflight must be sampled, never defaulted. render_snapshot publishes this family unconditionally, and per its own HELP text high lag with zero in-flight is the stalled-consumer reading. So a writer that fills the field with a literal 0 does not report "unknown" — it reports an outage, on every scrape, for a bus that is working. The pair is what separates two opposite conditions that lag alone renders identically:

⚠️ And a drained bus cannot tell you the sampler works: lag 0 / inflight 0 is both the honest drained reading and what a hardcoded zero looks like. The discriminating state is claimed-but-unacked work, which is why tests/cmdbus_writer_metrics.rs claims three records without acking and asserts the scrape reads inflight=3 while lag stays at 6 — lag counts undelivered plus unacked, so a poll-without-ack moves it by zero. Before ehdb v0.7.0 the coordinator had no accessor for the value, so 0 was the only thing a writer could compile (noetl/ehdb#402).

⚠️ The exposed lag is whole-shard, not per-pool. ehdb_feed_total_lag counts every subject past one committed cursor, so on a single-shard deployment carrying both commands.shared.shard.0 and commands.system.shard.0 it over-reports either pool's own queue. Safe for scaling up; the hazard is that a stuck system-pool command pins the global cursor and therefore pins lag high indefinitely. Per-subject lag is noetl/ehdb#303, open. See the ops wiki KEDA page for how this feeds the user pool's autoscaler.

The writer Deployment is not yet a checked-in noetl/ops manifest — it lives with the cutover automation in noetl/ai-meta (cmdbus-writer-shard0.yaml, its PVC, and the staged shadow → canary → flip → rollback scripts). The claimer side is the NOETL_COMMAND_BUS* env on the ordinary worker deployments in noetl/ops.

Two read-only EHDB surfaces the data-plane worker can expose, both disabled by default and gated behind NOETL_EHDB_ENABLED:

• In-cluster tier query (#178) — the GET /ehdb/tiers/{tier} route on the existing metrics/query port :9090 the noetl-server relays to (see Ports). Adds no new worker env var of its own — it rides NOETL_EHDB_ENABLED + the tier mirror config.
• External Flight SQL driver (#184) — a dedicated Arrow Flight SQL listener on port 8092 serving the projection tier read-only to apps outside the platform. Inert unless opted in; fail-closed (a non-loopback bind with no token alias is refused at startup, so it never serves open).

The driver is read-only and serves the projection tier through the same ProjectionDriver read contract as #178 (built on it, not a fork); it opens no write path and relays no secrets. The kind deployment + a pyarrow validation client live in noetl/ops (ci/manifests/noetl/ehdb-flightsql-deployment.yaml, ci/validation/ehdb_flightsql_client.py).

Not armed anywhere. NOETL_PROJECTOR_ENABLED is unset on every prod workload, so no loop runs and the orchestrator self-writes the snapshot exactly as today. Blast radius is zero until two flags are flipped — this one, and the server-side NOETL_PROJECTOR_OWNS_SNAPSHOT, which has its own default of false.

Added here because drift-audit.sh env-docs reported them absent (Rule 2a).

⚠ All six are read through env_u32/env_u64/env::var helpers rather than a single config struct, so a read-set scan that greps only env::var("NOETL_…") undercounts this module.

Both change what a run does, not what it reports, which is why both default off.

Same shape as the server:

Per agents/rules/execution-model.md "Secrets and credentials rule": business-logic credentials (third-party API tokens, tenant database DSNs) belong in the NoETL keychain via the server's /api/keychain/* routes. The allowlist mechanism above is a bridge only for credentials that already live in the pod env via existing trust (GKE Workload Identity, etc.).

• noetl_worker_build_info{version} — always 1; the version label identifies the running binary.

  Read this first when a metric you expect is missing. Registry::gather prunes metric families with no children, so every labelled metric is absent from /metrics until something increments it — registering it is not enough. That makes absence ambiguous between "this has never happened" and "this binary predates the metric", and the two want opposite responses.

  Where the label set is closed the series are pinned at 0 at startup so a scrape reads 0 rather than nothing. Where it is open — as on noetl_worker_event_emit_failed_total{event_type}, which takes a free-form string — pinning cannot work at all, and this gauge is the only way to tell the two cases apart. Added in noetl/ai-meta#238; before it, nothing on /metrics carried the version, so the question was answered from the Deployment's image tag — a different representation, which can disagree with what is actually running.

• Silent-failure signals — counters added 2026-08-05 closing the warn/error-without-metric triage (#238). Each covers a path that previously logged and counted nothing; all labelled sets are pinned at 0 at startup.

  metric	labels	what a non-zero value means
  noetl_worker_materializer_skipped_total	—	Data loss. A drained message had no event_id, so it was never projected — and was acked anyway. Under NOETL_EVENT_INGEST_PUBLISH_ONLY the materializer is the sole writer of noetl.event, so the event is permanently absent from the durable log. The materializer's six other counters do not move on this path.
  noetl_worker_materializer_ack_failed_total	stage	non_event_batch is the serious one: without that ack the batch poison-loops forever, so a sustained rate is a stalled materializer. after_project costs a redelivery, never a row. per_handle is a partial ack.
  noetl_worker_materializer_drain_failed_total	—	Drain polls failing outright. A materializer that cannot drain is a durable log that stops being written.
  noetl_worker_ehdb_claim_reconnect_total	feed, reason	Claim-coordinator reconnects. feed=commands stalls dispatch; feed=events stalls the materializer. One production pod logged 85 of these in 24 h with no metric at all before this.
  noetl_worker_claim_failed_total	—	Commands whose claim failed and were nacked. ⚠ This path retries with no backoff (#244), so a persistently failing command spins at full speed — measured at ~1000 log lines/second. The rate is the signal; the log flood is the symptom.
  noetl_worker_state_builder_replay_end_total	reason	Why a cold-rebuild replay stopped. Three of the four (complete, deadline, max_messages) are ordinary termination; feed_error means the rebuilt state is INCOMPLETE. Read the ratio, not the count.
  noetl_worker_event_emit_failed_total	event_type	An event was abandoned after its retry window. On the container-poll resume path a lost call.done means the execution never continues (#227).
  noetl_ehdb_dataplane_ops_total{operation="reachability"}	outcome	Whether the EHDB tier service was reached on a real operation. unreachable is a transport failure and demotes the tier; reached / rejected promote it (a service that answers, even to refuse a record, has been reached). This is the input to the serve decision, so it is the series that explains a primary_unavailable. ⚠ Deliberately not pinned at 0 — it is only reachable from a configured tier client, which keeps the flag-off path's /metrics byte-identical; absence here correctly means "no tier op has ever run".
  noetl_worker_state_builder_drive_builds_total	outcome	Seven outcomes, not the three an older doc comment claimed. fallback_incomplete and served_shard_mismatch are the interesting ones.

  Unlike the server, this crate builds its metrics eagerly in WorkerMetrics::new, so an unlabelled metric is always present at 0 and never absent-because-untouched. Verified against a released image: v5.105.0 declares 77 metrics and serves 45, with zero unlabelled-absent.

• Metrics: Prometheus surface at :9090/metrics per agents/rules/observability.md. Surface includes NATS pull rate, command claim outcomes, tool dispatch durations per tool kind, result-store PUT latency.

• EHDB tier queries (noetl/ai-meta#178): noetl_worker_ehdb_query_ops_total{tier,operation,outcome} (counter) + noetl_worker_ehdb_query_duration_seconds (gauge, last op) — recorded by the GET /ehdb/tiers/{tier} read handler. Like the other EHDB families, a disabled outcome records nothing, so a worker with EHDB off (the default) emits a byte-identical /metrics. Each query also rides an ehdb.query tracing span carrying execution_id.

• EHDB tier service — the server half (noetl/ai-meta#260, introduced by worker#270): the writer-fronted face on :9110 that primary promotes to authoritative. Until #260 it recorded nothing — its whole observability was four tracing lines, three on failure paths — so the only prod-side signal was a demotion counter recorded by the client, which cannot tell an unhealthy service from an unhealthy network.

  metric	labels	what it says
  noetl_ehdb_dataplane_ops_total{operation="tier_service.<op>"}	outcome	One per handled request. <op> ∈ health / append / read_execution / scan / unsupported / conn. Deliberately the same family as tier_client.<op>, so a prefix match puts both halves of one request side by side.
  noetl_ehdb_tier_service_duration_seconds	operation	Request latency histogram. Incremented by the same function as the counter, so the two cannot drift apart. Measures decode → store → encode, excluding the frame read and write, which are dominated by how fast the caller is.
  noetl_ehdb_tier_service_store_appends_total	—	Records this process has written. A counter, so a restart reads as a reset rather than a plateau.
  noetl_ehdb_tier_service_store_sequence	—	Highest global sequence the store holds — "does the tier I am about to promote actually contain anything", answerable without reading it. Sampled at startup, so a restart in front of a populated store does not read as empty.
  noetl_ehdb_tier_service_store_bytes	—	Store size on disk, sampled at startup and after each append.

  Outcomes, and the one distinction that matters for alerting: ok / hit / miss / invalid / unavailable / error / unsupported, plus accepted / closed / protocol_error / write_error / accept_error on conn. degraded is not !ok — a malformed request is ok=false, degraded=false (the caller got a correct refusal; the service is fine), while unavailable and error are degraded because a tier promoted here could not answer. Alert on degraded, never on !ok, or a client sending bad frames pages someone.

  On reads, hit vs miss is the load-bearing pair: an all-miss read stream means the store is serving, and empty.

  ⚠ These ARE pinned at 0 — but only once the listener binds. All 20 series are created at 0 from a single call site immediately after the bind in event_bus.rs, and below that bind the pin is unconditional: not gated on a store being configured, on tier mode, or on any NOETL_EHDB_*. (A pin inside a config branch is not a pin — server#315 shipped that mistake and left its skip reasons absent on exactly the configuration worth debugging.) With NOETL_EHDB_TIER_SERVICE_BIND unset nothing is pinned and /metrics is byte-identical to a build without the module, which is correct: absence then means this process has no tier service, and noetl_worker_build_info{version} above answers "is this binary too old to have the metric".

  Hosted by the process that sets NOETL_EVENT_BUS_HOST — the writer, not an ordinary worker pod — because that is the process holding the durable RWO volumes. The writer's Service already publishes :9090, so this needs no PodMonitoring change; noetl/ops#255 only has to be applied.

• Materializer / consumer lag — ehdb_events_group_lag{group}, exposed by the EHDB writer, not by the worker.

  This replaced a pair of worker gauges (noetl_worker_nats_consumer_pending + _ack_pending, labelled {stream, consumer}) written by a periodic JetStream lag poller. T5 deleted JetStream and the poller with it, so from that point the gauges were never written again — and because a labelled metric is absent from /metrics until something increments it, they simply stopped appearing rather than reading zero. The recording rule and the three backlog alerts built on them went permanently inert with no error anywhere. Both gauges are removed outright in noetl/ai-meta#242.

  The signal itself still matters and is unchanged in meaning: under the server's NOETL_EVENT_INGEST_PUBLISH_ONLY gate it is the earliest indication that published events are piling up un-materialized, and it climbs even when the materializer loop has stalled or died — the loop cannot report its own lag, so the measurement has to come from outside it. That is now the writer rather than a per-pod poller, which also removes the need to dedup across system-pool pods. It is what the CQRS flip guardrail alerts read (see noetl/ai-meta#103 and the ops runbook noetl-cqrs-publish-only-flip.md).

• NATS resilience / self-heal (noetl/ai-meta#163): two independent in-process reconnect paths keep a NATS bounce from taking the worker down. (1) The state-builder drain (system pool) recovers a lost pull consumer — noetl_worker_state_builder_consumer_recreate_total{reason}

  • the noetl_worker_state_builder_healthy gauge the /livez probe reads. (2) The main command loop (every pool) survives a hard NATS disconnect (nats-0 pod delete, full client disconnect) in-process instead of exit(1) + k8s crash-restart + full WAL replay: on a pull/ack/nack disconnect it rebuilds the NatsSubscriber with bounded exponential backoff (250 ms→10 s) and re-binds the same durable consumer (server-side cursor preserved; claim_command atomicity keeps exactly-once), recorded on noetl_worker_command_loop_reconnect_total{reason} — reason=pull|ack|nack (a disconnect surfaced on that op and the subscriber was rebuilt) / connect_error (a rebuild attempt failed, retrying). A rising pull/ack/nack rate with the pod's restart_count staying flat is the fix working. Like the sibling counter it is a labelled family, so a worker that never hit a disconnect emits no series for it (byte-identical /metrics).

• Graceful-shutdown seal (noetl/ai-meta#226): noetl_worker_shutdown_hosts_sealed + noetl_worker_shutdown_hosts_total, written once, immediately before exit. sealed < total on a terminating pod means an unsealed tail, and the next incarnation's resume over that log will clamp. A pod that terminates without ever writing them never reached the seal at all.

  This exists because a partial seal used to be observable only on the next boot, as a clamped=true resume over a log that had come back below its own persisted cursor — i.e. after the records were already gone. In prod the terminating pod logged sealing EHDB writer hosts before exit hosts=2, sealed the command bus, and stopped: no events seal, no Worker stopped, and no expiry line from the 15 s shutdown budget either, because the starving seal had also stopped tokio's timer. seal_all now logs at ERROR when any host is left unsealed, in the terminating pod's own log, while an operator is still watching.

• Events-consumer reattach (noetl/ai-meta#225): noetl_worker_events_consumer_redials_total{face} — face is group_claim (the :9104 named-group claim path) or wal (the :9108 fan-out the off-server state builder subscribes to). A rising count around a writer restart is the consumers reattaching; a flat count while a group cursor is not advancing is the wedge.

  Before #225 this counter could not have moved even while the consumers were wedged: neither events face carried the TCP keepalive

  • negotiated heartbeat the command claim path got in #208, so a half-open socket left the parked read neither yielding nor erroring — and the redial paths downstream were unreachable for want of an Err. noetl.event took no writes for 3h24m with /readyz ready.

• Materializer counters: noetl_worker_materializer_drained_total / _projected_total / _duplicates_total / _acked_total / _project_errors_total + _cycle_duration_seconds — the ack-after-materialize loop's throughput + the no-loss redelivery surface (project_errors = a batch left un-acked to redeliver).

• Scrape path (the metrics above only guard the flip if something scrapes them): the :9090/metrics port is exposed by a headless *-metrics Service per pool. On the kind dev cluster a VictoriaMetrics VMServiceScrape selects them; on prod (GKE) the cluster runs Google Managed Prometheus, which does NOT honor prometheus.io/scrape annotations — the worker pods are scraped by a GMP PodMonitoring (ci/manifests/noetl/gmp/podmonitoring-noetl.yaml, ops wiki Production monitoring (GMP)). Without that object the lag gauge + materializer counters never reach Managed Prometheus and the CQRS flip guardrail is blind.

• Tracing: tracing spans on every NATS message, every tool dispatch. execution_id is a span field.

• Logs: structured JSON via tracing-subscriber.

When changing any of the above, validate per agents/rules/deployment-validation.md:

1. cargo build --release --bins
2. cargo test --quiet
3. Build image locally + load into kind: kind load docker-image …
4. Apply the manifests against kubectl --context kind-noetl
5. Smoke-test:
   • Submit a playbook execution via the noetl CLI.
   • Confirm the worker claims the command, runs the tool, and POSTs lifecycle events back.
   • For env-var changes: verify kubectl exec … env | grep <VAR> shows the value AND the worker's startup log line confirms it was honored.

Only after kind passes does the change roll forward to Cloud Build + GKE.

• noetl-executor adoption — the shared execution core the worker pulls from crates.io.
• release-pipeline — Cargo + Docker image publish flow.
• worker-credentials — credential resolution in tool dispatch.
• nats-mcp-tool-kinds — tool kinds the worker dispatches.
• noetl/server sharding-design — the cross-component routing context.
• noetl/server deployment-specification — the server-side counterpart of this page.
• noetl/ops Helm chart + manifests — the deployment-time implementation.
• agents/rules/wiki-maintenance.md — the rule that requires this page to update in lockstep with code.