Off-server state builder — where the drive-build cost actually is

Measured on prod 2026-10-01. Recorded because a design decision rests on it: see noetl/ai-meta#367 and noetl/ai-meta#366.

Which pool indexes, and why the user pool's zeros are correct

NOETL_STATE_BUILDER=offserver is set on noetl-worker-system-pool only. noetl-worker-rust does not set it, so builder_mode() returns BuilderMode::Off — and WalDrain::from_env returns Ok(None) for Off, so the WAL drain is never started there. No drain means no indexing, which means every state-builder series on the user pool reads zero correctly. That is by design: Off means the server-built state is used.

So all-zero state-builder metrics on noetl-worker-rust are not a misconfiguration. Settled in noetl/ai-meta#368 after it looked like one.

Which pod runs it

NOETL_STATE_BUILDER=offserver is set on noetl-worker-system-pool, not on noetl-worker-rust. The user pool's state builder has never run — over 2d15h of uptime every noetl_worker_state_builder_builds_total outcome read 0 and index_events was 0. Look at the system pool when investigating drive builds.

The distribution (live system pool, 2d19h uptime)

outcome	builds	share	mean	total
cold_rebuild	740	9.8%	0.23 ms	0.17 s
cache_hit	1096	14.6%	2.84 ms	3.12 s
incremental	3380	45.0%	3.88 ms	13.13 s
incomplete	2297	30.6%	49.1 ms	112.83 s

chain_hops_sum / chain_hops_count = 8245 / 740 = 11.1 hops per cold rebuild.

⚠ An earlier version of this page cited "index_events 156 resident" as a live measurement. That was wrong. index_events was a stale gauge: finish_batch refreshed indexed_executions and index_bytes but not index_events, so after a terminal eviction the first two dropped to 0 while the third kept its old value. Prod read indexed_executions 0, index_bytes 0, index_events 1 — zero executions holding one event, which is what exposed it. Fixed in noetl/worker#345; see noetl/ai-meta#368. Do not read residency from index_events on any build older than that.

Series to read (note the noetl_worker_ prefix — not noetl_state_builder_):

noetl_worker_state_builder_builds_total{outcome}
noetl_worker_state_builder_build_duration_seconds_{sum,count}{outcome}
noetl_worker_state_builder_chain_hops_{sum,count}
noetl_worker_state_builder_drive_builds_total{outcome}
noetl_worker_state_builder_index_events

⚠ The incomplete total is HISTORICAL, not steady-state

The 2297 incomplete builds and their 112.83 s accumulated in a burst and are not recurring. Measured 2026-10-01 by driving 18 executions on live prod:

counter	delta over 18 executions
builds_total{incremental}	+53
builds_total{cold_rebuild}	+34
builds_total{incomplete}	+0
drive_wait{woken} / {timeout}	+0 / +0
drive_builds{fallback_incomplete}	+0

87 state builds, zero incomplete, zero retry waits, zero fallbacks. The counters were also frozen across a 3-minute idle window, and incomplete took +0 across ~40 minutes of mixed activity while incremental took +16.

Likeliest cause of the burst: pod start with a cold WAL index — every drive finds the tip unindexed while the drain backfills, and the drain holds the mutex the build is waiting on, which also explains the 49 ms. The pod has 2d19h uptime, so a start-time burst is consistent with the totals.

Consequence: do not read incomplete's share of cumulative time as an ongoing inefficiency. It is a cold-start cost. Read the rate, not the total.

MEASURED: the split, on worker v6.2.1

noetl_worker_state_builder_index_lock_wait_seconds_total{outcome} shipped in worker v6.2.1 (noetl/worker#344). Measured on noetl-worker-system-pool, ~45 min after a fresh roll:

outcome	builds	mean build	mean lock-wait	mean work	wait share
incomplete	4107	381.6 µs	376.6 µs	4.9 µs	98.7%
incremental	72	1592.9 µs	1.3 µs	1591.6 µs	0.1%
cold_rebuild	31	174.2 µs	1.3 µs	172.9 µs	0.7%

incomplete is 98.7% lock wait, and its work is 4.9 µs — the HashMap miss that path performs. The other two outcomes are the control: 1.3 µs of wait, so the mutex is not generally contended. It is contended specifically when the outcome is incomplete, because the drain holds it while applying the very tip the build is waiting for.

⚠ The 49.1 ms figure quoted below was a 2d19h aggregate and does NOT reproduce per-build. The real per-build wait on a fresh pod is ~377 µs — 119x smaller. The aggregate was dominated by a startup burst with a far larger WAL backlog than a normal roll faces. Reading the rate, not the total, applies to this page's own earlier numbers.

How to read these series:

mean lock wait = index_lock_wait_seconds_total{outcome} / builds_total{outcome}
mean work      = build_duration_seconds_sum{outcome}/count - mean lock wait

Confirmed empirically: the burst is cold-start

4107 incomplete builds in ~45 minutes on a freshly-rolled pod, versus 2297 accumulated over 2d19h on the previous one. Dispatch showed it from the outside too: the first two executions after the roll took 79 s and 36 s, then settled to 10 s — better than the 12–16 s pre-roll baseline.

So a worker roll costs a brief dispatch blip while the index backfills, and then runs at or better than before. Expect that; it is not a regression.

Why incomplete reads 49 ms — it is lock wait

build_duration_seconds includes acquiring the index mutex (build_started_at is set before index.lock().await), and the contention is causally tied to the outcome: a build returns incomplete precisely because the WAL drain has not applied the tip, and the drain holds that same mutex while applying it. apply_indexed, finish_batch and rehydrate_execution_from_wal all take it.

noetl_worker_state_builder_index_lock_wait_seconds_total{outcome} (noetl/worker#344) splits the two: build_duration − lock_wait is the work, and lock_wait / builds_total{outcome} is the mean wait. Merged but not yet deployed, and the ~45 ms wait / ~4 ms work prediction is therefore an untested hypothesis, not a result.

The retry loop is already event-driven

drive_wait{woken}=1884 vs {timeout}=280 — 87% woken (noetl/ai-meta#130), and 129 of 5216 drives (2.4%) reach the fallback. There is no fixed polling interval to shorten here.

Two things this overturns

1. cold_rebuild is the cheapest outcome, not the expensive one. 0.23 ms and 11 hops. The name invites the opposite assumption. The chain walk is not a bottleneck at this chain length, and advance_to already short-circuits an unchanged chain to CacheHit by comparing head_event_id — an O(1) change detector that already exists.

2. 87% of state-builder time is incomplete. That outcome means the worker's WAL index has not yet seen the server's expected_head, so the build correctly refuses and the drive retries (stateless_retry 129, fallback_incomplete 129 over the same window). It costs ~49 ms per occurrence — 20x any productive outcome — and it is a waiting problem, not a folding problem. Anything that optimises folding leaves it untouched.

Consequence for chain-certified projections (ai-meta#366)

A chain certificate here would be a redundant and more expensive version of the head_event_id comparison advance_to already does, and the total work it could remove is 16.4 s over 2d19h (cache_hit + cold_rebuild + incremental combined). The worker-side consumer was therefore not built — see ai-meta#367 for the decision and the conditions that would reopen it.

Related

• Chain-certified fold skip — the server-side consumer, merged and flag-off.
• worker/src/state_builder.rs — ExecutionChain::advance_to, the incremental walk and its cache.