Historical report imported 2026-10-10 from docs/experiments/benchmarks/hammer-reference-findings.md. Original SHA-256: a9be4b9885b5ce142f33599f56a32cf9b86aee5805ea4e6366eaa74b6c98a4a5. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Server pressure, native reference and remote edge findings

This record follows the prospective pressure protocol (local checkout reference docs/experiments/benchmarks/hammer-reference-protocol.md; not present at origin/main). The owner clarified 4 GB for the Fabric server, within the existing 20 GiB local laboratory, and a 100 GB storage ceiling. Actual server cgroups use the page-aligned floor below the decimal ceiling; the failed initial enforcement attempt and correction (local checkout reference docs/experiments/benchmarks/hammer-cgroup-alignment.md; not present at origin/main) remain recorded.

Status: reference and remote trials complete; all four repaired service cells passed after access was restored. Seventeen fast and three manual documentation checks passed; owned scratch cleanup and failure preservation are complete. Diagnostic reconciliation found no mismatches. The original identity-scan-02 outer launcher receipt remains unavailable and must not be reconstructed as an original success receipt. No production defaults or independent oracle changed.

Matched native reference trials

Three fresh processes generated 500,000 telemetry rows each, with 512-byte alternating compressible/high-entropy bodies and 1,000 identities. The executed libraries were Arrow 22.0.0, DuckDB 1.4.3 and Vortex 0.87.0, from authenticated pinned wheels. These releases are distinct from the inspected source pins. Each trial wrote both formats, then ran five query shapes in three rotating engine orders. All 135 measured answers matched complete matching counts and ordered Top50 IDs/bodies or full count/body-length totals. Five injected result defects were rejected by the output checker.

Median wall time across the nine observations per engine/query:

Query	Arrow full Parquet scan	DuckDB Parquet SQL	Vortex predicate
Time-range Top50	192.07 ms	28.74 ms	15.29 ms
Node/time Top50	188.54 ms	111.86 ms	62.76 ms
Rare literal Top50	648.21 ms	186.64 ms	65.85 ms
Absent literal	652.95 ms	180.49 ms	58.53 ms
Full count/body length	277.53 ms	108.19 ms	231.88 ms

Parquet occupied about 69.93 MB per dataset; Vortex about 69.06–69.11 MB. Vortex saved only about 1.2% storage on this distribution. Its writes cost 2.24–2.54 seconds wall and 4.23–4.75 seconds process CPU, against Parquet's 1.29–1.38 seconds wall and 1.29–1.34 seconds CPU. Source generation/Arrow conversion is recorded separately, and cumulative process high-water memory is not mislabeled as per-operation memory.

The useful mechanism is selective predicate execution. Arrow deliberately materializes full columns; DuckDB projects and aggregates natively; Vortex pushes predicates before conversion to Arrow. These are competing complete strategies, not a controlled encoding-only ablation. There was no cache flush, and no end-to-end Fabric service ran inside these engine timings.

For the measured dataset, a simple CPU break-even calculation is additional write CPU / query CPU saved relative to DuckDB. Using each seed's own write cost and median query CPU gives roughly 38–47 rare-text queries or 39–41 absent-text queries to recover Vortex's additional write CPU. Time-only queries require about 210–244. Full body-length aggregates never amortize it in these observations because Vortex also used more query CPU. These are derived workload-specific predictions, not a calibrated Fabric controller.

The first reference attempt failed when a Vortex unsigned timestamp column was compared with an implicitly signed Python integer. The separate registered retry (local checkout reference docs/experiments/benchmarks/hammer-reference-retry.md; not present at origin/main) uses explicit unsigned literals; the dataset and expected answers are unchanged. Both attempts retain their real exits. See successful raw trials (local evidence omitted from stage) and original failure (local evidence omitted from stage).

Service and remote observations

The first Scan cell recovered exactly 1,260,000 source logs and 4,704 ACKed Batches across 22 Segment publications. Every semantic, independent final-query, drain and resource gate passed. Peak server RSS was 728.17 MiB; first/final quiet medians were 697.207/697.215 MiB. The server cgroup peaked at 1,873,686,528 bytes, including charged cache; no high/max/OOM event occurred. Server CPU was 233.8 s. Collection-to-ACK p99 was 220.15 ms, while source-to-receive p99 was 1,033.05 ms. Scan query p99 was 499.11 ms for recent logs, 2,218.12 ms for absent text and 338.76 ms for CPU metrics. All 36 visibility probes returned within 4.61 seconds.

The producer-lateness diagnostic failed: p99 581.65 ms, maximum 606.08 ms. All offered data drained, but achieved per-second offer/acceptance rates must be used rather than treating the nominal 16,000 logs/s stage as a capacity qualification. Offline Python oracle memory and time are outside server metrics: the complete job took about 708 seconds and peaked near 7 GiB, within its own 20 GiB laboratory envelope. This was the first pre-repair service observation.

The first Walk comparison failed the live-query gates: four full-range absent-text requests returned HTTP 500. All 1,260,000 source logs recovered exactly, final query chains matched the oracle, custody/drain checks passed, and no OOM occurred. Peak server RSS was 307.52 MiB and quiet medians were 255.188/255.199 MiB; server CPU was 183.68 seconds. These lower costs do not nominate Walk because the live-query failure remains material. The two planned second-seed comparisons were halted under the registered stop rule until the separately registered repair and confirmation described below.

The original observer recorded exception representations without HTTP bodies. A separately registered diagnostic repeat (local checkout reference docs/experiments/benchmarks/hammer-walk-errorbody-protocol.md; not present at origin/main) captures bounded bodies and re-raises every error. Source inspection identifies movement retry exhaustion and journal-path replacement between reads as competing explanations; neither is established by the first run. That registration also raises the memory-lab evidence cap to 8 GiB within the owner's storage allowance, so the roughly 4 GiB raw failed case can be fully authenticated before removal. It adds no time allocation and changes no server memory limit or query gate.

The diagnostic repeated nine HTTP 500 responses (seven absent-text and two visibility probes), all with failed to fill whole buffer. Four deterministic real-journal regressions then ran on the old implementation: three failed by future-Batch substitution, an old offset causing unsigned-subtraction panic, and discovery substitution; the genuine CRC-corruption control passed. This establishes the identity defects independently of inference about the live EOFs.

The narrow repair and confirmation protocol (local checkout reference docs/experiments/benchmarks/hammer-journal-identity-protocol.md; not present at origin/main) keeps transient file handles, verifies first-group identity before offset reads, follows the immutable sealed name after rotation, and checks the expected group before materialization. It leaves durability, query defaults and retry count unchanged. All four unchanged regressions passed after repair. The broader server library command passed 68 tests with two ignored. A fresh release build binds all Rust source and binaries in its own manifest. The first repaired Walk case passed every semantic/resource gate with all 1,260,000 logs recovered, no HTTP errors, 294.69 MiB peak server RSS and 179.73 CPU seconds. Quiet RSS medians were 202.758/202.762 MiB. Producer lateness p99 was still 141.89 ms, above the separate 100 ms diagnostic objective. The second-seed Scan subsequently passed; its paired Walk subsequently passed after access was restored. Extra first-group decoding is a cost to measure; this repair makes no arbitrary-interleaving progress proof.

The first repaired pair has now completed (Walk then Scan, seed 2704101). Both passed every semantic/resource gate and recovered all 1,260,000 logs.

Measurement	Repaired Walk	Repaired Scan
Peak server RSS	294.69 MiB	693.69 MiB
Server CPU	179.73 s	238.26 s
Final quiet median RSS	202.76 MiB	664.43 MiB
Recent-log query p99	260.13 ms	558.67 ms
Absent-text query p99	2,413.72 ms	2,406.64 ms
CPU-metric query p99	277.32 ms	349.03 ms
Per-log collection-to-ACK p99	191.44 ms	212.16 ms
Per-request send duration p99	93.71 ms	80.22 ms
Maximum observed source-to-visible delay	4,560.56 ms	4,812.54 ms
Producer lateness p99	141.89 ms	719.14 ms

Walk used about 57.5% less peak RSS and 24.6% less server CPU in this pair; absent-text tail latency did not improve. Per-request send duration and per-log collection-to-ACK latency move differently because batching and waiting are distinct costs. The query clients are paced by completion, so query counts also differ (118–119 per Walk shape versus 104–105 per Scan shape). This is an end-to-end workload comparison, not an equal-query-count microbenchmark.

The second-seed Scan also passed all semantic/resource gates: 1,260,000 exact logs, 4,755 ACKed Batches, 717.29 MiB peak server RSS, 234.45 CPU seconds, 689.07 MiB first/final quiet medians and no live-query errors. CPU-metric query p99 was 332.28 ms; all 36 visibility probes returned within 4.88 seconds. Producer lateness p99 was 309.93 ms and still failed its separate objective. Its coordinator recorded exit 0 after 707.53 seconds; the original outer receipt was lost during the session transition, as recorded in the cleanup section.

The second-seed Walk completed after restored access: all semantic/resource gates passed, with 1,260,000 exact logs, 4,842 ACKed Batches, 285.61 MiB peak server RSS and 190.67 CPU seconds. Quiet median RSS stayed at 201.61 MiB. Server cgroup memory peaked at 1,419,046,912 bytes, including charged cache. The entire job (including independent decoding/grading) took 696.32 s; its enclosing launcher also exited 0 and cleaned its temporary tree. Final cell (local evidence omitted from stage).

Second-seed measurement	Scan	Walk
Peak server RSS	717.29 MiB	285.61 MiB
Server CPU	234.45 s	190.67 s
Recent-log query p99	490.65 ms	282.76 ms
Absent-text query p99	2,292.92 ms	2,631.58 ms
CPU-metric query p99	332.28 ms	276.28 ms
Per-request send duration p99	89.17 ms	69.71 ms

The lower RSS/CPU direction repeats, while absent-text p99 is worse in the second Walk cell. All four repaired cells still fail the separate producer-lag objective. These ordered finite pairs support a tradeoff, not a universal query-plan default or sustained-capacity qualification.

The causal distinction is between a pathname and the file it currently names. Owning a PathBuf does not make a journal locator stable when active.faj is renamed and recreated. An old offset can then address a valid but different Batch, or extend beyond the replacement file. The repair validates identity on the opened handle before using the offset, and verifies the requested group again before decoding rows. Handles expire at the end of each operation, so the repair introduces no long-lived descriptor cache that could retain reclaimed disk space. Its assumptions are append-only active files, immutable sealed files, and non-reused group IDs within the existing journal contract.

Both inspected droplets have one CPU and about 458 MiB RAM. edge-host-01 serves Caddy and receives no load. edge-host-02 is scoped to edge generation under a 128 MiB, 50%-CPU cgroup, forwarding through loopback SSH over the existing private path to the local server. Its small machine limit is independent of the local server allowance.

The owner's network question prompted the separately registered private-path diagnostic (local checkout reference docs/experiments/benchmarks/hammer-remote-observation-retry.md; not present at origin/main). Five Tailscale pings used the direct path at 42–49 ms; no relay was observed. Thirty echoes inside one SSH session had median RTT 44.60 ms and maximum 59.54 ms. Three 16 MiB transfers in each direction, compression disabled, gave median payload throughput 79.53 Mbps local→droplet and 41.44 Mbps droplet→local. All lengths and SHA-256 values matched. These measurements include SSH/Tailscale encryption, endpoint scheduling and short-transfer behavior, not raw-link capacity. See raw network observations (local evidence omitted from stage).

From the RTT alone, two synchronous workers have an upper-bound estimate of 2 / 0.04460 = 44.8 requests/s, before TLS/HTTP and server work. For the simulator's ten-log Batch this is roughly 448 logs/s, below its 1,000 logs/s offer. This is a prediction to test, not a measured ingestion rate. Bandwidth alone is therefore an insufficient admission model.

Eight real remote collectors recovered all 33,600 source logs and 600 ACKed Batches, with no duplicates, gaps or retries. Four complete query chains matched the independent oracle. Remote request duration was 173.46 ms median, 563.46 ms p99 and 629.62 ms maximum; these are durations on one remote clock, not cross-host source-to-visibility measurements. The remote cgroup peaked at 81.82 MiB, with 25.98 CPU seconds and no high/max/OOM or swap event.

The original edge-real-02 run retains its failed grade: the research drain checker read a cycle's pre-send ACK cursor even though the later final delivery line acknowledged that Batch. The separately registered drain regrade (local checkout reference docs/experiments/benchmarks/hammer-drain-regrade-proposal.md; not present at origin/main) copied immutable inputs, reran the unchanged delivery/query evidence checks and used the final observed ACK per node. It passed, while injected missing-final-ACK, source-backlog and missing-node cases were rejected. This was a research-checker defect; the original run and the minimal counterexample remain preserved.

The first remote attempt also remains failed: its sampler could not read the required resource files. A subsequent preflight found the system slice had no IO controller enabled. The retry explicitly requested IO accounting and checked every required counter before starting project children. The missing original path makes this causal diagnosis an inference, not a fully reproduced proof.

Remote simulator interpretation

The two/eight-worker discriminator is registered separately. Source inspection already identifies an important boundary: the simulator retains pending Batch bytes in memory and catches up scheduled offers only between synchronous passes over identities. Its watchdog sets a flag after 120 seconds, but that flag is checked after the catch-up and whole identity pass. It is not a hard 120-second process deadline. The external remote cgroup still enforces a 600-second limit. Record actual creation/ACK times and backlog, rather than calling late catch-up a sustained 1,000-log/s offer. The simulator does not establish native disk-Spool durability for its unacknowledged offers.

The two-worker run recovered 450 ACKed Batches / 4,500 logs exactly, but finished with 4,800 pending Batches. Its event/state counts imply only 5,250 Batches were actually created out of 6,000 scheduled; the worker's hardcoded offered_logs field incorrectly says 60,000. It ran for 297.25 seconds and recorded 51,339 memory-high events, with no OOM or swap. These expose generator accounting and deadline limitations as well as delivery overload. The independent delivery oracle passed and rejected a removed recovered Batch; the overall drain grade correctly remains failed.

The eight-worker case passed all delivery/drain gates: 6,000 ACKed Batches, 60,000 recovered logs and zero pending Batches. Its process workload finished in about 67 seconds, with a 73.4 MiB remote cgroup peak. The same machine limits therefore admitted substantially more completed work at lower peak memory in this ordered diagnostic pair. This supports the mechanism that greater delivery concurrency can reduce pending-state residency. The two cases' actual offer timing differs, so they do not isolate a universal worker-count speedup or establish sustained deployment capacity.

The retained traces were subsequently reduced with count/time reconciliation. Two-worker generation spanned 290.99 s; 750 of 6,000 scheduled Batches were never created, 450 were ACKed and 4,800 remained pending. Eight-worker generation spanned 59.90 s with all 6,000 created/ACKed and no pending Batches. Attempt RTT p50/p99 was 101.78/2,153.46 ms versus 63.86/134.75 ms; generated-to-ACK p50/p99 was 16.21/272.56 s versus 4.59/7.21 s. These event-stream client timings include queueing and service behavior, not isolated Tailscale link latency. Two-worker reduction (local evidence omitted from stage), eight-worker reduction (local evidence omitted from stage).

Both first reducer attempts failed because an unpacked digest string shadowed the hash function. A minimal counterexample (local checkout reference docs/experiments/benchmarks/remote-reducer-shadow-counterexample.md; not present at origin/main) records the original failure and one-variable fix; unchanged-count-rule retries exited 0. The original failed attempts remain failed. This repairs reporting, not the simulator's offered-load or deadline behavior.

Signal coverage: metrics and traces

This pressure round is log-heavy, not a balanced three-signal capacity trial. Twenty native Spindles also collect real host metrics on a configured 15-second cadence. The live query mix includes system.cpu.time; final complete CPU-metric query chains are checked against the unchanged independent oracle. In the first repaired pair these queries had p99 277.32 ms (Walk) and 349.03 ms (Scan); the second Scan measured 332.28 ms. These are query latencies under concurrent log load, not metric ingestion or metric source-to-visible latency.

The remote simulator includes 32 synthetic gauge points per identity every 15 seconds. The successful 100-identity, 60-second case therefore schedules 12,800 metric points alongside its 60,000 logs. Exact Batch custody covers their encoded bytes; its final metric query checks sim.metric.0 for one identity. This is not an exhaustive independent value query for every synthetic series, nor a metric-heavy cardinality or aggregation benchmark. Native remote collectors also collected host metrics. Tailscale bandwidth is shared by all signals; no separate maximum metrics/s or spans/s was measured.

No trace load was generated in this round. The simulator explicitly sets traces: Vec::new() and the native pressure fixture configures no trace exporter. Earlier native lifecycle evidence includes one span through public commit_traces, exact producer/recovery custody, and complete span queries through publication/restart in both plans. Separate TLS integration coverage exercises the OTLP listener. These tests do not establish SDK-exporter compatibility under load, trace throughput or trace freshness.

Existing metric query rows support Gauge and Sum; this is not general histogram analytics. Traces enter through the Spindle's loopback protobuf OTLP/HTTP endpoint and support stored span queries by time, node, name and trace ID. These capability boundaries must remain explicit when preparing an application pipeline trial.

The next proposed signal experiments belong to the existing labs:

Lab	Discriminator	Required evidence and null
Storage/memory	Gauge/counter cardinality and wide-span construction at matched encoded byte rates	Exact projected values, counter resets, retained raw bytes, sealing CPU and peak RSS; null: log-only bounds do not transfer to these representations
Query	Metric history/rates and trace-ID/parent-child retrieval across active, sealed and restarted storage	Full independent query chains, late/reordered spans, absent IDs, selective and broad reads; null: pruning costs or completeness degrade with signal/cardinality
Collection/reliability	Real SDK trace exports plus native metrics and logs, locally and from a remote edge	Export-to-Spool and server ACK clocks, per-signal freshness, backlog, replay and overload fairness; null: one signal starves another or accepted evidence is lost

These are preparation directions, not registered or executed new trials. Keep the 4 GB server cap, host-appropriate remote caps and data-drive cleanup; admit the concrete fixtures and controls before measuring them.

Model updates and next discriminators

These results change three research directions without nominating a new default:

1. Bound outstanding work as well as workers. For Batch arrival rate lambda, mean request duration L, and w synchronous workers, the simple service ceiling is w/L, additionally constrained by CPU and byte bandwidth. Pending work grows when arrival exceeds actual completion. A useful memory model is M_base + bytes_per_pending * pending + worker_workspace(w). Increasing workers can lower total memory when it drains enough queued state; after that point, worker overhead or server contention can dominate. The remote pair supports investigating this mechanism but does not calibrate it. Next compare fixed actual offer timing, a hard process deadline and explicit generated/accepted/ACKed counters across worker counts. The null is that the apparent benefit disappears when these confounders are controlled.
2. Account for construction before selecting compressed execution. Measure build/conversion CPU, peak live state, bytes and query mix together. The break-even counts above provide a falsifiable predictor for when a selective representation pays for itself. Test that predictor on a held-out Fabric workload with selective and full-read queries. The null is no end-to-end saving after ingestion, conversion, fallback and maintenance are charged.
3. Treat storage locators as identities with a validity check. A stable semantic row/group identity is distinct from a reusable physical path. The current repair validates identity at dereference with short-lived handles; a lifetime-pinned file could avoid repeated validation but retain reclaimed disk space. Any later cache must bound both descriptors and retained bytes, preserve corruption reporting, and test publication/reclamation interleavings. Retry exhaustion and long-run progress remain separate from the repaired substitution and out-of-range defects.

Spool append acceptance fractions remain unknown: the current native cycle stream reports successful appends but lacks a denominator of all attempted appends. Completed append and ACK rates are measurable and must not be labeled an acceptance probability. Longer soak, full bounded-sealer acceptance, installed service tests and target deployment qualification remain open.

Verification and cleanup

Commands, exact exits, resource observations, preserved failures and cleanup are recorded under the coordinator (local evidence omitted from stage). After access was restored, identity-walk-02, both repaired remote reductions and final-preflight-02 exited 0. The original reducer failures and the first preflight's syntax failure retain their original outcomes.

cargo xtask checks --profile fast with dedicated receipts in coordinator/final-fast-01/checks passed all seventeen checks, exit 0. The contained coordinator took 398.91 s; aggregate laboratory peak was 13.4 GiB, with no swap. That peak includes compilation/tests and is not Fabric server RSS. The source/manifest check and resource-admission rejection controls also passed in the separate preflight. All three manual documentation checks passed in final-docs-01, exit 0; the temporary authenticated Bun runtime was removed, and documentation hooks/CI remain disabled.

failure-preservation-01 exited 0 after 168.70 s: eleven owned failed trees contained 9,384,023,819 logical bytes (8.74 GiB). Complete membership and content verification accepted their 4,667,539,437-byte archives (4.35 GiB) before every original tree was removed. The logical reduction is 4.39 GiB; filesystem physical allocation may differ. The helper source/hash and inspectable failure excerpts are retained with the per-tree preservation records (local evidence omitted from stage). The bulky archives now reside under $FABRIC_DATA/evidence/hammer-reference-01/failure-preservation-01. The initial relocation verified every copy but failed the unchanged no-symlink evidence census. A bounded metadata repair replaced its links with regular location records; independent reconciliation then matched all destination hashes. The repair record (local checkout reference docs/experiments/benchmarks/hammer-archive-location-repair.md; not present at origin/main) preserves that failure, the path-normalization retry, and the separately charged maintenance commands. No failure outcome is relabelled by cleanup.

All four repaired service cells have coordinator exit-0 receipts and removed their successful fixture trees. The original second Scan's outer launcher receipt is absent after the session transition. The empty parent scratch path $FABRIC_DATA/scratch/work-l8v8_2fa was inspected and removed after confirming the exact unit inactive. The separate resumption protocol (local checkout reference docs/experiments/benchmarks/pressure-resumption-protocol.md; not present at origin/main) permits present-state cleanup inspection and journal capture, while preserving the unknown original outer exit/cleanup state. Strict closeout remains unrun; a later diagnostic report cannot manufacture the missing original receipt.

The diagnostic reconciliation (local evidence omitted from stage) inspected 55 completed coordinator jobs and two separately charged maintenance launches. It rejected three corrupted-receipt controls, matched all thirteen failure archives (including two later empty maintenance failure trees), and reported zero mismatches. Combined external archives and local memory-lab evidence were 5,056,827,373 bytes, within the existing 8 GiB category cap. Read-only SSH inspection confirmed edge-host-02/dev, no owned remote scratch and no remaining experiment units. No new remote workload ran here. The maintenance charge was 0.413 s; all earlier ledger charges remain retained.

Its report state is deliberately incomplete: three descriptions refer to the same missing original Scan launcher receipt. Exit 0 means the diagnostic report was produced without new mismatches, not that strict closeout passed. The raw journal confirms the invocation and current cleanup; original outer exit/cleanup remains unknown. Long soak, installation, signal-heavy workloads and qualification remain unrun in this round.