Experiment benchmarks lab completion run 01 - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/benchmarks/lab-completion-run-01.md. Original SHA-256: 6bd346163489c2ad745ef3e39da850c6691a9dc1f9d6c16562855e60d932638a. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Development/small completion campaign, run 01

Status: execution in progress. This record separates command completion from scientific acceptance. Registered scope (local checkout reference docs/experiments/benchmarks/lab-completion-protocol.md; not present at origin/main), standing plan (local checkout reference docs/experiments/benchmarks/dev-small-lab-plan.md; not present at origin/main), and raw evidence (local evidence omitted from stage) apply. Native binaries, working-tree snapshots, protocol copies, commands, exit status and resource observations are in each coordinator receipt. Historical results remain tied to their revisions.

Evidence collected so far

  • C1-6 passed: one development node with256MiB published history recovered all 295390 expected logs, including3000 new logs, with no mismatches or duplicates. Sampled server RSS peaked at52.95MiB; whole-job peak includes history seeding and independent grading and must not be substituted for server RSS.
  • Corrected Q2a passed both128-row and65536-row fixtures, plain and counted, with64 complete query chains per fixture/variant. The oracle remained unchanged. Each measured first page is retained exactly; continuations run after all first-page measurements. Counted phases have exactly one correct source loader per query. The full profiling job peaked at4.1GiB, including the graders.
  • Seven C2 cells passed their implemented gates: steady16/32/64/128/256MiB, outage64MiB and adversarial64MiB, three alternating fresh-process pairs each. Steady256MiB worst incremental builder heap was41673485bytes (39.74MiB), satisfying the80MiB ceiling and10% scaling bound from64MiB. Reference heap was1250.45MiB (about96.8% reduction). Three-pair median measured build times were5.335seconds bounded versus5.563seconds reference. At64MiB steady, times were1.412 versus1.381seconds. This supports approximately reference build speed with bounded heap on these fixtures, not near-instant publication for arbitrary journal sizes or an uninstrumented production timing guarantee.
  • Large rows failed pruning equivalence on their first pair; subsequent repetitions stopped as registered. Exact logical rows, applicable final-file bytes, physical filters, heap and cleanup gates passed. The candidate's byte-capped groups read9156 versus20350 reference rows across10second windows, with7140 matching rows in both. This is fewer reads, but still violates the registered equality rule. The failure is retained; no gate was waived.
  • The original five new storage tests passed after two retained fixture-setup failures (missing directories). More transition/fault cases are queued.
  • Pinned SDK1.38.0 preflight and180second trace-only cells passed; the preflight recovered600 spans. Development logs/metrics and mixed cells passed; mixed recovered3000 logs and3600 spans through a server restart. These deliberate restart cells are integration evidence, not steady-capacity qualification.
  • Nested-cgroup preflight verified descendant placement and actual kernel limits. Full capped mixed and pressure cells are queued.
  • Small mixed integration passed: twenty native Spindles recovered300000 logs, 18000 SDK spans and all3903 ACKed Batches exactly through a server restart. Raw trace bytes, projected queries, source identities and clock guards passed. Successful query observations had392.48ms p99; SDK local-Spool response p99 was118.47ms. Two query requests were refused during the deliberate stopped interval and remain recorded. Whole-job cgroup peak was2.48GiB, including producers and independent graders; no OOM or swap was observed. Success scratch was removed. These are finite integration results, not a sustained envelope.

Signal CPU comparisons include the defined consumer workload: mixed cells issue span queries in addition to log queries, while the log-only baseline does not. Total CPU therefore measures combined signal/consumer cost, not an isolated SDK ingestion effect. Query latency is retained and summarized by signal kind.

What the evidence changes

The large-row counterexample exposes a trade-off in the registered acceptance rule: smaller byte-capped Parquet groups preserve bounded memory and can improve time pruning, while differing from the reference layout. That is evidence for a future separately reviewed decision about equality versus non-regression, not permission to retrospectively turn this result into a pass.

Allocation profiling supports query materialization as a distinct memory cost. On counted warm broad tail queries, Scan used about171MiB incremental peak heap versus34MiB for Walk, yet their first-page wall times were both about112ms. Walk's empty tail query avoided most work (about0.38MiB versus137.49MiB Scan). Broad output/decode work remains relevant after reducing retained allocations. These are fixture-specific observations; combined allocator/phase instrumentation affects timing, nested snapshots are inclusive, and the OS cache was warm.

Admission and unresolved gates

C2 acceptance is failed, not qualified. Its large-row equality failure stops dependent sustained acceptance under the registered campaign admission rule. The full5460second soak, three1800second outage trials and12000second fresh development lifecycle are not represented as completed by shorter tests. Their launchers are prepared; independent correctness, integration, resource limits and environment preflights continue. No production default or release qualification changed.

Exact cumulative spill bytes were absent from the initial C2 public-API probe. A separately registered syscall observer will collect logical spill-write bytes without retroactively changing original receipts or using instrumented timing as the uninstrumented result. Installation still requires a compatible disposable Debian/systemd runtime; the local Fedora host and458MiB remote are not that proof.

Resource and cleanup accounting

All workload trees use the20GiB/no-swap cgroup launcher and mounted data drive. No remote workload has run. Success scratch is removed; failed fixture state remains owned pending byte-verified archival and cleanup. The isolated SDK venv is on the data drive and will be removed after dependent cells finish. Shared build caches are retained. Final resource/cleanup tables and required repository checks will be recorded before closing this campaign.