Experiment benchmarks coupled labs plan - kmosoti/FabricO11y GitHub Wiki
Historical repository record migrated on 2026-10-10. Original repository path:
docs/experiments/benchmarks/coupled-labs-plan.mdOriginal SHA-256:4ff5ed94cc218a34bd41d47ab4056282df01535b009e3a09ed9f18a731d9719fStated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.
Coupled cost and correctness labs
Current disposition: see completion preparation and executed run. The preparation status below is historical.
Status: prepared, not registered or executed, 2026-10-07. Owner requested lab preparation after the model review. The closed catalog campaign remains unchanged. This plan grants no oracle/contract change, qualification, remote execution, destructive fault, installation, release or default promotion.
Ownership and queue
| Lab | Accountable PI role | Packet | Queue |
|---|---|---|---|
| Recovery and operations | operations PI; coordinator until dispatch | Operations | O6 availability semantics; O7 physical-pruning criterion; O8 bounded collection overlap |
| Capacity and lifecycle | capacity PI; coordinator until dispatch | Capacity | C4 residual row cost; C5 workers × run-size feasibility |
| Query and availability | query PI; coordinator until dispatch | Query | Q4 reusable source snapshot |
The machine-readable queue (local checkout reference docs/experiments/benchmarks/coupled-lab-queue.json; not present at origin/main) separates preparation from
workload admission. No agent or native experiment is started by this document.
Use GPT-6.1-SOL medium for active PIs and task-scoped GPT-6 Luna assistance per
standing orchestration (local checkout reference docs/experiments/lab-orchestration.md; not present at origin/main). Root may own a short PI task;
three lab responsibilities do not require three continuously running agents.
Initial preparation dispatch estimate: 2,000 tokens per lab plus 1,500 shared
reserve. These are estimates, not measured/enforced token limits. Assistants get
one source inventory or fixture audit, at most 800 estimated tokens per task
from the owning lab allowance, a concise artifact and a stopping condition.
If agent quota is unavailable, the coordinator performs the bounded task once.
Admission order
- O6/O7 prepare source-grounded decisions, preserving both existing failures. Identify whether the defect is implementation, fixture, specification or underspecified contract. Do not assume an oracle must change.
- C4 diagnostic preparation and Q4 lifecycle/API design can proceed independently. No candidate mechanism is promoted while full fast remains failed.
- Register exact executable protocols in separate commits: actual commands, source/binary hashes, fixtures, seeds, metrics, guards and controls. No invented CLI in this plan is an executable instruction. Small correctness/diagnostic screens precede implementation and timing comparisons.
- Admit Q4 timing only after its coverage/lifetime controls pass. Admit C5 only after O6/O7 resolve the relevant checks and resource admission is executable. Admit O8 only after reviewing cursor/Spool ownership, cancellation and retry.
- Confirmation is separately budgeted after screens. Combine only supported candidates; preserve nulls. No composite is queued merely to consume budget.
Common evidence and decisions
Freeze baseline as the current working-tree source archive, not just Git HEAD; the workspace contains earlier uncommitted work. Record feature/env switches and compare one mechanism at a time. Existing failed trials stay failed. Seeds 42 for screening and 43 for held-out confirmation are proposed; freeze exact bytes, arrival/query schedules and hashes before execution.
Keep separate plain timing, counted allocation and stage-instrumented builds. Measure observer-on/off overhead before interpreting phase attribution. Report CPU seconds per accepted logical MiB and per complete query; logical bytes, encoded bytes and MB/MiB units must be explicit. Also record first-page/full-chain latency, sample counts, allocation calls/bytes/live peak, per-process RSS/HWM, simultaneous cgroup anon/file/kernel/pressure, source visits, decoded/materialized rows, discovery/refresh work, physical/ logical IO and temporary/retained bytes. Do not infer copied bytes from allocated bytes or sum stacked-device IO.
Native service cells additionally need offered, Spool-accepted, server-ACKed, refused and retried Batches/bytes per second; stable clocks; backlog amount, oldest age and slope; observation-to-Spool, Spool-to-ACK and ACK-to-queryable. Polling visibility is interval-censored. Mean waiting cannot certify p99. Exact raw custody, complete query chains, order, pagination, freshness, gaps and retained window are hard gates graded by independent existing oracles where their declared availability model applies. An unsupported model yields an explicit unresolved result, never an omitted envelope check.
For a performance candidate, first screen one alternating pair per frozen cell; do not call it a win. Nominate at most one configuration per mechanism for three fresh alternating pairs. Proposed confirmation rule: primary cost improves at least 10% in every pair, with CPU, peak memory and declared latency guards no more than 5% worse, exact semantics and no accumulating steady backlog. Quantile sample requirements and absolute service bounds must be fixed in registration; insufficient samples mean inconclusive, not passed. Diagnostic C4 and semantic O6/O7 have no artificial 10% performance target.
Resource and evidence admission
All workloads, including builders and validators, run serially through
python3 -B tools/resource_group.py -- COMMAND. Actual execution commands also
use the coordinator runner. Keep 16 GiB high / 20 GiB max, zero swap, 30-minute
outer deadline, 8 GiB owned scratch and 16 GiB free-drive reserve. Data-drive
scratch/caches only; no fallback to /tmp, system disk or RAM. Persistent
evidence may stay in the repository, explicitly distinguished from scratch.
Last campaign ledger: 9,192.043 of 14,400 recorded seconds used; approximately
5,207.957 remain before preparation checks. This is not a renewed four-hour
budget. Reconcile stage limits and all new receipts before admission. Use
catalog-coupled-* IDs while borrowing that envelope so the existing runner's
prefix accounting cannot be bypassed. Proposed staged ceilings within remaining
time: preparation/verification 300s, O6/O7 diagnostic reproduction 600s, C4 900s,
Q4 controls/screen 1,200s, synthesis/failure reserve 600s. These are maxima, not
instructions to run all cells. C5/O8 and fresh confirmation are deferred until
their projected cost fits remaining allowance or a separately scoped budget.
Capacity evidence already violated its 768 MiB allowance by 10,002,432 allocated bytes; do not launch further capacity jobs against that failed allowance. Before new measurements, prepare a separate prospective allocation/admission amendment or verified lossless compaction. Preserve the historical overage. Proposed new incremental evidence reservation is at most 192 MiB: 32 MiB operations, 48 MiB capacity, 48 MiB query and 64 MiB failure/coordinator overhead. This reservation is NOT adopted by this plan and does not replace existing lab caps. The last aggregate was 1,693,552,640 allocated bytes against 2 GiB; compute current headroom before admission. Native evidence, source snapshots, launcher receipts, raw oracles, archives and manifests all count. Charge unique inodes and allocated blocks, disclose aliases, and include projected failed-run archives before dispatch. If one cell cannot fit, reduce its prospective workload or defer it; never delete contrary evidence or change a frozen workload after measurement.
Stop admission on failed containment, disk/time/evidence budget, invalid clocks, custody/coverage error or unexpected backlog. Byte-verify failure archives before deleting owned scratch; successful jobs remove temporary fixtures/processes. Preserve shared build caches and original failure receipts. Reconcile usage after each cell, not only at closeout. Smaller application limits require separately verified child cgroups; the 20 GiB harness cap is not a deployment profile.
Deliverables
Each PI returns a source/fixture map, hypothesis/null disposition, raw evidence paths, exact commands/exits, resources/cleanup and one recommendation: reject, revise, confirm or defer. Root compares where costs moved, whether peak and total work disagree, and whether findings transfer between dev and small workloads. No algorithm, adaptive controller, new index, persistent format or generalized storage framework is authorized merely by this preparation.
Preparation validation
catalog-coupled-preparation-docs-01 ran the documentation profile via the
resource launcher and coordinator, exit 0 in 16.024 seconds. Documentation
links, checker probes and hook tests passed; rounded cgroup peak 252 MiB, zero
swap, owned scratch removed. This consumed existing frontier time; recorded
remainder is approximately 5,191.933 seconds before future admission. No native
experiment, prototype, oracle change or qualification ran. CI/hooks remain disabled.