Historical repository record migrated on 2026-10-10. Original repository path: docs/experiments/benchmarks/performance-frontier-labs/operations.md Original SHA-256: 483e61ce7316ce98163355b8b12c1f9cc5151a183f76ddc72bb35991e93e6747 Stated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.

Recovery and operations lab packet

PI: operations_landscape (GPT-6.1-SOL medium). State: source review complete; pending checks identified, no new execution. Shared plan and controls. Owned preparation paths: this packet and future operations-frontier-* protocols.

R3: complete the verification prerequisites

Reuse the registered completion storage/publication/fault protocols. The original five storage tests have receipts; later pre-publication pause, missing-table and span table/filter failure additions do not yet have execution evidence. Compile and execute those cases, relevant checks and the syscall observer's controls before interpreting a new performance run. Record each command/exit and source snapshot; preserve the original failed fixture setups.

H1: pending-journal and published representations remain exact across the registered transitions/failures with bounded cleanup. H0: a supported failure breaks custody, completeness, pagination or retry convergence. Reject vacuous success when a requested fault/pause did not occur. Use exact independent query/custody grading; verify optional-index fallback separately from unavailable mandatory tables. This is correctness work enabling performance changes, not a speed benchmark. No expansion into destructive host fault tests.

R4: useful performance inside enforced service budgets

First run existing capped ordinary mixed cells, development then small, under the registered signals protocol: experimental server/node maxima 512/128 MiB and 3072/256 MiB, declared aggregate placement, task caps and CPU quotas. Read back actual controls and verify every descendant. Installed small-node memory high is 128 MiB; the harness's experimental 192 MiB high must be labeled as such. CPU quota experiments are distinct from installed CPU weights.

H1: exact accepted telemetry and useful query/ACK behavior survive these caps. H0: a budget, correctness or drain condition fails. Observe per-service memory, pressure/throttling, offered/accepted bytes, SDK rejections, query errors, ACKs and drain, excluding producers/graders from application memory claims.

After ordinary success, use the separately registered pressure protocol: small Spools, 1 MiB journal, 64 KiB files, server stopped for 45 seconds during continued production, then recovery. H1: backpressure preserves admitted custody and the queues drain. H0: loss, unexplained gaps, unbounded growth or failed recovery. Separate SDK exports refused before admission from successfully committed bytes; retain lost-response ambiguity. Intentional downtime is not steady latency.

For any later signal-cost comparison, hold query demand identical between log-only and mixed variants. Existing mixed cells add span queries and therefore measure combined producer/consumer cost, not isolated trace-ingestion overhead. Visibility polling currently yields coarse upper bounds; never report it as the exact instant of queryability.

Stop on missing enforcement/injection, custody mismatch, unexpected OOM/crash, clock failure, disk floor, deadline or surviving descendants. Archive failures before cleanup. Supported installation and sustained qualification remain separate; no remote workload is assigned. A Luna assistant may perform one receipt/status/cleanup audit (estimated 1500 tokens), then the PI reviews it. Maximum lab workload 90 minutes within the shared four-hour allowance.