Experiment benchmarks catalog cross system findings - kmosoti/FabricO11y GitHub Wiki
Historical report imported 2026-10-10 from
docs/experiments/benchmarks/catalog-cross-system-findings.md. Original SHA-256:dde1f9f2dda038a3efec212839bb04692375506819791a2e0ce338d494ceeb02. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized;$FABRIC_DATAis the mounted experiment drive. See evidence policy.
Cross-system research execution
Execution resumed on 2026-10-08 after the owner restored permission and the required data mount reported writable. The cause of its previous read-only state is unestablished. No mount repair, privileged installation or remote-host workload was performed. Historical unavailable runs retain their outcomes.
Readiness and source retrieval
The registered readiness slice (local checkout reference docs/experiments/benchmarks/catalog-cross-system-readiness-protocol.md; not present at origin/main)
ran python3 -B tools/resource_group.py -- python3 -B tools/bench/labs/catalog/cross_system_readiness.py
and exited 0. Its receipt (local evidence omitted from stage)
records the exact commands, source hashes and preflight charges.
| Observation | Result | Scope |
|---|---|---|
| Existing sealer controls | 6 passed, cargo exit 0 | Prefix order, build failure/panic, reclaim retry, native checkpoint failure and exact replay |
| Turso download | curl exit 0; 17,004,455 compressed bytes | Pinned revision, no build |
| Archive inventory | 4,180 regular files; 77,779,672 decoded bytes | Complete regular-member hashes, archive inspected without extraction |
| Source comparison | 24 complete prior Git blobs matched | Selected source review corroborated by downloaded bytes |
| Evidence preservation | Five source snapshots and member inventory read back exactly | Gzip artifacts beside the receipt |
| Elapsed | 21.19 s execution; 27.66 s charged including preflights | Same historical campaign ledger |
| Cgroup peak | 990,945,280 bytes (945 MiB) | Includes compilation, tests and retrieval; not server RSS |
| Memory events/swap | No high/max/OOM events; zero swap | External containment observation |
| Cleanup | Owned scratch removed | Shared data-drive build cache retained |
CPU accounting in the internal receipt ends before the final inventory; the launcher reports 20.545 CPU-seconds for the entire service. Device-mapper and physical-device I/O counters are both recorded and must not be summed as independent bytes. The two original preflights consumed 6.467 seconds and are charged explicitly rather than resetting the budget.
The download corroborates source provenance. It establishes no Turso speed, memory bound or superiority over Fabric. The source review retains the findings about snapshot location, soft cache capacity, buffer-pool fallback, scoped borrowing and cancellation ownership.
D1: progress within a worker group
The prospective protocol (local checkout reference docs/experiments/benchmarks/catalog-prefix-progress-protocol.md; not present at origin/main) selects a small
mechanism: join existing scoped workers in journal order and reclaim after each
join. The previous implementation collected all join results first. No later
completion can advance reclamation across an earlier unpublished label, so a
completion channel adds no prefix progress to this ordering.
The baseline receipt (local evidence omitted from stage) records cargo exit 101 for the intended assertion: label 1 remained unreclaimed while sibling 2 was blocked. The three-label fixture releases the sibling before asserting, so the rejected scheduler leaves no stranded worker. The original source and failure output remain preserved.
The candidate receipt (local evidence omitted from stage) records cargo exit 0 for all eight sealer controls. The unchanged new regression now reclaims label 1 before releasing label 2, refuses to cross that hole and finishes in order 1,2,3. A reclaim-error control also verifies that all started siblings finish and no later group starts. The implementation removes the intermediate result vector and processes existing handles in order.
That candidate phase then timed out during the history integration target,
after six tests completed successfully. Its exit is failure, not a completed
history result. The full interrupted fixture is preserved by the separately
registered checks allocation (local checkout reference docs/experiments/benchmarks/catalog-prefix-checks-allocation.md; not present at origin/main), which uses
historical runtime and the remaining campaign budget. The
final checks receipt (local evidence omitted from stage)
records all 17 unchanged fast gates passing, including the full workspace test
target (265.775 s). The manual profile then rejected a semicolon in the new
Mermaid annotation; its checker probes and hook tests passed. That failed
documentation outcome remains recorded. The annotation was corrected in both
canonical and mirrored diagrams; the separately registered
documentation closeout (local checkout reference docs/experiments/benchmarks/catalog-prefix-documentation-closeout.md; not present at origin/main) records its
recheck (local evidence omitted from stage).
Individual gate receipts live under the same campaign's recovery directory.
This supports earlier progress in the controlled schedule. No service memory, throughput or ACK-latency gain is measured. The initial candidate run peaked at about 1.4 GiB cgroup memory including build/test work, with no OOM or swap. The full fast run peaked near 11.5 GiB, inside the 20 GiB cap. Both interrupted owned trees are preserved with exact archive readback by the following slices; the historical failed receipts are unchanged.
Remaining research
M1 still needs simultaneous reservation, requested-allocation, RSS and cgroup measurements. The existing mixed-signal memory probe does not measure all four; the current query profiling probe is logs-only. Q1 needs a bounded adapter for whole-query costs and unchanged independent query grading. Neither can be called complete from the readiness controls.
After a D1 correction survives the controls, measure its interaction with the
commit thread: earlier checkpoint work may release journal capacity sooner but
compete with ACK processing. Vary sibling build skew and checkpoint cost before
claiming improved sustained ingest or query latency. Compressed matching and
consumed-work merge scheduling remain separate hypotheses in the
three-lab queue (local checkout reference docs/research/cross-system-lab-queue.json; not present at origin/main).
The full queue, C5 service grid, long-soak behavior and deployment qualification remain incomplete. The renewed permission did not reset historical runtime or evidence consumption; the first slices retain those limits.