Experiment benchmarks catalog cross system findings - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/experiments/benchmarks/catalog-cross-system-findings.md. Original SHA-256: dde1f9f2dda038a3efec212839bb04692375506819791a2e0ce338d494ceeb02. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Cross-system research execution

Execution resumed on 2026-10-08 after the owner restored permission and the required data mount reported writable. The cause of its previous read-only state is unestablished. No mount repair, privileged installation or remote-host workload was performed. Historical unavailable runs retain their outcomes.

Readiness and source retrieval

The registered readiness slice (local checkout reference docs/experiments/benchmarks/catalog-cross-system-readiness-protocol.md; not present at origin/main) ran python3 -B tools/resource_group.py -- python3 -B tools/bench/labs/catalog/cross_system_readiness.py and exited 0. Its receipt (local evidence omitted from stage) records the exact commands, source hashes and preflight charges.

Observation Result Scope
Existing sealer controls 6 passed, cargo exit 0 Prefix order, build failure/panic, reclaim retry, native checkpoint failure and exact replay
Turso download curl exit 0; 17,004,455 compressed bytes Pinned revision, no build
Archive inventory 4,180 regular files; 77,779,672 decoded bytes Complete regular-member hashes, archive inspected without extraction
Source comparison 24 complete prior Git blobs matched Selected source review corroborated by downloaded bytes
Evidence preservation Five source snapshots and member inventory read back exactly Gzip artifacts beside the receipt
Elapsed 21.19 s execution; 27.66 s charged including preflights Same historical campaign ledger
Cgroup peak 990,945,280 bytes (945 MiB) Includes compilation, tests and retrieval; not server RSS
Memory events/swap No high/max/OOM events; zero swap External containment observation
Cleanup Owned scratch removed Shared data-drive build cache retained

CPU accounting in the internal receipt ends before the final inventory; the launcher reports 20.545 CPU-seconds for the entire service. Device-mapper and physical-device I/O counters are both recorded and must not be summed as independent bytes. The two original preflights consumed 6.467 seconds and are charged explicitly rather than resetting the budget.

The download corroborates source provenance. It establishes no Turso speed, memory bound or superiority over Fabric. The source review retains the findings about snapshot location, soft cache capacity, buffer-pool fallback, scoped borrowing and cancellation ownership.

D1: progress within a worker group

The prospective protocol (local checkout reference docs/experiments/benchmarks/catalog-prefix-progress-protocol.md; not present at origin/main) selects a small mechanism: join existing scoped workers in journal order and reclaim after each join. The previous implementation collected all join results first. No later completion can advance reclamation across an earlier unpublished label, so a completion channel adds no prefix progress to this ordering.

The baseline receipt (local evidence omitted from stage) records cargo exit 101 for the intended assertion: label 1 remained unreclaimed while sibling 2 was blocked. The three-label fixture releases the sibling before asserting, so the rejected scheduler leaves no stranded worker. The original source and failure output remain preserved.

The candidate receipt (local evidence omitted from stage) records cargo exit 0 for all eight sealer controls. The unchanged new regression now reclaims label 1 before releasing label 2, refuses to cross that hole and finishes in order 1,2,3. A reclaim-error control also verifies that all started siblings finish and no later group starts. The implementation removes the intermediate result vector and processes existing handles in order.

That candidate phase then timed out during the history integration target, after six tests completed successfully. Its exit is failure, not a completed history result. The full interrupted fixture is preserved by the separately registered checks allocation (local checkout reference docs/experiments/benchmarks/catalog-prefix-checks-allocation.md; not present at origin/main), which uses historical runtime and the remaining campaign budget. The final checks receipt (local evidence omitted from stage) records all 17 unchanged fast gates passing, including the full workspace test target (265.775 s). The manual profile then rejected a semicolon in the new Mermaid annotation; its checker probes and hook tests passed. That failed documentation outcome remains recorded. The annotation was corrected in both canonical and mirrored diagrams; the separately registered documentation closeout (local checkout reference docs/experiments/benchmarks/catalog-prefix-documentation-closeout.md; not present at origin/main) records its recheck (local evidence omitted from stage). Individual gate receipts live under the same campaign's recovery directory.

This supports earlier progress in the controlled schedule. No service memory, throughput or ACK-latency gain is measured. The initial candidate run peaked at about 1.4 GiB cgroup memory including build/test work, with no OOM or swap. The full fast run peaked near 11.5 GiB, inside the 20 GiB cap. Both interrupted owned trees are preserved with exact archive readback by the following slices; the historical failed receipts are unchanged.

Remaining research

M1 still needs simultaneous reservation, requested-allocation, RSS and cgroup measurements. The existing mixed-signal memory probe does not measure all four; the current query profiling probe is logs-only. Q1 needs a bounded adapter for whole-query costs and unchanged independent query grading. Neither can be called complete from the readiness controls.

After a D1 correction survives the controls, measure its interaction with the commit thread: earlier checkpoint work may release journal capacity sooner but compete with ACK processing. Vary sibling build skew and checkpoint cost before claiming improved sustained ingest or query latency. Compressed matching and consumed-work merge scheduling remain separate hypotheses in the three-lab queue (local checkout reference docs/research/cross-system-lab-queue.json; not present at origin/main).

The full queue, C5 service grid, long-soak behavior and deployment qualification remain incomplete. The renewed permission did not reset historical runtime or evidence consumption; the first slices retain those limits.