Historical report imported 2026-10-10 from docs/research/design-consolidation.md. Original SHA-256: 673c183675cb15bfdc622248cbc693cf335721da3b6a2112c080eb7893b28296. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Design consolidation: what the experiments decide

Status: exploratory research, tier 6 of the source-of-truth order. This page collapses the prototypes, measurements and hypotheses of the ledger into one practical design for Fabric, with each decision tied to the evidence behind it and the gate it still needs. It changes nothing by itself: every item below is a candidate for an ADR, a milestone or a contract change under the rules of AGENTS.md, and the gates column says which. Where the evidence is against something, that is written down too.

The design, in one paragraph

Keep the product's spine: one custody record per Batch, per-kind projections sorted by one total key under one Segment lifecycle, answers that say what they covered. Change four things, in this order. First, make every query cost its answer rather than its window, by promoting the tail key index and the threshold walk together with a file-order fallback, under the executable specification as the correctness reference. Second, keep the unsealed tail short and cheap: implement the bounded sealer as accepted, after checking its ceiling on real text, and treat the tail's entry count, not its bytes, as the quantity to bound. Third, store the payload once: a canonical record the node emits, so that the custody copy and the queryable copy are one object; adopt it on the wire only after the block tail is shown to remove the per-entry decode. Fourth, measure storage on real text from now on, because the synthetic workload hides every layout question.

Decisions

Each row: the decision, what it rests on, what it would cost, and the gate it needs before it is product.

#	Decision	Evidence	Cost	Gate
D1	Promote the tail key index and the threshold walk together (ledger L-03, L-04), with a frame cache so that key order costs no more than the sequential scan when nothing stops.	Selective shapes 220 ms to 4 to 13 ms (tail-index run 01); wide-window limit 50 320 ms to 20 ms, memory 242 to 59 MiB (threshold run 01); 0 mismatches in 1,500 + 1,200 + 800 random queries with pages; the walk's overhead when nothing stops it was frame re-decoding and a 64-frame cache removes it, 1,246 to 556 ms on the real tail (optimality run 01, A4).	A read lock on the index so queries do not serialise; the cache; the index's extension off the commit path; about 95 bytes per tail entry.	Q2 of the direction review; the registered soak for ACK latency while the index extends; the walk shown to refine spec::threshold_walk (T7).
D2	Hold the budget boundary as designed (L-05) and do not promote it yet.	The semantics hold on four states (budget run 01); the price is pages (12 become 113); it cannot cut between sources that start at the same key; its rate half is untested for want of a counter workload.	A contract change (a new reason in unavailable, a budget in the request) in its own commit.	A consumer that wants bounded latency per request more than fewer pages; C2 (a boundary inside a source) and C4 (counters) run first.
D3	Implement the bounded sealer as accepted (ADR-0022), after checking its memory ceiling on real text.	The soak failed on sealer memory (356 MiB peak per 64 MiB file); the accepted design holds about 43 MiB on synthetic records; real text carries 2.7 times the records per file and the ceiling was never measured on it.	The milestone as recorded.	Hypothesis D6 run on the real-text journal before implementation starts; HIST-1/2 on its output.
D4	Count the tail in entries. Replace the by-bytes figure (3.2 ms per MiB) in every estimate with 4.7 µs per entry, and bound the tail by entries as well as bytes when the sealer falls behind.	Stock tail cost is 4.7 µs per entry on both the real and the synthetic tail; real text holds 2.7 times the entries in 64 MiB (real-corpus run 01).	A figure in the cost model and the operations page; possibly a journal_entries cap beside journal_bytes.	A1 run as a three-point ladder; a configuration change is a contract item.
D5	Adopt the Observation record and FOB1 as the node's payload, replacing the OTLP bytes in the Batch, with the custody table holding the blocks once and, if the rest of hypothesis D3 holds, no projections at all.	14.5 against 18.2 bytes per record compressed on real text; the custody table halves (encoding run 01); one copy answers a text search within 1.2 to 1.4× of the projection from a third of the bytes (optimality run 01); the codec is canonical, fuzzed, 532 ns to encode and 377 ns to decode to the view (cost model); traces fit the same record.	A wire-format change (a new payload slot or envelope version) and a Segment table change, each its own ADR; the Spindle's encoder; the server's custody reader.	A3 run (a block tail removes the per-entry decode) and D3 of the hypothesis suite (projections unnecessary over blocks) before the wire decision; ADR-0023 moves from Proposed to Accepted only then.
D6	If D5 is not taken, shrink the custody copy anyway: a hash per row group instead of per Batch (L-19), then a framing-once custody encoding (L-20).	The per-Batch hash is 17.6 % of a real-text Segment, the raw copy twice the body column (storage layout run 01).	A manifest version 2 (ADR-0020 amendment); replay verifies per row group.	B1 run with its flipped-byte control; B2's census.
D7	Keep time-first sort, Parquet projections and 8,192-row groups until a hypothesis says otherwise.	Encodings and sort order move logs by nothing and metrics by 16 to 29 % of a small share (storage layout run 01); the walk's floor on overlapping 1 MiB Segments is a single-row-group artefact until C1 is run.	None.	D4 (1,024-row groups), C1, D1, D2 of the suite are the tests that would change this.
D8	Measure on real text, and on counters. Retire the synthetic soak lines as the storage fixture; keep them for time where they agree. Add a captured counter series so rates can be tested at all.	The synthetic bodies compress the same under any encoding and hide the duplicate-copy question; the gauge-only generator left L-05's rate oracle vacuous.	The corpus fetch and generator option now in tools/research; one Spindle capture for B4 and C4.	None; a fixture decision.
D9	Make the specification the correctness reference for every read-path change, with the Python oracle as a diversity check on row derivation and the stock server as a regression reference.	query-semantics.md: the definition, eight theorems as properties, the walk and the budget proved equal to the definition on random inputs.	A proof-friendly restatement for the bounded proofs to finish; a model-based test of the server against spec over small journals.	The matrix rows QS-1 and QS-2; a registered check in its own commit.
D10	Keep traces out of the contract, with their place reserved.	The record carries locators on every kind at no cost when absent (one tag bit); the survey shows every store joins by trace_id columns and bloom filters.	None now.	A contract change if a consumer appears.

The optimality bounds page states the floor each decision approaches and the measured distance from it.

What the evidence is against

• A query engine, planner or IR. Every measured cost was removed by three mechanisms on the existing path; nothing in the random sets needed an operator the path lacks (direction review).
• Template or CLP-style log compression now. Block Zstd finds what the crude template split finds (0 to 15 % over the plain column); the known gains are in searching compressed archives at scales Fabric does not retain.
• Node-first or series-first sort. Nothing on logs, a few percent on metrics, and it would break the disjoint row-group property the walk and the pruning rely on.
• A wide-event single table. Metrics are 3 to 13 % of a Segment and want their own encodings; every surveyed system that tried kept them apart.
• Treating any program as the oracle. The stock differential proves sameness, not correctness; the Python oracle shares its author's reading of the contract. The specification is the reference, and the two programs are checks on different things.

The order of work

1. D9's model-based test, then D1's promotion under Q2 with the soak: a week.
2. D6 of the suite (the sealer's ceiling on real text), half a day, then D3's implementation as the milestone.
3. B3, A1, B1: the cheap storage facts, a week in all.
4. A3 and the suite's D3: the block tail and projection-free Segments; the wire decision (D5) rests on them.
5. B5 and C1 at product-size Segments, then the registered gates on the target host.