Research storage prior art - kmosoti/FabricO11y GitHub Wiki

Historical report imported 2026-10-10 from docs/research/storage-prior-art.md. Original SHA-256: 323552685328814e1baf883640b8ecbc5da5f6dde62b9e20954fded30b04bafe. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Prior art: physical storage of logs, metrics and traces

Status: research notes, tier 6 of the source-of-truth order. A web survey made on 2026-10-02 for the storage direction. Every claim carries its source; items marked unverified could not be confirmed from a primary source. Nothing here is a decision.

The Turso source review examines a pinned Rust engine revision for storage lookup, memory ownership, borrowed reads, cost models and deterministic testing. It compares those mechanisms with Fabric's implementation; it records no executable performance result or database adoption decision.

The cross-system program adds the 2026-10-08 source reviews, coupled cost models and draft three-lab queue for memory accounting, compressed representation, selective computation and ownership/recovery. It distinguishes current implementation from research candidates. The later source synthesis links downloaded implementation dissections and the admitted continuation experiments; the older survey below remains historical context.

Summary

Production systems split into three families. (1) Per-kind engines: Prometheus/Mimir TSDB for metrics, Loki chunks for logs, Tempo Parquet blocks for traces, each with a layout tuned to one access pattern. (2) One general columnar engine with per-kind tables: ClickHouse deployments (SigNoz, ClickStack/HyperDX, Uptrace, Jaeger), Elastic (TSDS and LogsDB index modes), and Parquet-on-object-storage systems (InfluxDB 3, OpenObserve, Parseable, GreptimeDB) keep logs, traces and metrics in separate tables or streams but share one storage engine, codecs and query planner. (3) One row model for everything: Honeycomb and Meta's Scuba store "wide events" and derive traces from trace_id/span_id fields; Datadog Husky stores logs, traces, RUM and profiles as "structured logs" in one custom column store. Even in family (3), metrics are the exception: Honeycomb stores metrics in dedicated metrics datasets, and Monarch/Gorilla show why dense numeric streams want delta-of-delta and XOR encodings rather than a sparse event layout.

The encoding facts that recur: dictionary-encode low-cardinality strings; store attributes as typed columns (static, dedicated, or dynamically shredded) rather than JSON blobs when queries must be cheap; sort by a low-cardinality grouping key (tenant, service, resource fingerprint, _tsid) before time; keep trace_id/span_id as first-class columns in every kind; and for log bodies, separate the static template from variables (CLP, Drain, LogGrep).

OpenTelemetry data model and OTel Arrow (OTAP)

Grafana Tempo (traces, Parquet)

Grafana Loki (logs) and Prometheus/Mimir (metrics)

Gorilla (VLDB 2015) and Monarch (VLDB 2020)

  • Gorilla: delta-of-delta timestamps and XOR floats "compress each by series down from 16 bytes to an average of 1.37 bytes, a 12x reduction"; two-hour blocks were chosen because longer blocks gave "diminishing returns". https://www.vldb.org/pvldb/vol8/p1816-teller.pdf (2015)
  • Monarch: tables have a target schema and a metric schema; "target ranges are used for lexicographic sharding and load balancing among leaves"; the in-memory store does "only light compression such as timestamp sharing and delta encoding", with one timestamp sequence shared by about ten series; finalized points use delta and run-length encoding; distributions carry exemplars. https://www.vldb.org/pvldb/vol13/p3181-adams.pdf (2020)

ClickHouse-based stores: SigNoz, ClickStack, Uptrace, Jaeger

Datadog Husky

  • Third-generation event store for logs, traces, NPM, RUM and profiler events, all treated as "structured logs"; custom columnar file format on object storage with a FoundationDB-backed metadata store. https://www.datadoghq.com/blog/engineering/introducing-husky/ (2022-05-17)
  • Fragments hold many columns, each with a header and fixed-size row groups; writers emit about 1,000-row fragments, size-tiered lazy compaction yields about 1M rows; fragments are bucketed by tenant and time window and rows are sorted by tags such as service, status, env, timestamp; "locality compaction" reduced query worker replicas by 30%. https://www.datadoghq.com/blog/engineering/husky-storage-compaction/ (2025-01-29)

Honeycomb (Retriever) and Scuba

Elastic TSDS and LogsDB

Quickwit, Parseable, OpenObserve, InfluxDB 3, GreptimeDB

Log compression papers: CLP, CLP-S, LogReducer, LogGrep, Drain

Columnar format references: Procella, Lance, Iceberg/Delta, Parquet Variant

The "wide events" argument and its critics

Comparison table

System One engine for all three? Row model Attribute encoding Sort key Log-body encoding Reported compression
OTAP (wire) One format, per-signal batches Normalized batches + attribute tables Dictionary + typed value columns, parent_id Sorted for locality Body as AnyValue column 1.6-2x (logs) to 3-7x (multivariate metrics) over OTLP+zstd
Tempo Traces only One row per trace, nested Key/typed-value lists + up to 20 dedicated cols Trace ID n/a dedicated cols: -75% tail latency; no ratio given
Loki + TSDB Logs only Stream chunks Labels in index; structured metadata symbols Stream, time Raw lines, gzip/snappy/lz4 5-10x needed to fill 1.5 MB chunk
Prometheus/Mimir Metrics only Series chunks Labels in block index Series, time n/a 1-2 bytes/sample
Gorilla Metrics only In-memory series n/a Series, time n/a 1.37 bytes/point, 12x
Monarch Metrics only Schematized target+metric tables Key columns in target schema Lexicographic by target n/a timestamp sharing ~10 series/sequence
SigNoz (ClickHouse) One engine, per-kind tables Row per log/span/sample Typed Maps + JSON resource + materialized cols ts_bucket, resource_fingerprint, ... Body string + tokenbf blocks scanned 99.5% -> 0.85%
ClickStack One engine, per-kind tables Row per record Map(LowCardinality, String) ServiceName, Timestamp Body + tokenbf ZSTD(1); no ratio given
Jaeger/ClickHouse Traces (OTel model) Row per span Nested typed columns per level service, name, start_time n/a 8.6x
Husky One engine; metrics separate (unverified) Event rows Schemaless per-column tenant/time bucket, then tags, time event columns not published
Honeycomb Events; metrics datasets separate Wide events One file per field Time segments fields LZ4; no ratio
Scuba Events Wide events, sampled Per-column dict/bitpack/delta/lz4 Time fields not published
Elastic TSDS/LogsDB One engine, index modes Documents Doc values, flattened, dynamic mappings _tsid/host.name, @timestamp text + synthetic source -70% (TSDS), -65% (LogsDB)
Quickwit Logs, traces (Jaeger) Documents in splits fast fields + doc store time-pruned splits inverted index ~3x
InfluxDB 3 Metrics (table model) Rows in Parquet columns least-cardinality tags, time n/a 10-100x vs raw
OpenObserve / Parseable One engine, per-kind streams Rows in Parquet evolving schema time partitions columns vendor claims only
CLP / CLP-S Logs logtype + vars table dictionaries archive/segment template + dict + encoded vars 32x avg; 169x at Uber; 21.9-186.8x JSON

Design facts that transfer

  1. OTAP already defines a columnar, Parquet-compatible layout for all three signals: a main batch per signal plus attribute tables keyed by parent_id, with dictionary encoding and typed value columns. (OTEP 0156; otel-arrow data_model.md)
  2. Dictionary encoding of attribute keys and low-cardinality values is the single most repeated technique (OTAP, Tempo dedicated columns, ClickHouse LowCardinality, Scuba, CLP).
  3. Promoting the most-used attributes to dedicated typed columns pays: Tempo measured generic attribute columns at up to 70% of block size and the top ten span attributes at over 50% of attribute bytes. (Grafana, 2024-01-23)
  4. Dynamic typed subcolumns with an overflow bucket are the current answer to arbitrary attribute sets: ClickHouse JSON (max_dynamic_paths 1024, shared data column), Parquet Variant shredding (typed_value + binary value), Elastic flattened plus field limits.
  5. Sorting by a low-cardinality grouping key before time is reported to matter more than codec choice: SigNoz resource fingerprint (99.5% -> 0.85% blocks), Elastic _tsid/host.name (up to 30% from sorting alone), InfluxDB 3 least-cardinality-first, Husky service, status, env, timestamp, Jaeger service_name, name, start_time.
  6. Trace-ID-first sort is used only where the workload is "fetch one trace" (Tempo); search-first systems sort by service and keep a bloom filter on trace_id instead (Jaeger/ClickHouse, Tempo's per-block bloom).
  7. Time bucketing is layered on top of the sort key, not replaced by it: SigNoz ts_bucket_start (30 min), ClickStack day partitions, Prometheus/Mimir 2-hour blocks, Husky tenant/time buckets, Quickwit and Honeycomb segment min/max timestamps, Iceberg/Delta per-file min/max.
  8. Dense numeric series want delta-of-delta timestamps and XOR/Gorilla values: 1.37 bytes/point (Gorilla), 1-2 bytes/sample (Prometheus), and SigNoz applies the same codecs inside ClickHouse.
  9. Monarch's timestamp sharing (one timestamp sequence per ~10 series from the same target) is a cheap encoding for metrics that share a scrape; OTAP's multivariate-metrics gain (3-7x) is the same observation at the wire.
  10. Log bodies compress best when the static template is separated from variables: CLP 32x average versus gzip 16x, Uber 169x, μSlope 2.34x over zstd on JSON, LogReducer showing numeric fields dominate the residual.
  11. Nested resource > scope > record storage avoids repeating resource attributes per record (Tempo schema rationale, OTLP proto shape).
  12. Metrics remain the exception in "one row model" systems: Honeycomb stores metrics in dedicated datasets; Husky's published posts cover events, not metrics; the cost critique quotes $33,000/month versus $0.80/month for the same signal as wide events versus metrics.
  13. Cross-kind joins in practice are plain columns: trace_id/span_id on logs and spans in every schema surveyed, and exemplars carrying trace_id on metric points (OTel) with Prometheus holding exemplars only in memory and WAL.
  14. Reported chunk and file sizing: Loki 1.5 MB compressed chunks, Honeycomb about 1 GB / 1M events, Husky about 1M rows after compaction, Prometheus about 120 samples per chunk, Quickwit splits of about 10M docs.
  15. Vendor ratios (OpenObserve 140x, Uptrace 10:1) are unaccompanied by workload definitions and should not be used as design inputs without a registered workload.
⚠️ **GitHub.com Fallback** ⚠️