Historical report imported 2026-10-10 from docs/research/application-use-cases.md. Original SHA-256: ebce969d9248d4c216862cde549aba464bae4eb42fc41e8ada5332987873300a. Measurements and outcomes retain their stated revisions. Links and local environment labels are normalized; $FABRIC_DATA is the mounted experiment drive. See evidence policy.

Application, deployment and user-experience investigations

Status: source-grounded research design, 2026-10-09. The owner requested broader RCA, application performance, Nomos/configuration, platform-build and website use cases. These are proposed consumers and experiments, not implemented integrations, registered benchmark results or new product promises. Nomos's repository and API have not been identified; its adapter below is deliberately generic. This design grants no new experiment time or resource allocation.

One connected investigation

Fabric could connect a deployed change → application execution → user outcome. For example: a visitor clicks Submit, the API waits 800 ms for a connection, the request fails, and the interface never confirms success. A recent pool-limit change is a hypothesis. Establish which instances actually applied it, compare matched traffic, then test a controlled restoration. A nearby deployment event alone does not identify the cause.

The evidence must distinguish five events: user intent, browser request, server processing, durable business outcome and user-visible confirmation. A click is not a purchase; an HTTP success is not necessarily a completed job; a completed job is not proof the browser displayed confirmation. That separation also makes the same investigation useful for background jobs and agent tools.

Use case	Useful question	Smallest useful deliverable
RCA	Where did this request wait, and which explanations survive the evidence?	Trace/log/metric timeline with evidence references, competing explanations and missing observations
Application performance	Which route or dependency regressed under comparable work?	Per-route outcomes, latency distributions, throughput and wait/resource attribution
Nomos deployment/configuration evaluation	Did the intended version/configuration actually run, and improve the affected workload?	Desired-versus-observed rollout record and a comparison labelled improved, regressed or inconclusive
Platform releases and build pipelines	Is this artifact cheaper to build and run while producing equivalent successful outcomes?	Reproducible build/job and runtime comparison with cache, platform and resource context
Website experience	Which interactions fail or feel slow, on which routes/devices, and why?	Bounded interaction/error/performance events linked to backend requests where propagation exists
Background and data pipelines	Is a job stuck, retrying, duplicated or completing too late?	Job/attempt identity, queue and service time, retry history and independent completion ledger
Agent/tool workflows	Was time or cost spent on useful work, retries or an incorrect result?	Run/step/tool records, measured usage where available, verified outcomes and failure evidence
Capacity and telemetry health	Are we outgrowing the deployment, or merely losing visibility?	Actual offer/accept/query rates, queue age, custody, per-signal coverage and retention

This is an operational evidence direction. Session replay, billing-grade product analytics, autonomous deployment/rollback and a security analytics platform are separate scopes, not implied features.

Current foundation and missing pieces

Boundary	Present repository evidence	Needed for these uses
Intake	Spindle collects bounded local newline logs and host metrics; loopback OTLP/HTTP protobuf trace intake answers after Spool commit	Browser/application event intake and deployment/build adapters; general application metric/histogram intake is absent
Custody	Durable Spool, replay, server ACK and storage lifecycle	Explicit mapping of each application's acceptance boundary onto custody; client queuing is not an ACK
Search	Log body substring, named metric history/rate, span trace-ID/name, snapshot pagination and answer metadata	Client correlation and statistics first; arbitrary attribute predicates, joins, percentiles and SLO evaluation are not current query features
Identity	Node/Batch identities; trace/span/parent IDs; projected string row attributes	Artifact/configuration/rollout provenance and stable application event/job IDs; resource attributes are not exposed by the current row projection
Configuration	Spindle desired/applied collection configuration, revision and error observations	Nomos desired/observed deployment state; Spindle configuration is not application deployment state
Consumers	CLI and authenticated structured HTTP responses	Investigation/comparison views; browser dashboard and agent facade are proposals; query currently uses admin authorization, not a dedicated read-only role
Completeness	Retained-scan completeness, gaps, unavailable sources, retained bounds and node freshness	Per-signal/application coverage and independent outcome denominators; fresh host metrics cannot certify trace or log completeness

Source boundaries: product contract, Spindle, retained history, control, consumer experience, trace decision. Implementation anchors: trace intake, row projection, queries, CLI. Nothing here changes those interfaces.

Proposed integration boundary

flowchart LR
    Browser[Browser events - proposed] --> Intake[Same-origin application intake - proposed]
    Nomos[Nomos and build records - proposed] --> Adapter[Server-side adapter - proposed]
    Intake --> Adapter
    Adapter -->|bounded local logs or supported trace exports| Node[Existing Spindle]
    App[Instrumented backend] -->|local logs and traces| Node
    Node --> Spool[(Spool)]
    Spool --> Server[Fabric server]
    Server --> Query[Existing query API]
    Query --> Client[Investigation and comparison client - proposed]

Start with application-owned structured log lines within current line limits, plus existing trace export, rather than changing FAB1 or putting browser code inside the semantic core. The client can parse selected structured log bodies; there is no implied server-side JSON/attribute search. Numeric application data in log events is not equivalent to an implemented metric receiver. The existing RCA packet uses synthetic Gauge/Sum Batches solely as a laboratory fixture.

Proposed correlation fields are service/environment, artifact and configuration digests, rollout/instance identity, request/trace identity, event/job/attempt ID, route template, named action, outcome, source time and sampling policy. Record receive/commit/ACK/query observations separately. Use bounded dimensions for metrics; request, session and trace IDs must not become unbounded metric labels. Keep raw evidence and links to the exact query snapshot behind derived reports.

The browser crosses a different trust boundary from an enrolled Spindle. A same-origin application intake should validate scope, body size, rate and allowlisted fields, then use server-side credentials. Never ship Fabric admin or node credentials in JavaScript. Treat browser IDs and timestamps as untrusted; prevent cross-tenant/session joins and distinguish replay of an event from a second genuine action. Choose route templates and named controls rather than raw form contents, DOM text or URL query strings. Detailed replay is unnecessary for the first useful website investigation.

As of the source review, OpenTelemetry JavaScript marks browser instrumentation experimental and largely unspecified. Browser OTLP transport uses HTTP rather than gRPC and needs appropriate CSP/CORS/proxy configuration. OTel JavaScript, exporter guidance. The existing Fabric trace listener is loopback-only: it is not a public browser collector. Browser compatibility must be tested for the actual site.

For user experience, record LCP, INP and CLS with page/route/device context; distinguish field measurements from laboratory tests. INP requires interactions, so a noninteractive page-load trial cannot establish interaction quality. Web Vitals. Observe JavaScript errors, unhandled rejections and failed application requests with explicit capture limits. Cross-origin visibility and buffer loss mean absence of browser errors is not proof of a healthy session. Performance data, error events, unhandled rejections. sendBeacon() returning true establishes browser queuing, not server receipt or durable Fabric custody. Track queued, application-accepted, Spool-committed, server-ACKed and query-visible events as separate observations. Beacon contract.

Comparison and diagnosis rules

Evaluate only comparable populations. Preserve attempted, admitted, completed, failed and timed-out work; sampled traces are not the request denominator. For deployment trials, record per-instance observed applied artifact/configuration and partial-rollout state, not just the controller's desired state. For builds, freeze source/dependency/toolchain/platform identity and distinguish full versus incremental and cold versus warm-cache work. Faster failed jobs are not a win.

A comparison is eligible only when workload/outcome definitions, applied-state coverage, independent denominators, observation coverage and clock/retention boundaries support it. Missing evidence produces inconclusive, not improved. For eligible trials, report effects and uncertainty by cohort and repeat; freeze practical improvement/regression tolerances before execution. Include errors, timeouts and missing outcomes. Do not average per-instance percentiles to invent a fleet percentile; reconstruct the relevant distribution or report its limits.

A synthetic confounder illustrates why: baseline traffic has 900 fast requests at 100 ms and 100 slow requests at 500 ms, averaging 140 ms. A candidate has 990 fast requests at 110 ms and 10 slow requests at 550 ms, averaging 114.4 ms. Its overall mean looks 18.3% better although both cohorts are 10% worse. At the baseline 90/10 mix, the candidate averages 154 ms. This is arithmetic, not a Fabric measurement. Randomized/interleaved matched trials and per-cohort outcomes are the discriminators; before/after timestamps alone do not solve it.

An investigation should expose: observed symptom, supported immediate mechanism, competing causes, evidence references, coverage limitations, and the cheapest next discriminating probe. Preserve revisions when late evidence arrives. Separate query correctness from interpretation correctness. Cross-query answers do not currently share a transactional snapshot; record that limitation.

Experiment cards and laboratory ownership

U1 now has native journal/restart evidence: seven cases under both plans, 350 exact chains and 30 rejected corruptions. Published-storage and interpretation checks remain unrun. All other cards are proposed/unrun. Each needs a separate registered execution protocol with frozen inputs, commands, scoring, resource/time limits and rejection controls before dispatch. H1 is the claim to test; H0 is the competing explanation or no-useful-effect outcome. The coordinator remains accountable until a scoped PI assignment. Keep the existing three labs, and give assistants bounded subtasks.

Card / primary lab	H1 and smallest discriminating workload	H0 / control that must prevent a false success	Record
U1 RCA / Query	Existing seven scenarios yield supported mechanisms or justified abstention across pending, late, published and restarted views	Wrong attribution, leaked controller truth, missing evidence treated as health, or late rows entering an old snapshot	Independent exact-query verdict; supported/unsupported claims; abstention; query count/time/bytes; evidence references
U2 APM / Query	Fixed-route requests with injected pool, dependency and CPU waits can be distinguished with targeted instrumentation	Same total latency with different mechanism; overlapping spans; uninstrumented wait must remain unresolved	Request success/error/timeout denominator, p50/p95/p99, queue/service time, CPU, trace coverage and agent usage if used
U3 Nomos rollout / Collection	Per-instance applied-state records associate outcomes with the actually running candidate	Desired-only state, failed or partial rollout, stale post-restart observations	Desired/observed artifact and config digests, rollout fraction, observation age, failed applies, exposure by cohort
U4 Config comparison / Query	A controlled configuration change improves matched successful work without hiding failures	Remove a log source while host metrics stay fresh; change traffic mix; change sampling	Matched cohort effects, independent attempted/completed counts, telemetry coverage, uncertainty and inconclusive rate
U5 Platform/build / Storage	Candidate reduces CPU/RSS/time per equivalent successful build or runtime job	Warm-cache candidate against cold baseline; different artifact/toolchain; faster failed output	Wall/CPU time, peak RSS and cgroup memory, disk read/write/peak bytes, artifact validity, cache and environment identity
U6 Website journey / Collection	A bounded click → request → backend span → visible outcome chain identifies a failed interaction	Double click versus retransmission; wrong session/tenant join; missing propagation; API succeeds but UI fails	Eligible interactions and outcomes, browser overhead, dropped events, correlation coverage, latency and body/cardinality bounds
U7 Browser delivery / Collection	Intake exposes loss/retry/backpressure without claiming unreceived evidence is durable	Beacon queues then network drops; intake 429; server outage; oversized events; cross-origin errors invisible	Counts at each acceptance stage, duplicate handling, reject/drop reasons, queue age/bytes, receipt boundaries
U8 Web experience / Query	Route/device cohorts distinguish slow rendering, slow interaction and backend delay	Changed device mix, background-tab effects, no interactions, telemetry suppression	LCP/INP/CLS distributions and eligible samples, errors, route/device mix, missing measurements and capture overhead
U9 Mixed signals / Storage	Logs, metrics, spans and event metadata stay bounded with useful selective queries under the 4 GB server cap	Unique IDs treated as metric labels; large attributes; trace-heavy load crowding out logs	Actual rates by signal, ACK/visibility/query tails, RSS/cgroup/cache memory, CPU, disk/retention, exact custody and query results
U10 Jobs and agents / Query	Run/job/attempt identity explains retries and distinguishes completion from useful success	Tool returns success but output fails independent validation; replay counted as new completion; late correction	Queue/run/tool time, retry/replay counts, validated outcomes, measured token/cost usage where available, evidence revision
U11 Telemetry health / Collection	A coverage-aware report distinguishes healthy traffic from missing/stale collection	Quiet app, disconnected source, removed path and sampled spans can all yield zero errors	Per-source/signal progress, known gaps, unknown coverage, independent traffic count, freshness and retention
U12 Investigation cost / Storage	Bounded query windows, projections and selective retrieval reduce investigation cost without changing answers	Narrow window excludes skewed/long spans; early stopping omits evidence; broad fallback overwhelms memory	Exact full-chain equivalence, sources/rows/bytes read, CPU/RSS, latency, query budget, fallback and abstention

Collection means the collection/reliability lab; Storage means storage/memory; Query means query/investigation. Each card has one owner and cross-lab review where custody, storage or query boundaries meet. A useful performance win must retain evidence fidelity and outcome equivalence. No combined score can offset lost evidence with faster responses.

Execution order and product increment

1. Pressure final checks and owned cleanup are complete, with the original second-Scan launcher receipt still missing. Packet validation and the native U1 journal/restart matrix have run; next add the separately scoped published-storage and interpretation checks. The fixture-only protocol is registered separately (local checkout reference docs/experiments/benchmarks/use-case-preparation-protocol.md; not present at origin/main).
2. Implement one instrumented demo API and a bounded investigation client for U2/U4/U11. Use independent request records and client-computed statistics; establish app evidence quality before adding dashboards or automatic advice.
3. Add one generic deployment/build recorder for U3/U5, then bind it to Nomos after its real interface and identity are known. Report comparison results; do not automatically change production configuration or roll back.
4. Add one demo website flow with the proposed intake boundary, U6/U7/U8. First test a success, an API failure and a browser-only failure with receipt and loss controls. This provides a useful human view and the same structured evidence for an agent.
5. Exercise U9/U12 under concurrent mixed signals; extend U10 to one background job and one tool workflow. General metric intake, histogram semantics, read-only authorization and richer filters need explicit implementation scope and contract/design decisions where applicable.

Start in local development, then use a small multi-service deployment with the same evidence contracts. Match actual workload and compare conditions on the same host before interpreting cross-host differences. Fabric stays capped at 4 decimal GB (page-aligned down); the local laboratory stays within 20 GiB, swap disabled, on the mounted data drive, within the 100 GB storage ceiling. Every workload/validator uses the resource launcher and existing finite ledger; remote edges retain smaller host-specific caps and owned scratch cleanup.

The near-term usable product is a small investigation and change-evaluation workbench: explain a slow/failed request, compare an applied change, or follow a broken website interaction, with exact evidence and visible uncertainty. Its value depends as much on refusing misleading conclusions as on fast search.