Tier: Advanced

qsv caches at four levels. The flagship is the stats cache — feeding pre-computed statistics into "smart" downstream commands so they don't redo work. The other three are the frequency cache, the fetch / fetchpost / describegpt HTTP cache, and the Luau lookup-table cache.

This page explains the file layouts, who reads them, when they invalidate, and how to deliberately bust them.

qsv leans heavily on sidecar files next to your CSV. Run stats once, and downstream commands get smarter and faster without re-reading the source file. The convention is:

my_file.csv                        — source
my_file.csv.idx                    — row-offset index
my_file.stats.csv                  — human-readable stats CSV
my_file.stats.csv.data.jsonl       — machine-readable JSONL (the actual cache)
my_file.stats.csv.json             — cache signature (args + qsv version) used for invalidation

The minimum to be useful:

qsv stats --stats-jsonl my_file.csv

The full version for max downstream benefit:

qsv stats --everything --infer-dates --infer-boolean --cardinality --stats-jsonl my_file.csv

Or — set an env var to let qsv create it automatically the next time a smart command needs it:

export QSV_STATSCACHE_MODE=force

Three modes:

• auto (default) — use the cache if it exists and is fresh
• force — auto-create the cache on first smart-command invocation
• none — ignore the cache entirely

• frequency — short-circuits all-unique columns (rowcount == cardinality) using a sentinel ALL_UNIQUE value; needs the cache to handle ID columns without OOM
• schema — skips redundant type inference; faster generation
• validate — uses inferred ranges, formats, and patterns from the cache
• pragmastat — date-aware mode requires the cache (stats -E --infer-dates --stats-jsonl)
• pivotp — smart aggregation auto-selection based on data type
• tojsonl — smart JSON type inference (string/number/bool/null per column)
• sample — extra checks for --systematic, --weighted, --cluster modes
• sqlp scoresql — query plan analysis (filter selectivity, join cardinality)
• describegpt — deterministic statistical context for the LLM

The cache is timestamp-keyed against the source CSV. If the source file's mtime changes, the cache is considered stale and silently regenerated (unless QSV_STATSCACHE_MODE=none).

Deliberate bust:

# Delete the whole set, including the .json signature sidecar, so a signature
# never outlives the statistics it describes.
rm my_file.stats.csv my_file.stats.csv.data.jsonl my_file.stats.csv.json
# Or force a rerun:
qsv stats --force --stats-jsonl my_file.csv

1. An args-changed recompute left a stale sidecar behind (#4439). Deleting <FILESTEM>.stats.csv on an args change did not delete its <FILESTEM>.stats.csv.json signature sidecar, so the next run validated against the stale sidecar and trusted it. The reproducer is three ordinary runs:

qsv stats f.csv -E -c 1      # writes the full cache
qsv stats f.csv --typesonly  # args changed: CSV deleted, sidecar survives
qsv stats f.csv -E           # served the --typesonly output as though it were --everything

schema, frequency, tojsonl and viz smart all inherited that wrong answer. The sidecar is now deleted with its CSV, as a unit.

2. A CSV with an empty column name broke every stats-cache consumer (#4410, #4412). The CSV→JSON conversion behind the cache omitted any key whose cell was empty, so a column with an empty name lost its field key — and field was the only non-optional string in the cache record, so the cache then failed to deserialize with missing field `field` . Reported against tojsonl, but schema and frequency failed identically. field is now always emitted, keyed by name rather than position, so a future reordering of the stats columns cannot silently reintroduce it. field is also now #[serde(default)], so the poisoned caches already written to disk — which still pass mtime validation after an upgrade — keep loading rather than hard-failing.

3. Date renderings are no longer coerced to 0.0 in the cache (#4441). On a Date/DateTime column, cells like median are type-dependent RFC3339 renderings, not numbers, but were typed as floats for the JSON conversion — which coerced them to 0.0. They are now carried losslessly as strings; a consumer that wants a number parses it, using the row's type to know that's meaningful. Pre-fix caches stay on disk and still pass mtime validation, so the fabricated zeros a pre-fix cache carries for a date row are dropped at load time — only a rendering that reads as a number on a date row is dropped, so this is a no-op for any cache written after the fix.

my_file.freq.csv.data.jsonl        — frequency distribution per column

qsv frequency --frequency-jsonl my_file.csv

Stored as JSONL with metadata per column. Two sentinel values appear for memory-bounded columns:

• ALL_UNIQUE — when rowcount == cardinality (typical of ID columns)
• HIGH_CARDINALITY — when cardinality exceeds the smaller of QSV_FREQ_HIGH_CARD_THRESHOLD (default 100) and QSV_FREQ_HIGH_CARD_PCT (default 90%) of rowcount

• describegpt — frequency context for LLM data dictionaries
• sqlp scoresql — filter-selectivity scoring

Same as the stats cache: stale if source CSV's mtime is newer. The cache is NOT used when --ignore-case, --no-trim, or --weight are active (those change how values are bucketed). Use --force to regenerate.

my_file.pschema.json               — Polars schema (column types + Polars dtypes)

qsv schema --polars my_file.csv

Every Polars-powered command: sqlp, joinp, pivotp (and lens, count, color, prompt, scoresql when reading Polars-supported formats).

Pre-computed Polars dtypes mean Polars skips the inference scan it would otherwise do. Big speedup for repeated queries against the same file. Generate it once, check it into git, every Polars command becomes faster.

If the file's columns change, regenerate:

qsv schema --polars --force my_file.csv

Affected caches: fetch/fetchpost (--disk-cache & Redis), geocode (the OpenCage result cache), apply summarize, and describegpt.

redb cannot read sled's format, so entries are recomputed on first run after upgrading. Expect one slow run, then normal speed. Redis caches self-heal; nothing to do.

This applies to QSV_DISKCACHE_TTL_SECS, QSV_REDIS_TTL_SECS, geocode --cache-ttl, and apply summarize. cached v3 rejects a zero TTL at build time, so 0 had to be given a meaning — and it got the opposite one.

• geocode cache-info reports the entry count as "unknown", not 0. RedbCache::cache_size() is Ok(None) by design, and reporting 0 would be actively misleading.
• The recomputable external-API caches (fetch/fetchpost disk cache, geocode OpenCage lookups) keep sled's non-durable behavior via .durable(false), avoiding an fsync per write on the per-row hot paths.

Four cache backends, chosen via CLI flag:

1. In-memory LRU (default) — non-persistent, 2M entries, lost on process exit. Tune with --mem-cache-size.
2. Disk (--disk-cache) — stored at ~/.qsv-cache/fetch/ (or --disk-cache-dir). TTL: 28 days (QSV_DISKCACHE_TTL_SECS).
3. Redis (--redis-cache) — shared across machines. Default redis://127.0.0.1:6379/1. TTL: 28 days (QSV_REDIS_TTL_SECS).
4. None (--no-cache) — for live data.

Cache keys are the URL + GET parameters. Two calls to the same URL return the cached value (within TTL).

Cache keys include the POST body — so two identical posts share a cache slot.

Cache keys include the prompt and the data context — so identical questions against unchanged data are cached.

By TTL or by manual delete:

# Disk cache
rm -rf ~/.qsv-cache/fetch/

# Redis cache (database 1 for fetch, 2 for fetchpost, 3 for describegpt)
redis-cli -n 1 FLUSHDB

Or set QSV_DISKCACHE_TTL_REFRESH / QSV_REDIS_TTL_REFRESH to refresh TTL on cache hit (keeps hot URLs cached indefinitely).

See HTTP & Web and Recipe: Fetch & Cache.

When a Luau script calls qsv_register_lookup("table", "URL_OR_PATH"):

• Local files are read directly.
• Remote URLs (http://, https://, dathere://, ckan://) are downloaded and cached in $QSV_CACHE_DIR (default ~/.qsv-cache/).

The cache key includes the URL. Subsequent runs reuse the cached copy until you delete it.

See Lookup Tables and Scripting (Luau / Python) → luau.

• Commit .stats.csv.data.jsonl and .pschema.json to git for reference datasets you query repeatedly. Tiny files, huge speedup.
• Don't commit .idx — they're regenerated cheaply and platform-specific.
• In CI, prefer QSV_STATSCACHE_MODE=force — eliminates "did I remember to run stats first?" mistakes.
• For multi-stage Make-based pipelines, list the cache files as explicit dependencies so make invalidates them when the source changes.

• Performance Tuning — when each cache matters
• Environment Variables — QSV_STATSCACHE_MODE, QSV_CACHE_DIR, TTL vars
• docs/PERFORMANCE.md — canonical performance reference
• docs/STATS_DEFINITIONS.md — what's in the stats cache
• Aggregation & Statistics → stats
• Validation & Schema → schema
• SQL & Polars — .pschema.json for sqlp / joinp / pivotp
• HTTP & Web — fetch / fetchpost cache modes
• Lookup Tables — Luau cache
• Recipe: Build a Data Pipeline — practical cache layout