Tier: Advanced

This page is a workflow guide to performance — when to reach for which knob, in what order. Numbers quoted on this page link to BENCHMARKS.md as the source of truth.

If you have five minutes to make qsv faster on your workload, in order:

1. Index your file. qsv index data.csv — one-time cost (~14s on 15 GB). Speeds up 9 commands (and improves sniff's inference).
2. Pre-populate the stats cache. qsv stats --stats-jsonl data.csv — makes frequency, schema, validate, pragmastat, pivotp, describegpt, sqlp, and scoresql smarter.
3. Set QSV_AUTOINDEX_SIZE. Auto-creates the index for any file above the threshold; auto-refreshes stale ones.
4. For Polars commands, generate a Polars schema. qsv schema --polars data.csv — written to data.pschema.json, picked up automatically by sqlp / joinp / pivotp.
5. For files > RAM, use extsort / extdedup instead of sort / dedup.

Most users stop here. The rest of this page is for the remaining 1%.

New in 21.1.0, the qsv-tune script (scripts/qsv-tune.sh for Linux/macOS, scripts/qsv-tune.ps1 for Windows) inspects your machine's RAM, CPU cores, and disk type — plus the qsv binary's compiled-in memory allocator — and emits a tuned set of QSV_* environment variables in .env format.

It is dry-run by default: with no arguments it prints an annotated, ready-to-use .env block to stdout and writes nothing.

# Print recommended settings (writes nothing):
scripts/qsv-tune.sh

# Capture the suggestion:
scripts/qsv-tune.sh > my-qsv.env

# Merge the QSV_* block into ./.env (idempotent, with a .bak backup):
scripts/qsv-tune.sh --write

# Windows (best-effort):
.\scripts\qsv-tune.ps1
.\scripts\qsv-tune.ps1 -Write

Allocator tuning (jemalloc/mimalloc) is reported separately as export lines, not written into .env — allocators read their config at process start, before qsv loads .env, so those variables must live in the real shell environment.

The recommended values are heuristic starting points — always benchmark against your own data shapes (see docs/PERFORMANCE.md). Implements issue #2829.

qsv builds a <file>.idx sidecar that lets commands skip parsing rows they don't need.

The sniff win is about accuracy, not just speed: instead of inferring the schema from only the first --sample rows, an indexed file is sampled across its whole length — the first & last 5 rows, 5 rows each around the 25th/50th/75th percentiles, and the rest random — so column types and dates that only appear late in the file are detected. Falls back to the contiguous first-N sample when the file is not indexed.

This also sharpens stats date type inference. stats --infer-dates uses its default --dates-whitelist sniff, which runs sniff to decide which columns to treat as Date/DateTime before computing their summary stats. On an indexed file that sniff step uses the distributed sample, so date columns whose date values only appear late in the file are correctly typed (and get date-aware Min/Max) — whereas the first-N sample would miss them and leave the column inferred as text or a number.

Index size is typically ~1-2% of the source file. A 15 GB CSV indexes in ~14 seconds and the index is ~27 MB.

qsv index huge.csv
# Or — auto-index everything above 100 MB:
export QSV_AUTOINDEX_SIZE=100000000

See Indexing, Compression & Diff → index.

Running qsv stats --stats-jsonl <file> writes two sidecar files:

• <file>.stats.csv — human-readable
• <file>.stats.csv.data.jsonl — machine-readable (the file other commands actually consume)

Every "smart" command (🪄 in the README legend) automatically picks up the JSONL and uses it:

• frequency — short-circuits all-unique columns (would otherwise blow memory)
• schema — skips redundant type inference
• validate — faster generated-schema runs
• pragmastat — date-aware mode requires the cache
• pivotp — smart aggregation auto-selection
• tojsonl — smart JSON type inference
• sqlp scoresql — query plan analysis
• describegpt — deterministic statistical context for the LLM
• sample --systematic/--weighted/--cluster — extra checks against cardinality

Set QSV_STATSCACHE_MODE=force to automatically create the cache whenever a smart command runs on a file that doesn't have one yet. See Stats Cache & Caching.

qsv detects logical CPUs and uses all of them by default. Cap with QSV_MAX_JOBS=N if you want to share CPU with other workloads.

Always-multithreaded (🚀): apply, applydp, blake3, datefmt, dedup, diff, excel, extsort, geocode, joinp, jsonl, replace, snappy, sort, sqlp, template, to, tojsonl, validate.

Multithreaded with an index (🏎️): frequency, sample, schema, search, searchset, split, stats.

For the full mapping, see the README legend.

Commands marked 🤯 load the entire file into memory:

• dedup (unless --sorted)
• pragmastat
• reverse (unless an index is present — then constant memory)
• sort — for files > RAM use extsort
• stats — for --cardinality / --quartiles / --median modes
• table
• transpose (unless --multipass or --long)

Commands marked 😣 use memory proportional to column cardinality:

• frequency
• schema
• tojsonl

For huge files where you can't avoid memory mode, set the safety env vars:

export QSV_MEMORY_CHECK=1                  # opt into pre-check
export QSV_FREEMEMORY_HEADROOM_PCT=10      # reserve 10% of free RAM

With QSV_MEMORY_CHECK=1, qsv computes available memory (including swap), applies a platform multiplier (1.3× macOS, 1.15× Linux, 1.0× Windows), subtracts the headroom, and aborts if the file won't fit.

stats and frequency can switch to Apache DataSketches for constant-memory operation:

# stats: approximate quartiles (t-digest) and cardinality (HyperLogLog);
# under --everything, the cap is what bounds the mode/cardinality tables
qsv stats --everything --cardinality-method approx --quantile-method approx \
  --mode-cardinality-cap 10000 data.csv

# frequency: top-K with bounded error
qsv frequency --sketch-method frequent_items --sketch-map-size 16384 data.csv

Both add ~1.5% error in exchange for O(1) memory per column. Use when exact methods OOM.

New in 24.0.0: parallel stats runs warn when exact mode & cardinality tracking may not fit in memory (estimated > 80% of available). Those tables hold every distinct value of every column, and each chunk keeps its own copy, so --everything on a 1M-row, 44-column NYC 311 file peaks at 607 MB with --jobs 1 but 1.2 GB with --jobs 16. The fix the warning suggests: cap the tables, keeping approximate cardinalities.

qsv stats --everything --mode-cardinality-cap 10000 --cardinality-method approx data.csv

On that file a cap of 10000 cut peak memory by ~65% and runtime by half. Defaults and output are unchanged.

Also new in 24.0.0: frequency is 1.5× faster by default (2× with --weight) with lower peak memory, and stats --infer-dates (1.43×) and sniff (1.81×) no longer contend for shared regex locks across threads.

As of 21.1.0, the prebuilt qsv binaries on native targets are Profile-Guided-Optimization (PGO) tuned — the compiler optimizes hot paths based on real execution profiles, so you get extra performance out of the box with no action required (#1448). Just download a prebuilt binary.

24.0.0 exception — Apple Silicon. The 24.0.0 aarch64-apple-darwin prebuilt was built without PGO (#4740); see the note on Benchmarks. With Rust 1.99, the PGO-instrumented binary crashed before main(). The cause was the cc crate: it passed Rust's PGO flags on to Apple clang, whose profile format (v10) didn't match Rust 1.99's (v11, LLVM 23), so C dependencies such as jemalloc crashed the profiling runtime. 25.0.0: Apple Silicon PGO is restored. #4757 first worked around it, and cc 1.8.0 then fixed it upstream by no longer forwarding the flags (rust-lang/cc-rs#1986). With cc 1.8.0 in Cargo.lock, the workaround was removed (#4795).

Building your own PGO binary on macOS (with scripts/build-pgo.sh or cargo pgo): make sure Cargo.lock has cc 1.8.0 or newer. With an older cc, the instrumented binary segfaults before main() whenever Apple clang's LLVM major version differs from rustc's.

Through 23.0.x, x86_64 prebuilts targeted the baseline instruction set to avoid SIGILL on older CPUs. New in 24.0.0: they target x86-64-v3 — AVX2/FMA/BMI2 — so a native build mainly adds newer extensions such as AVX-512. If you build from source on your own machine, opt into your CPU's full instruction set:

CARGO_BUILD_RUSTFLAGS='-C target-cpu=native' \
  cargo build --release --locked --bin qsv -F all_features

Apple Silicon, Windows-on-ARM, IBM Power, and IBM Z prebuilds always have CPU optimizations enabled — no need to rebuild on those platforms.

qsv ships a nightly-toolchain prebuilt with extra optimizations enabled. See docs/PERFORMANCE.md#nightly-release-builds.

qsv builds with mimalloc by default. For huge-file workloads, jemalloc is sometimes faster. Both have their own environment variables — see the mimalloc options and jemalloc options docs.

To see which allocator your binary uses: qsv --version shows it in the version string.

export QSV_RDR_BUFFER_CAPACITY=131072    # reader buffer, default 128 KB
export QSV_WTR_BUFFER_CAPACITY=524288    # writer buffer, default 512 KB

Bump up on slow disks (network mounts, spinning disks). Bump down for memory-constrained environments.

Polars-powered commands (🐻‍❄️): color, count, joinp, lens, pivotp, prompt, schema, scoresql, sqlp. They respond to Polars's env vars:

export POLARS_VERBOSE=1                # log plan + execution to stderr
export POLARS_PANIC_ON_ERR=1           # panic instead of returning an error
export POLARS_BACKTRACE_IN_ERR=1       # include backtrace in errors

For the full set, see Polars env vars.

The single biggest Polars optimization is the Polars schema (qsv schema --polars data.csv). It tells Polars the column types without an inference scan.

• qsv index <file> (or QSV_AUTOINDEX_SIZE)
• qsv stats --stats-jsonl <file> (or QSV_STATSCACHE_MODE=force)
• qsv schema --polars <file> (for Polars commands)
• Replace sort → extsort, dedup → extdedup for files > RAM
• For huge cardinalities: --cardinality-method approx / --quantile-method approx
• For huge files: --sketch-method frequent_items in frequency
• Set QSV_MAX_JOBS to share CPU; QSV_MEMORY_CHECK=1 for safety
• Build with target-cpu=native if compiling from source
• Point extsort/extdedup at a fast spill location (NVMe preferred) with their tmp-dir flag (extsort --tmp-dir, extdedup --temp-dir), or the OS TMPDIR/TEMP env var
• If on x86_64 and getting SIGILL, build from source for your own CPU (expected on CPUs without AVX2 since 24.0.0) — the portable qsvp variant was retired in 22.0.1

• docs/PERFORMANCE.md — canonical reference
• docs/PERFORMANCE_TLDR.md — TL;DR
• docs/BENCHMARKS.md — methodology
• qsv.dathere.com/benchmarks — live benchmark dashboard
• Environment Variables — every QSV_* var
• Stats Cache & Caching
• Indexing, Compression & Diff
• Recipe: Larger-than-RAM CSV
• Binary Variants — which binary to build or download