Tier: Intermediate Commands covered: get

get fetches tabular data once from a local path, an HTTP(S) URL, a CKAN portal, or cloud object storage, and stores it in a managed, queryable disk cache. Once cached, the resource can be read by any qsv command via the dc: prefix — no re-download, no re-parse.

Think of it as the acquisition layer that sits in front of the rest of qsv: get owns "how do I get this file and keep it fresh?", and every other command just reads dc:<name>.

For each source, get:

• Stores the blob compressed (zstd by default) and content-addressed by BLAKE3 — identical content is stored once.
• Sniffs the delimiter of CSV/TSV ingests so the cached copy parses correctly even when the source uses a non-comma delimiter.
• Auto-builds a qsv index for the cached file, so downstream commands get instant random access and exact record counts.
• Records rich metadata — ETag, Last-Modified, compressed/uncompressed sizes, record count, and TTL.
• Revalidates instead of re-downloading. Re-fetches send a conditional request (If-None-Match / If-Modified-Since); an unchanged resource returns 304 Not Modified and the cached copy is kept.
• Streams large remote objects in parallel byte-ranges — tune with QSV_GET_PART_SIZE (default 8 MiB) and QSV_GET_CONCURRENCY (default 4, clamped 1–64).

sftp:// is planned for a later release.

Cloud credentials are read from the standard AWS_* / AZURE_* / GOOGLE_* environment variables (and IAM roles). Use --cloud-opt key=value (repeatable) for one-off overrides such as region=us-east-1, a custom endpoint=…, or skip_signature=true for public buckets.

# Fetch a CSV into the cache, then read it with another command
qsv get https://example.com/data.csv --name data.csv
qsv stats dc:data.csv

# Seed a CKAN reference table by name
qsv get "ckan://covid-vaccinations?" --name vax.csv

# Cloud object storage (requires the get_cloud feature)
qsv get s3://my-bucket/data.csv --name data.csv
qsv get gs://my-bucket/data.csv --cloud-opt skip_signature=true

Once a resource is cached under a logical name, any qsv command reads it by prefixing that name with dc::

qsv get https://example.com/big.csv --name big.csv
qsv stats     dc:big.csv
qsv frequency dc:big.csv
qsv sqlp      dc:big.csv "SELECT * FROM big WHERE amount > 100"

• The logical name is set with --name. If omitted, it defaults to the source's terminal path segment (e.g. big.csv from …/big.csv). --name is ignored when multiple sources are given.
• Reading a dc: handle transparently decompresses the blob to a temp CSV with a sibling .idx.
• New in 24.0.0: a frequency --frequency-jsonl dc:<name> cache is saved into the durable disk cache beside the stats cache and restored when the temp CSV is recreated, so the next frequency/viz run on the same dc: dataset hits it instead of recomputing (#4641). get cache-prune/cache-clear reclaim it. Re-materializing a dc: dataset also now streams the blob to disk rather than decoding it in memory first.
• Stale dc: entries are auto-refreshed according to the entry's refresh policy (see below) — a dc: read can trigger a conditional revalidation before handing the data to the command.

--sample N is a cheap peek at a single source — it streams just the first N data records (re-attaching the sniffed header row) to stdout or --output and caches nothing. No dc: entry is created. Because it stops reading early, a huge remote file is barely touched.

# Peek at the first 10 rows of a remote CSV without caching it
qsv get https://example.com/big.csv --sample 10

# Skip ~500 MB, then peek 10 rows
qsv get https://example.com/big.csv --offset 500 --sample 10

# Uniform random preview of 20 rows
qsv get https://example.com/big.csv --sample 20 --random

A glob (e.g. data/*.csv) or directory source fetches every matching tabular file (.csv/.tsv/.tab/.ssv) in one call — each cached separately under its own dc: handle. Supported for local paths and, with the get_cloud feature, cloud buckets/prefixes. --name is ignored when a source expands to multiple files.

# Cache every CSV in a local directory
qsv get '/data/*.csv'
qsv get /data/

# Cache every matching object under an S3 prefix (requires get_cloud)
qsv get 's3://my-bucket/exports/*.csv'

Each cache entry carries a TTL and a refresh policy that together decide what a dc: read does when the entry ages out:

As of 23.0.1, cache-list reports each entry's age and whether it is stale, so you can judge a cached response before reusing it rather than after.

"Stale" here means exactly one thing — will this entry re-fetch on its next dc: use? — and cache-list reads it from the same predicate the refresh path gates on, so the two can never disagree. An entry is due for refresh when its refresh policy is not never, its TTL is non-negative, and its age has reached that TTL. Note this is TTL-driven, not policy-driven: --refresh always does not make an entry due sooner, it only makes the eventual fetch unconditional.

Age is clamped at zero, because downloaded_at can legitimately sit in the future — a backward clock step from an NTP correction or a resumed VM, or metadata written by another machine when QSV_CACHE_DIR points at a shared mount. A future timestamp reads as "just fetched" rather than as a negative age.

cache-list / cache-info accept --json for machine-readable output.

# Inspect, verify integrity, then prune
qsv get cache-list
qsv get cache-list --verify
qsv get cache-prune --older-than=30d

# Retune a single entry
qsv get cache-set-ttl   data.csv --ttl=86400
qsv get cache-set-policy data.csv --refresh=never

• --sample <n> / --offset <mb> / --random — preview a single source without caching (see Sampling preview above).
• --timeout <secs> — request timeout (default 60; 0 = no timeout). For cache downloads it's an inactivity timeout; for preview mode (--sample/--offset/--random) a total-request timeout.
• --cache-dir <dir> — cache directory (default ~/.qsv-cache; overrides QSV_CACHE_DIR).
• -o, --output <file> — for a single source, also write the fetched (decompressed) data to <file> (use - for stdout).
• -q, --quiet — suppress progress/summary output on stderr.

See Environment Variables for the full list.

These solve different problems:

• get fetches one resource (a whole file) into a managed cache, so any command can read it via dc:. Acquisition + caching of datasets.
• fetch / fetchpost make one HTTP call per row to enrich a CSV from an API, caching individual responses. Per-row enrichment.

Reach for get to pull a dataset once and reuse it; reach for fetch to hit an API for every row.

• Command Reference (index)
• docs/help/get.md — canonical flag reference
• HTTP & Web → fetch / fetchpost — per-row HTTP enrichment
• Stats Cache & Caching — the wider qsv caching story
• Recipe: CKAN Integration
• Environment Variables