Tier: Intermediate Commands covered: validate, schema, sniff (see also profile for standards-based metadata generation)

This is qsv's data quality story. Use schema to infer a JSON Schema 2020-12 spec from representative data, edit the schema by hand to tighten the rules, then validate to enforce it against new data at up to 780,000 rows/sec. Use sniff for fast metadata (delimiter, types, MIME) when you don't need a full schema scan.

For the deep-dive, see docs/Validate.md.

JSON Schema Validation Draft 2020-12 with the fastest validator in Rust (jsonschema-rs). Up to 780k rows/sec on NYC 311 with the bundled schema.

Without a schema argument, validate just verifies RFC 4180 conformance and UTF-8 — a cheap sanity check before running stats (which assumes well-formed input).

With a schema, three output files are produced:

• <input>.valid.csv — rows that passed
• <input>.invalid.csv — rows that failed
• <input>.validation-errors.tsv — row_number, field, error for every failure

One custom format and two custom keywords beyond stock JSON Schema 2020-12:

• "format": "currency" — ISO 4217 currency validation (USD, €, ¥, $1,000.00, (€100,00), -USD100.00, …). A custom format, not a keyword.
• dynamicEnum (keyword) — enum whose values come from a CSV (local, HTTP/HTTPS, dathere://, or ckan://). Use this when the valid-value set is too large to hardcode or it changes over time.
• uniqueCombinedWith (keyword) — composite-key uniqueness check across columns.

Example: cheap RFC 4180 + UTF-8 sanity check

qsv validate NYC_311_SR_2010-2020-sample-1M.csv
# Exit code 0 = valid; non-zero = problem; details to stderr.

Example: quarantine ragged rows instead of aborting

Without a schema, validate normally aborts on the first "ragged" row — one with the wrong number of columns. --split-ragged is an opt-in mode that keeps going and separates the file instead:

qsv validate --split-ragged data.csv
# data.csv.valid    — the well-formed rows
# data.csv.invalid  — the ragged rows
# data.csv.validation-errors.tsv — a report, appended

It returns a non-zero exit code when any ragged row is found, so it still fails a pipeline while handing you both halves of the file. Only column-count mismatches are diverted — malformed quoting and non-UTF-8 data still abort.

validate has no parser-config options of its own, but it honors the global QSV_COMMENT_CHAR in both modes. Commented lines (including a commented header) are skipped and are not written to the .valid file:

QSV_COMMENT_CHAR='#' qsv validate --split-ragged data.tsv

Example: validate NYC 311 against the bundled schema

qsv validate \
  NYC_311_SR_2010-2020-sample-1M.csv \
  resources/test/311_Service_Requests_from_2010_to_Present-2022-03-04.csv.schema.json
ls NYC_311_SR_2010-2020-sample-1M.*
# .valid.csv  .invalid.csv  .validation-errors.tsv

Example: dynamicEnum against a CSV of valid NYC agencies

{
  "type": "object",
  "properties": {
    "Agency": {
      "type": "string",
      "dynamicEnum": "NYC_agencies.csv"
    }
  }
}

dynamicEnum accepts a local path or a URL — including ckan:// to fetch a resource by CKAN ID.

Example: uniqueCombinedWith for a composite primary key

{
  "properties": {
    "Created Date":       { "uniqueCombinedWith": ["Complaint Type", "Incident Address"] }
  }
}

This enforces that the combination of those three columns is unique across the file.

Example: currency format validation

{
  "properties": {
    "Sale Price": { "type": "string", "format": "currency" }
  }
}

Example: validate a directory of CSVs against one schema

qsv validate exports/2024/ schema.json

Example: fancy regex (look-around, backreferences) when needed

qsv validate --fancy-regex data.csv strict-schema.json

By default validate uses the same fast regex crate as search. --fancy-regex opts into the slower but more expressive fancy-regex crate for schemas that need it.

See also: /docs/help/validate.md, docs/Validate.md — deep dive, schema, Lookup Tables — for dynamicEnum patterns, Recipe: JSON Schema Validation.

Two modes:

• JSON Schema mode (default) — produces a <input>.schema.json with inferred types, formats, ranges, and enum / pattern constraints. Edit the file by hand to tighten rules (mark required fields, add dynamicEnum lookups, set custom currency formats, …) before using it with validate.
• Polars Schema mode (--polars) — produces a <input>.pschema.json consumed by sqlp / joinp / pivotp to skip type inference. Edit to change inferred Polars data types where needed.

Both reuse the stats cache when available (run qsv stats --cardinality --infer-dates --stats-jsonl <input> first for max speed).

Example: infer a JSON Schema from NYC 311

qsv stats --cardinality --infer-dates --infer-boolean --stats-jsonl \
  NYC_311_SR_2010-2020-sample-1M.csv
qsv schema NYC_311_SR_2010-2020-sample-1M.csv
# Produces NYC_311_SR_2010-2020-sample-1M.csv.schema.json

The generated schema has inferred types, min/max ranges (for numerics), date formats, enum values (for low-cardinality columns), regex patterns (when --pattern-columns is used), and the qsv-specific currency, dynamicEnum, uniqueCombinedWith custom keywords where appropriate.

--enum-threshold is applied consistently (Unreleased, after 25.0.0, #4830). Which columns got enum, const and format constraints used to depend on the other columns in the file. With no column at or under --enum-threshold, every column got an exhaustive enum. With at least one, single-value columns lost their const, and --strict-formats skipped high-cardinality String columns, so an email column with many distinct values got no format: email. Now only columns at or under the threshold get an enum (with --ignore-case, String columns are counted case-insensitively), every single-value column gets const, and --strict-formats checks every String column.

Generated schemas can therefore be stricter. A const locks a column to the one value it held in the input, so validate rejects later data where it changes (a new phone number or URL, say). Generate the schema from a representative dataset, then loosen or remove const/enum constraints where values are expected to change. --strict-formats now keeps every distinct value of each String column in memory while it checks them, which costs more on large files with many distinct text values.

--ignore-case schemas validate their own data (Unreleased, after 25.0.0, #4831). --ignore-case lowercases the values it collects, and validate is case-sensitive, so a String column's enum/const (e.g. ["false","true"]) used to reject the source's True/TRUE. With --ignore-case, schema now writes them as an equivalent case-insensitive regex instead, e.g. "pattern": "^(?i:false|true)$". Values are escaped so that ., |, ( etc. match literally, and the Turkish İ is accepted alongside the i̇ it lowercases to. Integer enums stay enums.

The inline (?i:…) modifier is regex syntax that qsv's validate understands, as do recent JavaScript engines (tested with AJV 8 on Node 26). Other or older JSON Schema validators may not support it. If a schema must work outside qsv, generate it without --ignore-case.

Example: re-validate the source data — schema should always pass for the data it was inferred from

qsv validate \
  NYC_311_SR_2010-2020-sample-1M.csv \
  NYC_311_SR_2010-2020-sample-1M.csv.schema.json
# Exit code 0; no .invalid.csv or .validation-errors.tsv produced

Example: Polars schema for speeding up sqlp

qsv schema --polars NYC_311_SR_2010-2020-sample-1M.csv
# Produces NYC_311_SR_2010-2020-sample-1M.pschema.json

# Now sqlp / joinp / pivotp pick it up automatically
qsv sqlp NYC_311_SR_2010-2020-sample-1M.csv \
  "SELECT Borough, COUNT(*) FROM nyc311 GROUP BY Borough"

Example: validate a manually-fine-tuned schema is itself valid JSON Schema

qsv validate schema fine-tuned-schema.json

See also: /docs/help/schema.md, validate, JSON Schema 2020-12 spec, stats, Stats Cache & Caching.

(Also covered in Selection & Inspection → sniff. Mentioned here because it's the lightweight cousin of schema for validation workflows.)

sniff samples the first N rows (default 1,000) to infer delimiter, header row, preamble rows, quote character, encoding, average record length, total row count, and per-field types & names. Works on local files and remote URLs (without downloading the whole file).

With the magika feature compiled in, it also identifies 200+ file types via Google's AI-based content detector — useful for CKAN harvesting.

Example: probe a remote dataset before downloading

qsv sniff https://opendata.cityofnewyork.us/data.csv
# Reports delimiter, headers, content length, estimated row count, field types

Example: MIME-type-only probe for fast URL health checks (CKAN-style)

qsv sniff --no-infer https://example.com/data.xlsx

Example: stricter sampling for an oddly-shaped CSV

qsv sniff --sample 0.20 --delimiter '|' weird.csv

For guaranteed (non-sampled) type inference, use stats or schema — they scan the entire file.

• Command Reference (index)
• Aggregation & Statistics → stats — guaranteed type inference (vs. sniff's sample)
• SQL & Polars — schema --polars produces the .pschema.json consumed here
• Metadata Profiling (profile) — standards-based metadata generation & validation (DCAT / Croissant / CKAN)
• Lookup Tables — dynamicEnum against local files, HTTP URLs, dathere, CKAN
• docs/Validate.md — canonical reference for custom keywords
• Cookbook → JSON Schema Validation
• Cookbook → Clean & Normalize