Tier: Intermediate Commands used: readstat, validate, search, writestat Anchor dataset: resources/test/readstat_sentinels.sav — a 4-row SPSS file with value labels and user-defined missing values Requires: qsv 24.0.0+ (writestat and readstat --dictionary are new in 24.0.0)

A statistician hands you an SPSS .sav file. You need to fix or filter it with CSV tools, then hand it back as .sav, and they expect to find it the way they left it:

• rating still stores codes 3/4, with the labels "Good"/"Great" attached
• "Refused" (code 8) and the unlabeled code 9 are still declared missing values, not just blank cells
• the variable formats and missing-value ranges are still there

Plain CSV keeps none of that. On the way out you lose the labels and the declared missing values. On the way back in you get a file with no metadata. In survey data, an empty cell can mean one of several things, and those differences matter for the analysis.

The fix is to carry the metadata beside the CSV in a JSON Schema data dictionary. readstat --dictionary writes the dictionary, and writestat --dictionary reads it to rebuild the file.

# bundled with qsv
cp resources/test/readstat_sentinels.sav survey.sav

Outside the qsv repo:

curl -L -o survey.sav https://raw.githubusercontent.com/dathere/qsv/master/resources/test/readstat_sentinels.sav

See what a plain conversion drops:

qsv readstat survey.sav -o /dev/null
# 4 rows exported.
# 5 sentinels (user-defined missing values) in 3 variables (income, rating, name) were
# written as empty cells, e.g. 99 in income. Use --sentinels-as value to keep them,
# or --sentinels-as none to skip this check.

Five cells hold a reason for being missing, and the plain CSV writes all five as empty.

qsv readstat survey.sav -o survey.csv \
  --value-labels --sentinels-as label --compress-numeric \
  --dictionary survey.schema.json

id,income,income_null,rating,rating_null,name,name_null,plain
1,1500.5,,Good,,Ana,,1
2,,99,,Refused,,NA,2
3,,950,Great,,Cy,,3
4,,,,9,,,4

What each flag does:

The dictionary records everything writestat needs to undo those choices. Here is an excerpt for rating:

"rating": {
  "type": ["string", "null"],
  "enum": ["Good", "Great", null],
  "x-qsv": {
    "value_labels": [
      {"value": 3, "label": "Good"},
      {"value": 4, "label": "Great"},
      {"value": 8, "label": "Refused"}
    ],
    "missing_values": {"discrete": [8, 9]},
    "source_format": "F8.2"
  }
}

The top-level x-qsv.readstat_flags block also records the flags you used. That is how writestat knows that rating holds decoded labels and rating_null holds sentinel labels.

The dictionary is a regular JSON Schema, so validate checks the CSV against it:

qsv validate survey.csv survey.schema.json
# All 4 records valid.

Now edit the CSV. This example drops respondent 4:

qsv search -s id -v '^4$' survey.csv > survey_clean.csv
qsv validate survey_clean.csv survey.schema.json
# All 3 records valid.

Re-validate after every edit before you write the file back. If an edit adds a value the dictionary doesn't allow (for example a new rating label), validate reports it here instead of writestat failing later.

Keep the column names. writestat matches CSV columns to the dictionary by name, including the <name>_null sentinel columns. If you rename or drop a column, update the dictionary to match.

qsv writestat --dictionary survey.schema.json survey_clean.csv -o survey_clean.sav
# 3 rows written to "survey_clean.sav".

Read the new file back the same way. The CSV is identical except for the row you dropped:

qsv readstat --value-labels --sentinels-as label --compress-numeric survey_clean.sav

id,income,income_null,rating,rating_null,name,name_null,plain
1,1500.5,,Good,,Ana,,1
2,,99,,Refused,,NA,2
3,,950,Great,,Cy,,3

Underneath, the codes are back. The labels "Good"/"Great" are stored as 3/4 again, and "Refused" is the declared missing value 8 again:

qsv readstat --sentinels-as value survey_clean.sav
# id,income,income_null,rating,rating_null,name,name_null,plain
# 1.0,1500.5,,3.0,,Ana,,1.0
# 2.0,,99,,8,,NA,2.0
# 3.0,,950,4.0,,Cy,,3.0

The variable metadata matches the original as well:

for f in survey.sav survey_clean.sav; do
  qsv readstat --metadata json "$f" \
    | jq -cS '.variables[] | {name, value_labels, missing_doubles, missing_strings}' > "$f.meta"
done
diff survey.sav.meta survey_clean.sav.meta && echo "metadata identical"
# metadata identical

jq -S sorts the keys, because the rewritten file may list a variable's value labels in a different order.

Only SPSS .sav files can declare missing values. Writing the same CSV and dictionary to Stata is refused, and the error lists everything that would be lost:

qsv writestat --dictionary survey.schema.json survey_clean.csv -o survey_clean.dta
# usage error: A Stata .dta file can't hold all of this file's metadata:
#   - the sentinels of "income" (column "income_null"): only SPSS .sav files declare missing values
#   ...
#   - the missing values of "name": only SPSS .sav files declare them
# Pass --lossy to write it anyway, without them.

--lossy writes the file anyway and prints a warning for each item it leaves out. Whatever the format can hold still comes through. Here the Stata file keeps rating's value labels:

qsv writestat --lossy --dictionary survey.schema.json survey_clean.csv -o survey_clean.dta
qsv readstat --value-labels survey_clean.dta
# id,income,rating,name,plain
# 1,1500.5,Good,Ana,1
# 2,,,,2
# 3,,Great,Cy,3

SAS transport (.xpt) also refuses value labels. To keep SAS labels, keep the .sas7bcat catalog and the dictionary alongside the data.

A name that isn't a valid SPSS variable name is an error even with --lossy, because qsv rename can fix it first:

qsv writestat data.csv -o data.sav
# usage error: These column names aren't valid SPSS variable names: "label text". ...
# Rename them first (e.g. with 'qsv rename').

qsv writestat data.csv -o data.xpt infers the column types from the CSV, which can also come from stdin. Use this for CSVs that never came from a statistics package.

readstat accepts --select, --offset, --limit, and --sample/--seed, and the dictionary describes only the part that was written. See Conversion & IO → readstat.

Name the dictionary <stem>.schema.json after the CSV and viz smart finds it on its own. It then uses the variable labels as chart titles and the measure/dimension roles to pick charts. See Visualization.

• Tracking user-defined missing values reads the file on a single thread, so --jobs has no effect with --sentinels-as.
• SPSS .sav/.zsav files that declare missing values are read whole while their missing values are tracked, so memory scales with file size. SPSS portable .por files are always read whole.
• --compress-numeric reads the file twice: the first pass checks only the float columns.
• --dictionary reads text-coded columns a second time when it needs to find labels that are also ordinary values.

• writestat is marked experimental. Open the files it writes in the package that will read them before you rely on them.
• Some losses come from the format itself: Stata integers are written in the narrowest type that fits, and SPSS portable files get default display formats.
• .zsav and .sas7bdat can't be written. Use --compress for a compressed .sav, or .xpt for SAS.

• Conversion & IO → readstat and writestat
• Conversion & IO → A data dictionary from the file's own metadata
• Validation & Schema
• Recipe: JSON Schema Validation
• docs/help/readstat.md and docs/help/writestat.md: canonical flag reference