Recipe DCAT US data gov - dathere/qsv GitHub Wiki

Recipe: A DCAT-US v3 Catalog Entry for data.gov

Tier: Intermediate Commands used: profile, jq Anchor dataset: resources/test/boston311-100.csv, a 100-row sample of Boston 311 service requests Requires: qsv 24.0.0+ (canonical DCAT-US v3 output and stricter validation are new in 24.0.0)

Problem

You publish a CSV and need a DCAT-US v3 catalog entry for it: the metadata data.gov harvests into its public data inventory. The entry is only useful if it validates. A record the harvester rejects leaves the dataset unfindable, however open the data is.

Writing that JSON by hand is slow and easy to get wrong. qsv profile builds it from the data itself (columns, types, row count, checksum, bounding box) and validates it against the GSA schema that data.gov uses. What it can't infer, such as a description and a contact, you supply once in a small seed file.

On 23.0.1 and earlier, profile emitted the older prefixed JSON-LD form (dct:title, …), which data.gov's validator rejects. See Metadata & Profiling → Choosing a profile.

Data

# bundled with qsv
cp resources/test/boston311-100.csv boston311.csv

Outside the qsv repo:

curl -L -o boston311.csv https://raw.githubusercontent.com/dathere/qsv/master/resources/test/boston311-100.csv

Solution

1. Profile the bare CSV and see what's missing

dcat-us-v3 is the default profile. --catalog wraps the Dataset in the Catalog envelope harvesters expect, and --validate checks the result against the vendored GSA JSON Schema:

qsv profile boston311.csv --validate --catalog -o boston311.metadata.json
jq '.projection_warnings' boston311.metadata.json

[
  { "field": "contactPoint", "severity": "required", "message": "missing contactPoint" },
  { "field": "dataset/0", "severity": "required", "message": "\"description\" is a required property" },
  { "field": "dataset/0", "severity": "required", "message": "\"contactPoint\" is a required property" },
  { "field": "dataset/0", "severity": "required", "message": "\"identifier\" is a required property" }
]

A CSV can't tell profile what the dataset is about, who to contact, or what its identifier is. Without --strict, those gaps are listed in projection_warnings and the file is still written.

2. Make the gaps fatal with `--strict`

For a data.gov submission, you want missing Required fields to stop the run:

qsv profile boston311.csv --validate --strict --catalog -o boston311.metadata.json
# qsv profile --strict: 3 schema violation(s):
#   - dataset/0: "description" is a required property
#   - dataset/0: "contactPoint" is a required property
#   - dataset/0: "identifier" is a required property
echo $?
# 1

No output file is written.

3. Seed what the data can't say

Write the missing facts once, in CKAN package terms. profile maps them to DCAT-US fields (notes → description, name → identifier, contact_point → contactPoint):

{
  "package": {
    "title": "Boston 311 Service Requests (sample)",
    "name": "boston-311-sample",
    "notes": "A 100-row sample of City of Boston 311 service requests.",
    "publisher": "City of Boston",
    "contact_point": {"fn": "Open Data Team", "hasEmail": "[email protected]"},
    "license_id": "cc-by"
  }
}

Save it as seed.json and run again:

qsv profile boston311.csv --validate --strict --catalog \
  --initial-context seed.json -o boston311.metadata.json
# qsv profile: wrote `boston311.metadata.json`
jq 'has("projection_warnings")' boston311.metadata.json
# false

It exits 0 and reports no findings at all.

4. Read the projection

The projection block is the catalog entry. As of 24.0.0 it uses canonical, unprefixed DCAT-US v3 keys with no @context:

jq '.projection.dataset[0] | del(.distribution, .spatial)' boston311.metadata.json

{
  "@type": "Dataset",
  "title": "Boston 311 Service Requests (sample)",
  "description": "A 100-row sample of City of Boston 311 service requests.",
  "identifier": "boston-311-sample",
  "publisher": { "@type": "Organization", "name": "City of Boston" },
  "contactPoint": { "@type": "Kind", "fn": "Open Data Team", "hasEmail": "mailto:[email protected]" },
  "dcat-us:accessLevel": "public",
  "conformsTo": [
    { "@type": "Standard", "title": "DCAT-US 3.0", "identifier": "https://resources.data.gov/dcat-us/3.0.0" }
  ]
}

Note what profile filled in on its own:

hasEmail gained its mailto: prefix.
license_id: cc-by became the license URL on the Distribution (http://creativecommons.org/licenses/by/4.0/).
spatial is a bounding box computed from the latitude/longitude columns.
The Distribution carries mediaType, byteSize, a SHA-256 checksum and a column-level table schema for all 29 columns.

dcat-us:accessLevel keeps its prefix on purpose: it is a Project Open Data field, not a v3 property, and agencies still need it.

5. Extract the catalog and check it at data.gov

jq '.projection' boston311.metadata.json > data.json

data.json is a Catalog conforming to https://resources.data.gov/dcat-us/3.0.0. qsv's validator runs the same GSA schema bundle that data.gov's own validator at https://harvest.data.gov/validate/ uses, and it is tested against GSA's 23 good/bad conformance fixtures. Pasting data.json there is a good last check before you register a harvest source.

Variations

What a broken record looks like

Overrides under dataset_info are written to the projection verbatim, which makes them a quick way to see how --validate reports problems. This seed adds a non-ISO date and a misspelled key:

{
  "package": { "...": "as above" },
  "dataset_info": {
    "/projection/modified": "2024-12-15 08:30",
    "/projection/keywrd": ["311"]
  }
}

qsv profile boston311.csv --validate --initial-context broken.json -o broken.metadata.json
jq '.projection_warnings' broken.metadata.json

[
  {
    "field": "modified",
    "severity": "recommended",
    "message": "\"2024-12-15 08:30\" is not valid under any of the schemas listed in the 'anyOf' keyword"
  },
  {
    "field": "keywrd",
    "severity": "recommended",
    "message": "`keywrd` is not a DCAT-US v3 dataset property and is not a known qsv extension. It will be ignored by data.gov harvesters. Check the spelling, or whether the property was removed upstream."
  }
]

Both findings are new in 24.0.0. Before that, format was only an annotation, so a malformed date passed locally and failed at data.gov, and unknown keys passed silently.

Both are Recommended, so --strict still exits 0 on this file. --strict aborts only on Required findings, with severity taken from the GSA schema's own requirementLevel annotations. Read projection_warnings even when --strict passes.

Dates from CKAN are normalized

Dates that come in through package fields are normalized to UTC. An offset-less "metadata_modified": "2024-12-15T08:30:00" is emitted as "modified": "2024-12-15T08:30:00Z". A date profile can't parse (for example 15/12/2024) is left out of the projection rather than emitted invalid, so check that modified is present if you rely on it.

More fields

dcat-init-context.json fills every slot profile reads, including bureauCode, programCode, accrualPeriodicity and the access/use restrictions. Copy the parts you need. Wrap a value as {"value": ..., "force": true} to beat both inference and any DCAT markup discovered from a URL input. See Metadata & Profiling → Seeding & overriding values.

Profile straight from a URL

qsv profile https://…/data.csv works too, and if the response advertises DCAT markup (Link: rel=describedBy), the publisher's own metadata is merged in as a base layer. See Metadata & Profiling → URL inputs & DCAT discovery.

Inside CKAN

DataPusher+ reads the ckan block of the same output; the projection is unchanged by it. See Recipe: CKAN Integration.

Caveats

publisher is Recommended, not Required, in the GSA schema bundle qsv vendors, so a record without it passes --strict. data.gov still expects to know who published the data, so set it.
--strict only aborts on Required findings. Recommended findings, like the bad date and misspelled key above, appear only in projection_warnings.
Anything reading projection.dct:* keys from 23.0.1 output must switch to the bare names. The ckan block did not change.