#119 Search provenance

Opened 2026-10-16. Frozen once merged. The system as it stands is described in Design and Data safety.

ingest add --search printed which provider found what, then kept nothing. After a 2022:2026 search (122,966 articles, 56,926 found only by ScienceDirect's full-text match) the per-provider sets were gone, so the ScienceDirect-only articles could not be held back from run. This change keeps each run and adds a selection by provider.

Where it is kept

Two catalog tables: searches, one row per run (query as given, years, time, and per provider the rendered query with its record, article and no-ID counts), and search_hits(search_id, provider, article_id).

• A JSONL file under data_root holds ids that enrichment later merges away, and "found only by X" becomes a script that resolves and intersects them. Every other selection already reads the catalog.
• A blob or artifact in the catalog: artifacts are keyed by (article, stage, source), and a run is not one article's. A blob is opaque to SQL, so a selection loads every id and intersects in Python.
• Tables answer "found by A and no other" from one index, and sit beside the aliases that resolve those ids.

Merges are read through, not applied

A hit stores the article id it resolved to when the run was recorded. Reads follow articles.merged_into from there, so a retired article's hit counts for its survivor and a provider that found both halves counts once. _merge does not touch search_hits.

The first version moved the rows inside _merge. The production chain ran code that did not know the table: its merges left hits on retired articles, identifiers of a retired id is empty (its aliases moved), and searches manifest --only-found-by wrote a row of nulls and counted the article twice. Reading through the merge is correct however the merge was made, and costs one pass over articles.merged_into.

No schema version

The tables are CREATE TABLE IF NOT EXISTS in the DDL run on every open, so an existing catalog gains them empty. A version bump would copy the whole catalog to catalog.pre-v3.sqlite for a change that rewrites nothing, and code at the earlier version could no longer open it. Searches run before this are not recoverable.

Selecting

ingest searches list, and ingest searches manifest ID with --found-by P (repeatable; any of them) or --only-found-by P (P and no other provider of that run). It writes a JSONL identifiers manifest for run --manifest. The complement of "only ScienceDirect" is --found-by the other providers; there is no --not-found-by.