PR 129 Empty Fingerprints - neurostuff/ns-pond-ingestion-workflow GitHub Wiki
Opened 2026-10-17. Pending merge; frozen once merged. The system as it stands is described in Design, Data safety and Migration.
The migration imported artifacts from the pre-catalog caches with no fingerprint, and
is_fresh let an empty fingerprint pass whatever the plan expected. They could never go
stale: on 2026-10-17 that covered 649,854 downloads, 202,562 metadata, 925 analyses, 914
uploads and 6 extractions. Now an artifact is fresh only when its fingerprint equals the
expected one. ingest repair stores the expected fingerprint without redoing the work.
Fingerprints chain. An extraction's fingerprint hashes its download's, and an empty
upstream contributes nothing, so an extraction made from a migrated download carries the
hash of an empty upstream. In a 20,000-article probe of production, 8,798 pubget and 3,064
ACE extractions had that shape. Filling the downloads alone would make about 595,000
extractions stale, and everything after them. So the repair carries each artifact made
from a filled one to the fingerprint it is now expected to have, through triage, analyses,
prose, resolve, roles and space. The summary fields roles compares (prose.passages_read,
resolve.prose_read, resolve.basis) are rewritten with the fingerprint they copy.
The repair learns what each stage expects from its own plan, with no second copy of any
fingerprint_for logic. Every stage is planned twice, as ingest run --dry-run plans it:
- Against the catalog as it stands, with an empty fingerprint passing (the old rule). This records which artifacts were fresh.
- Against a view of the catalog with the rewrites made so far. An artifact a plan asks about is given the fingerprint asked for if it was fresh in pass 1, or had none, and that makes it fresh.
Pass 1 has to use the old rule. An extraction set aside as unusable lets the next source's stand in only while it counts as fresh, and the stand-in is carried only if the first pass saw it.
Prose records, in each passage's inputs, the fingerprint of the metadata it was read with, and takes an earlier answer only while that is still the metadata's. Prose read while the metadata had no fingerprint recorded "", so filling the metadata alone would send every such passage back to the model on the next prose run. The repair relabels the recorded input in each prose payload of an article whose metadata it fills. That writes a new blob: 2,973 prose payloads on production.
An artifact no plan asks about cannot be carried. A plan blocks when the model it needs
is not served (roles, whose fingerprint hashes the served checkpoint), or when it is
missing an input (passages without a text hash). The dry run counts these as
not asked, stale, and --apply refuses while that count, empty, runs again or
fresh, runs again is above zero, naming what the plans blocked on.
-
54,644 downloads are second sources. The download plan stops at the first fresh
source, so they are never asked about. Nothing re-runs for them while the first
source stays fresh. Once it goes stale (a version bump or
--refresh download), the plan falls through, and with no fingerprint the second source is not reused: up to 54,644 downloads again, ACE ones over the network, where the old rule would have kept the stand-in. - 914 uploads are legacy rows the ingester never sent. Upload re-sends them on main as well, so nothing changes for them.
On a 3,111-article sample copied from production, the ingest run --dry-run plan after
repair --apply is identical, line for line, to the plan under the old rule. On
production (read-only snapshot, 20 to 29 min, 5.4 GB): 595,210 downloads, 202,562
metadata, 925 analyses and 6 extractions are filled, about 1.13 million artifacts are
carried, and 2,973 prose payloads are relabelled. 573,563 passages cannot be asked about
because 602,001 extractions record no text hash. Roles' 3,861 are asked about only
while the role models are served. The write is one transaction: 2.2 million rows took
7.5 s under the write lock on a copy of production, below the 30 s busy timeout of a
concurrent run. Blobs are written before the lock is taken.
- Wait for the running chain to finish.
- Record text hashes with
ingest refresh-text. - Serve the role models, and use the config the runs use.
- Run
ingest repair -c <config>.not asked, staleshould be about 0, and--applyrefuses otherwise. - Run
ingest repair --apply --backup <new path> -c <config>.