PR 129 Empty Fingerprints - neurostuff/ns-pond-ingestion-workflow GitHub Wiki

#129 Empty fingerprints

Opened 2026-10-17. Pending merge; frozen once merged. The system as it stands is described in Design, Data safety and Migration.

The migration imported artifacts from the pre-catalog caches with no fingerprint, and is_fresh let an empty fingerprint pass whatever the plan expected. They could never go stale: on 2026-10-17 that covered 649,854 downloads, 202,562 metadata, 925 analyses, 914 uploads and 6 extractions. Now an artifact is fresh only when its fingerprint equals the expected one. ingest repair stores the expected fingerprint without redoing the work.

Why filling is not enough

Fingerprints chain. An extraction's fingerprint hashes its download's, and an empty upstream contributes nothing, so an extraction made from a migrated download carries the hash of an empty upstream. In a 20,000-article probe of production, 8,798 pubget and 3,064 ACE extractions had that shape. Filling the downloads alone would make about 595,000 extractions stale, and everything after them. So the repair carries each artifact made from a filled one to the fingerprint it is now expected to have, through triage, analyses, prose, resolve, roles and space. The summary fields roles compares (prose.passages_read, resolve.prose_read, resolve.basis) are rewritten with the fingerprint they copy.

How it decides

The repair learns what each stage expects from its own plan, with no second copy of any fingerprint_for logic. Every stage is planned twice, as ingest run --dry-run plans it:

  1. Against the catalog as it stands, with an empty fingerprint passing (the old rule). This records which artifacts were fresh.
  2. Against a view of the catalog with the rewrites made so far. An artifact a plan asks about is given the fingerprint asked for if it was fresh in pass 1, or had none, and that makes it fresh.

Pass 1 has to use the old rule. An extraction set aside as unusable lets the next source's stand in only while it counts as fresh, and the stand-in is carried only if the first pass saw it.

Prose records, in each passage's inputs, the fingerprint of the metadata it was read with, and takes an earlier answer only while that is still the metadata's. Prose read while the metadata had no fingerprint recorded "", so filling the metadata alone would send every such passage back to the model on the next prose run. The repair relabels the recorded input in each prose payload of an article whose metadata it fills. That writes a new blob: 2,973 prose payloads on production.

An artifact no plan asks about cannot be carried. A plan blocks when the model it needs is not served (roles, whose fingerprint hashes the served checkpoint), or when it is missing an input (passages without a text hash). The dry run counts these as not asked, stale, and --apply refuses while that count, empty, runs again or fresh, runs again is above zero, naming what the plans blocked on.

What stays empty

  • 54,644 downloads are second sources. The download plan stops at the first fresh source, so they are never asked about. Nothing re-runs for them while the first source stays fresh. Once it goes stale (a version bump or --refresh download), the plan falls through, and with no fingerprint the second source is not reused: up to 54,644 downloads again, ACE ones over the network, where the old rule would have kept the stand-in.
  • 914 uploads are legacy rows the ingester never sent. Upload re-sends them on main as well, so nothing changes for them.

Measured

On a 3,111-article sample copied from production, the ingest run --dry-run plan after repair --apply is identical, line for line, to the plan under the old rule. On production (read-only snapshot, 20 to 29 min, 5.4 GB): 595,210 downloads, 202,562 metadata, 925 analyses and 6 extractions are filled, about 1.13 million artifacts are carried, and 2,973 prose payloads are relabelled. 573,563 passages cannot be asked about because 602,001 extractions record no text hash. Roles' 3,861 are asked about only while the role models are served. The write is one transaction: 2.2 million rows took 7.5 s under the write lock on a copy of production, below the 30 s busy timeout of a concurrent run. Blobs are written before the lock is taken.

Operator order

  1. Wait for the running chain to finish.
  2. Record text hashes with ingest refresh-text.
  3. Serve the role models, and use the config the runs use.
  4. Run ingest repair -c <config>. not asked, stale should be about 0, and --apply refuses otherwise.
  5. Run ingest repair --apply --backup <new path> -c <config>.
⚠️ **GitHub.com Fallback** ⚠️