Opened 2026-10-09. Frozen once merged. The system as it stands is described in Design and Data safety.

pubget began keeping superscripts and subscripts (#58, neuroquery/pubget#66). Shipped as an extractor version bump, that would have re-run extraction, triage, the analyses model and upload for every JATS article. None of them reads the text: tables come from their own files. ingest refresh-text instead rebuilds each extraction's text from its download and rewrites the file in place. The extraction keeps its payload and fingerprint, so nothing goes stale.

sync copies an extraction's text into the corpus as processed/<source>/text.txt, but nothing records which copy came from which text. The refresh therefore replaces a corpus file only when its bytes equal an extraction text it rewrote, and leaves an old text that rebuilt two different ways alone. A corpus file that is not an exact copy is never touched.

	count
OK JATS extractions	477,800
rewritten	381,005
already identical	96,713
no text file	82
corpus text.txt replaced	19,627
corpus copies matching no current extraction, left alone	1,271

The run took 12 minutes on 32 workers. The code ran from /data/alejandro/iw-text-refresh, with upstream pubget installed beside it (--target), so the shared venv was not changed. Run directory: staging/text-refresh-20261009/.

What was checked before writing. A dry run first. Then 300 sampled rewrites were compared against today's builder with sup/sub off: all were byte-identical. So the refresh added the sup/sub text and nothing else, and every old text can be regenerated exactly. The extraction texts were therefore not backed up. The 19,627 corpus files were: corpus-texts-before-refresh.tar.zst (258 MB, verified).

What was checked after.

• A second dry run found nothing left to change.
• Every replaced corpus file equals its rebuilt extraction text.
• In the stored text of 300 corpus papers, citation markers went from 82.7% to 97.4%, and to 100% in the replaced records.

• Path.read_text turns \r into \n. Two texts that keep a carriage return never compared equal to their own rebuild. The command now reads and writes bytes.
• 1,271 corpus pubget texts match no current extraction. These are copies from an older extraction that a later sync did not replace. A later sync would replace them.