Opened 2026-10-09. Frozen once merged. The system as it stands is described in Design. Measurements: PR-58-Measurements.

Task lineage (roadmap L4) traces each task back to the papers that introduced it, through the sentence that describes the task and the references that sentence cites. Nothing kept that link. The text builders kept a marker's characters ("[12]", "Smith et al., 2001") but not the reference it named, and no stage fetched a reference list. This change adds a references stage that stores, per extraction, the reference list and every in-text citation, with offsets for the marker and its sentence.

Stacked PRs: #58 reads what the downloads mark, #59 put the superscripts back into the corpus text, and #60 adds Crossref's list for sources that mark nothing.

Semantic Scholar hands out each citation's sentence (its contexts), which looked like the shortcut. Measured against the exact XML links on 620 papers, it isn't:

• it withholds the reference list for 152 of 260 Elsevier papers ("elided by the publisher"), and for Springer, OUP and SAGE papers too;
• only 56% of the references it does return have a context;
• finding a context in our text is easy (98.6%), but a context found verbatim cites the reference it is listed under only 87% of the time;
• its API licence forbids redistributing what it returns.

The downloads already hold exact links, so those are read first.

XML (JATS from pubget, PMC and Europe PMC; Elsevier) marks each citation with its entry's id. The citation elements are wrapped in private-use sentinels, the extractor's own text builder is re-run on the marked XML, and where the sentinels land are the markers' offsets.

• Elsevier rebuilds byte-identical to the stored text for 96.5% of papers.
• pubget rebuilds identically for half. Its stylesheet pads elements with spaces, and older extractions rendered tables differently. Offsets are carried over by a line-then-character diff.
• So the reader and the extractor never drift, pubget's text assembly is one function, article_text(), called by both.

Publisher HTML (ACE) links its markers to the page's list, under a different attribute for each publisher. The text is readability's and can't be rebuilt from the page, so a link is placed by its own text plus the 40 characters before it. That context is unique within the text for 96% of links.

The sentence comes from pondie's splitter, with one added rule: a marker right after a full stop belongs to the sentence before it. That rule decides 3–5% of citations.

pubget's text_extraction.xsl dropped every <sup> and <sub>. In JATS text, 17.5% of citation markers were missing (27% for PMC), and with them statistical notation like p_FWE. neuroquery/pubget#66 added keep_superscripts and keep_subscripts, and article_text() turns both on. The corpus was rebuilt without re-running any stage; see #59.

• ACE text fuses blocks ("open-plan office.Such episodes"), so two sentences read as one. The citation is right; its sentence is too long.
• About 30% of ACE-only pages hold no reference list. Springer lists without in-text links. ScienceDirect lazy-loads the list. Older MIT pages are abstract only, and some saved "articles" are 404 pages.
• Publisher links are wrong at times. Wiley author-year links can point at the wrong entry, and some citations carry no link at all.
• Citations in tables, captions and footnotes (about 2% of XML links) have no place in the body text, and are counted rather than stored.
• sync does not write any of this yet. The format pondie reads is still to be chosen.