PR 51 Coordinate Space - neurostuff/ns-pond-ingestion-workflow GitHub Wiki

#51 Coordinate space from the text

Opened 2026-10-06. Frozen once merged. The system as it stands is described in Design.

A table whose coordinate space nobody could name was stored as OTHER and uploaded that way. This change adds a space stage between analyses and upload that reads the space from the article's own words. The rules and the per-rule accuracy are in the pull request. This page records why it is a stage, what it did to the corpus, and what it found that it does not fix.

Why a stage, and not a step inside analyses

analyses is the model call. Anything that changes its output has to change its fingerprint, and that re-runs the model for the whole corpus. Reading text costs minutes. As its own stage, space can be revised and re-run without touching a single extraction.

Its payload has the same shape as analyses, so upload and sync read it as they read analyses. An article with nothing to fill writes byte-identical JSON, and the blob store, being content-addressed, keeps it once.

What it reads, and what it skips

Reading order is the table's caption and footer, then the Methods, then the Results. Each place is ranked by how directly it states the space the coordinates are reported in. The rest of the article is skipped on purpose:

section why it is not read
introduction, discussion they describe other studies' coordinates
front matter "Montreal Neurological Institute, McGill" is an affiliation
references Talairach & Tournoux 1988 is cited whatever space a paper used

The sectioniser is adapted from pondie's. pubget and Elsevier text marks headings in markdown. ACE and PDF text has only bare lines ("MATERIALS AND METHODS"), so those are matched as whole lines against the top-level names. Methods or Results is found for 98.6% of analysed tables.

What it did to the corpus

Measured read-only over all 39,227 articles with analysed tables:

tables articles
stored OTHER 4,022 2,667
filled 2,025 (1,522 MNI, 503 TAL) 1,106
left OTHER 1,997

Most of what is left is not stereotactic. Of 249 sampled articles whose text never names MNI or Talairach, 136 mention no imaging software either: vehicle locations, sampling sites, haplotype counts.

Found, not fixed

8.3% of tables whose caption names a space are stored in the other one. That is 458 of 5,490 sampled, and 326 of them are Elsevier.

Every extractor stamps neurosynth's article-level _neurosynth_guess_space on its tables, and create_analyses lets a value the extraction "read" beat the model. The guess reads SPM as MNI, but SPM papers that convert and report Talairach are common. So a caption saying "Talairach coordinates" loses to a software name.

space only fills OTHER and leaves this alone. Correcting it would rewrite spaces that are already uploaded, which is a decision of its own.