Changes - neurostuff/ns-pond-ingestion-workflow GitHub Wiki

Changes

One record per change that is worth explaining: what it replaced, what it measured, and the questions that shaped it. These are frozen once merged โ€” the pages describing the system as it stands are Design, Data safety and Migration, and each change edits those rather than adding a second copy of them.

change what it did pages
#11 Catalog refactor replaced the six stage caches with one article catalog keyed on stable identity before ยท benchmarks ยท profiling
#29 Keep every table stopped the extract stage filtering, and kept the caption and footnote with every table โ€”
#51 Coordinate space from the text added a space stage that reads an unknown table's space from its caption, Methods and Results โ€”
#54 Coordinates in prose added prose and resolve stages that read coordinates written in the text and merge them with the tables' โ€”
#58 References and citations added a references stage: each extraction's reference list and in-text citations, placed in its text; pubget keeps superscripts measurements
#59 Text refresh added ingest refresh-text, and rebuilt 381,005 JATS texts with their superscripts without re-running a stage โ€”
#60 Crossref reference lists added a reflist stage: Crossref's list, named by OpenAlex, for sources that mark nothing and entries without DOIs โ€”
#119 Search provenance kept each add --search run in the catalog and added ingest searches to select its articles by provider โ€”

Adding one

Whether a change earns a record is decided by AGENTS.md โ€” roughly, architecture and measurements earn one; bug fixes and new flags do not. This page covers the layout.

GitHub wikis are flat: a file committed at Foo/Bar.md is served at /wiki/Bar, so directories buy nothing and page names are a single global namespace. Hence the PR-<n>- prefix โ€” without it, the second change wanting a "Benchmarks" page would have nowhere to put it.

For a change that earns a record:

  1. Add PR-<n>-<Short-Name>.md as the record, plus any PR-<n>-<Topic>.md pages it needs. Open each child with a breadcrumb back to the record.
  2. Add a row to the table above.
  3. Edit the living pages so they describe the system after the change. Records are frozen once merged; living pages are not.
  4. Add the group to _Sidebar.md, which renders on every page.
โš ๏ธ **GitHub.com Fallback** โš ๏ธ