#29 Keep every table

Merged 2026-09-29. Frozen. The system as it stands is described in Design.

Before this change the extract stage was also a filter. Four separate places threw away part of what the document said, and each one was found by measuring what actually reached the serialiser rather than by reading the code.

The full measurements and the reasoning behind each fix are in the pull request. This page records what it did to the corpus, and the one thing that went wrong afterwards.

What was being thrown away

where	what was lost	how it was found
ace	every table ACE did not itself recognise as an activation table	100% of 500 sampled ace tables carried coordinates — a filtered set has no negative class
pdf	the header flags and spans Docling detected	# header marker present on 3.6% of pdf tables against 92.3% after
ace, elsevier, pdf	the caption and the footnote	elsevier's footnote is in ce:legend, populated 59.9%, and the extractor read only ce:table-foot, populated 0%
pdf	the footnote Docling did not link	reading the linked field alone recovered 3.9%; position recovers the rest

15.7% of real tables state their coordinate space only in that surrounding text, so a dropped caption is a dropped space.

What it did to the corpus

Re-extraction of all 29,525 ace articles:

	before	after
tables	35,404	108,381
rescued from the HTML	0	72,977
carrying coordinates	—	33,563 (31%)

Across ace, elsevier and pdf, 131,667 articles were re-extracted. 93.7% of the serialised text of the curated training tables changed, and 1,591 of them gained a footnote they had never had.

The stage stopped being a filter

Keeping every table is only half of it. extract no longer decides which tables are worth anything — it extracts, and a separate stage judges. That judgement is #30 Triage, which exists because this change left 1.33M tables where there had been 385K, and something has to say which of them earn a model call.

An extract artifact's summary still carries tables_with_coordinates, because the API reads it, but nothing in the pipeline routes on it any more.

What went wrong afterwards

The re-extraction ran with the default catalog_root, which is a relative ./.catalog. cache_root was absolute and shared, so the payloads landed in the shared cache and the results landed in a private catalog nobody reads. Every stage reported success. Nothing said otherwise.

It was found three days later, when curated training data rebuilt from the catalog still carried the old captions. The recovery merged the cache payloads into the catalog under the fingerprints the stage would have written — the same eight keys, blob = sha256 of compact sorted JSON, fingerprint = blake2b over ("extract", source, EXTRACTOR_VERSIONS[source]) with the download artifact's fingerprint as upstream — and was verified against 1,638 artifacts the pipeline itself had produced: fingerprint, blob and summary matched on all of them. 129,751 rows merged; no other stage was touched.

Settings.ensure_directories now logs both roots and warns when an absolute cache_root sits beside a relative catalog_root. See Data safety.