Replacing the stage caches with one article catalog → Before and after, measured

Every number was produced on this machine (Python 3.13, local SSD) or read directly off the live corpus on beast. Scripts are inline so they can be re-run.

$ ssh beast-proxy 'sqlite3 "file:$R/metadata/index.sqlite?immutable=1" \
    "select count(*), cast(avg(length(payload_json)) as int) from metadata_entries;"'
205027|23687

raw_metadata is 87–92% of every metadata payload and nothing downstream reads it.

Same workload: 40,000 articles with pmid + doi + pmcid, one download artifact each.

Writes are slower because each artifact now also writes a gzipped blob and an attempt row. That is the trade: ~3.7 µs per article per run, against 718 µs saved on every subsequent read, plus failures that stop being retried forever.

write 2000 download entries          :    0.143s  (0.071 ms/entry)
partition_cached_downloads (2000)    :    0.123s  (0.062 ms/id)
get_cached_article_metadata x200     :    0.150s  (0.750 ms/lookup)
load_download_index x200 (open only) :    0.066s  (0.330 ms/open)

register 40,000 articles              :    2.72s ( 68.1 us/article)
record 40,000 download artifacts      :    6.17s (154.3 us/artifact)
plan all 40,000 in 500-article batches:    0.43s ( 10.7 us/article)
single artifact lookup x5,000         :    0.09s ( 18.2 us/lookup)
resolve identifier x5,000             :    0.18s ( 35.8 us/resolve)

load_cached_analysis_collections read the entire create-analyses index into a dict regardless of how many articles the run was about. The scheduler streams in batches of 500.

RSS while streaming all 40,000 : 77 -> 78 MB (delta 1 MB)

The old figure is linear in corpus size; the new one is linear in batch size.

Real legacy indices, migrated locally. Sources verified byte-identical afterwards with md5sum -c.

The metadata slice is the one that matters, and it splits two ways:

• total on disk falls 3.4× — gzip on payloads the old index stored raw.
• the queryable database falls 48×, from 169.6 MB to 3.5 MB, because raw_metadata moved out of the row. Extrapolated to the full index, the part that gets opened, indexed and scanned goes from 5.2 GB to roughly 110 MB.

Migration throughput: 88,743 download rows + 27,408 upload rows in 46.5 s.

From staging/logs/pipeline.log, one real run — 9,421 of 14,293 extractions were cache hits, and metadata was fetched for all 14,293 anyway:

15:23:32  Semantic Scholar for 14293 articles
15:23:47  ... returned 14119                     (15 s)
15:23:47  PubMed for 14293 articles              <- should have been 174
15:24:48  ... returned 14293                     (61 s)
15:24:48  fallback for 14293 articles            <- should have been 0
15:25:38  enriched 14293                         (50 s)

Cause: _is_complete required all nine metadata fields, including keywords, license and open_access, which no provider returns together. Every article was therefore "incomplete" forever.

After: metadata is its own stage with its own fingerprint, so cache hits are not re-enriched at all, and is_sufficient stops the provider waterfall once title + authors + three of four core fields are present. On that run the work would have dropped from 14,293 articles to 4,872 — and on a rerun, to zero.

Superseded by profiling.md, which measured it. The short version of what that pass found, against the guesses made here:

• The dominant cost is work that fails and repeats — 1.5 h in one log fragment — not any hot loop. Recording failures is worth 134–182 min across four runs.
• The extraction process pool was slower than no pool, 2–50x, because spawn re-imported nilearn and sklearn in every worker. Now 1.4x faster on the production path and up to 52x on small batches.
• ingest --help took 3.3 s; it now takes 0.46 s.
• The catalog write path is within 1.16x of its floor and was never the problem.