Paper extraction

Turning a research paper into a structured, source-grounded record. The record is what a later step turns into an sbt2 strategy, so a published strategy can be reproduced on its own period and then judged on data after publication. It is the sbt2-papers distribution of the workspace, imported as sbt2.papers, and installed with pip install "sbt2[papers]" (see Library). It and the backtester never import each other, by import-linter contract: the only link between them is the record's format.

Pipeline

paper id / PDF
   │
1. acquire ──► PDF + metadata (title, authors, first-version date, DOI)
2. parse   ──► Document: sections, paragraphs with character offsets, tables as grids, equations as LaTeX
3. triage  ──► keep / reject + tags (asset class, frequency, data needed)
4. extract ──► draft record: every field with evidence spans and a status
5. verify  ──► every field checked against its spans; gaps and conflicts listed
6. review  ──► a person fills the gaps; each one becomes a recorded assumption
   │
frozen, versioned record (JSON) ──► codegen (separate step) ──► sbt2

1. Acquire. The PDF and metadata from arXiv, SSRN, a DOI or a local file. The first public version's date is recorded: everything the paper reports is in-sample for it, so the sbt2 test part starts after that date. Published anomalies are known to weaken after publication (McLean & Pontiff 2016).
2. Parse. Behind a Parser interface: MinerU or Docling for the body (formulas as LaTeX, tables as cell grids), GROBID for the paper's structure and references. The Document keeps character offsets, pages and bounding boxes, so every later claim points back to its source.
3. Triage. A cheap model answers fixed questions: is there a tradable rule, which asset class and rebalancing frequency, is the data in the catalog. Rejected papers stop here, before the expensive stages.
4. Extract. Schema-driven and grounded, in the style of LangExtract: one pass per section, then a merge. Every field has a value, its evidence (the source spans) and a status: stated, inferred or missing.
5. Verify. An independent check of every field against its spans, with a different model or prompt from the extraction; numbers must parse and reported results must agree with each other. It produces a completeness score and the list of gaps.
6. Review. Required before a record is frozen. A person resolves every gap and every inferred field; each resolution is stored as an assumption with its reason. An assumption is a design choice, so it counts as a trial when the strategy's results are deflated in sbt2 (see Results).

Record

A generic core, so papers outside trading fit too, and domain schemas on top. The schemas are pydantic models, which give both the LLM's structured output and validation.

• Core: Paper, Document, Span, Evidence, Field[T] (value, evidence, status), Assumption.
• TradingStrategy, the first domain schema:

Group	Fields
Universe and data	instruments, filters, data frequency, point-in-time requirements
Signal	formula (LaTeX), description, lookbacks
Portfolio	construction, rebalancing, execution timing (e.g. next open)
Costs	what the paper assumed
Sample	the paper's in-sample and out-of-sample periods, first-version date
Parameters	the values used, and any ranges the authors searched (they count as trials)
Reported results	metric × period × variant, the numbers a reproduction is compared with

Storage

The same approach as sbt2's results store: one folder per paper, one immutable artifact per stage. Each artifact is keyed by the hash of its inputs plus the parser, model and prompt versions, so rerunning a stage whose inputs haven't changed is a cache hit, a prompt change reruns only what depends on it, and every record names what produced it.

Modules

sbt2/papers/
  acquire/    Source ABC: arxiv, ssrn, doi, local
  parse/      Parser ABC: mineru, docling, grobid; the Document model
  llm/        Model ABC over providers with structured output; the only code that calls an API
  schema/     core (Span, Evidence, Field, Assumption) and domains/trading.py
  extract/    triage, extract, merge
  verify/     grounding, consistency, completeness
  store/      per-paper artifacts and cache keys
  api.py      extract(paper_id, schema) -> Record, the one public entry point
  cli.py

Parsers, sources and models are abstract base classes with an explicit-registry factory, like sbt2's variants. The LLM provider is pluggable behind llm; one provider is implemented first.

Tests

• Gold set: 10–20 strategy papers annotated by hand, e.g. from Papers With Backtest, whose code already exists.
• Measured: field accuracy, grounding rate (the share of stated fields whose spans really support them) and gap detection.
• Golden tests as in sbt2: fixed PDFs and cached model responses must keep producing the same records, so a prompt or parser change shows up as a diff.

Why structural guards

Gençay 2026 found that every LLM-discovered strategy failed a leakage-safe, search-aware evaluation, and that deflation can't catch look-ahead: a planted oracle with a Sharpe of 35 passed the deflated Sharpe ratio and PBO. A published rule implemented as written is close to a pre-registered hypothesis and faces little deflation; every change made to match the paper's numbers is another trial. Hence the evidence on every field, the recorded assumptions, and test dates after publication. Look-ahead in the generated code needs a structural guard in the codegen step, not a statistical one.

Related work

PaperCoder / Paper2Code (planning → analysis → generation) is the model for the codegen step. PaperQA2 and OpenScholar search and answer across many papers, for finding candidates before stage 1. RD-Agent runs report → factor → Qlib backtest, for factors only. ORKG models a paper as problem, method and results. See References.