Backtesting - kingabzpro/BetterTradingAgents GitHub Wiki

The walk-forward harness answers "is the pipeline better than buy-and-hold?" It replays the full analysis at each grid date using only data known at that date, grades every call against SPY after costs, and compares with baselines. Mock mode is the free default.

uv run python -m app.backtest --tickers NVDA,AMD,META --start 2026-03-01 --end 2026-06-30 --step 21

How a replay flows

  1. Pick the grid dates.
  2. Snapshot point-in-time data for each date (nothing from the future).
  3. Run the pipeline (mock agents by default, real LLMs with the cost warning).
  4. Grade every call against SPY after costs.
  5. Aggregate into a report with baselines and, optionally, holdout honesty.

Point-in-time data

Six months of OHLCV ending at each decision date and company news filtered to published on or before the date (anti-look-ahead, enforced in code). Fundamentals come in three vintages: excluded by default, --sec-edgar-fundamentals serves US statements exactly as filed on or before each date (SEC records the filing date of every figure, so it is a real point-in-time source and the fundamental analyst runs), and --allow-current-fundamentals replays current-vintage values with a large look-ahead warning in the report. Social posts have no point-in-time archive either, so sentiment replays with an honest neutral instead of leaking present-day chatter. Portfolio context and decision memory are disabled during replay so nothing leaks from after the date.

Grading with costs

BUY earns the window's return minus a round-trip cost, SELL scores zero by default (long-only; a flag grades shorts), HOLD scores zero, each compared with SPY over the same window. Aggregates: hit rate, cumulative return, Sharpe, max drawdown, and a buy-and-hold baseline.

Baselines and honest experiments

  • Baselines: every report includes an all-HOLD baseline, per-ticker buy-and-hold, and a deterministic momentum baseline under the same costs.
  • Holdout: dates before the holdout date form the tune period and dates on or after it an untouched test period, each with its sample size, a 95% bootstrap interval for mean alpha, and a plain verdict that refuses to promote results on thin samples.
  • Paired experiments: two backtests that differ in exactly one dimension (depth, rebuttals, forecast, sentiment, model), sharing the same snapshot cache, dates, costs, and seed, reporting the paired alpha difference with its own interval. Tune debate depth here, not by eyeballing one report.

Reproducibility

Each report ships with a manifest recording the code revision, decision policy version, configuration, a hash of every snapshot consumed, and the random seeds; the same manifest plus cache reproduces the same mock report.

Mock mode and costs

Mock mode is the default: free and deterministic. Running real LLMs prints a cost estimate first and flags the result as high memorization risk, because the model's training data already contains historical outcomes.

Reports and cache

Reports land in the backtests folder as JSON plus markdown; snapshots are cached in SQLite, and an offline flag makes cache misses fail instead of hitting the network, so warm re-runs make zero network calls. A smoke script regenerates the 3-ticker baseline report, and the committed holdout report honestly states its sample is too small to promote anything.