Quality - sbOogway/sbt2 GitHub Wiki

Quality

v1 had no lint, no type checking and no CI. v2 has all of them from day one.

Safeguard Purpose
Synthetic test data A small generated dataset, so tests run fast with no downloads.
Golden-run tests A fixed spec must keep producing the same numbers. This catches refactors or nautilus upgrades that change results. See Golden-run tests below.
Nautilus characterization tests Pin every nautilus behavior the design relies on, so a wheel upgrade names the assumption it broke. See Nautilus characterization tests below.
Funding round-trip test The hand-written funding-rate file must stream back through a BacktestNode as the same FundingRateUpdates and settle the expected payment. See Data. Part of the characterization tests.
Book verification Replayed L2 deltas must match the next snapshot. See Data.
Lint + type checking ruff, pyright and import-linter, enforced by pre-commit hooks that CI runs. Warnings fail like errors: pyright runs with --warnings, and pytest turns any warning a test raises into a failure (filterwarnings = ["error"]), so a dependency's new deprecation fails the bump that brings it. Beyond its defaults, ruff enforces the function rules of AGENTS.md: at most three arguments, no boolean flag arguments, no commented-out code and a complexity cap. It also rejects access to private members, naive datetimes, unused arguments (an override that ignores one is marked @override) and magic numbers outside the tests, and runs its security rules (bandit) on the shipped code. The Typer commands in cli.py and pytest tests, whose fixtures arrive as arguments, are exempt from the argument limit. Pyright also fails on any use of a deprecated API.
Protocol checks the repository formats and lints the protocol's .proto files and rejects breaking changes against sbt2-protocol/ on main; the backend fails when its committed generated Python is stale against them. See Protocol checks below.
Dependency audit Dependabot alerts and security updates watch sbt2-backend/uv.lock for known vulnerabilities and open a pull request with the fixed version. nautilus-trader isn't covered: it comes from Nautech's index, not PyPI.
CI Woodpecker CI (v3), self-hosted, with GitHub as the forge. The server runs in rootless Podman; the agent runs on a separate LAN host and starts the steps in its own Docker daemon, capped at 4 GB of memory and without swap, so CI can't starve that host's other services. Pull requests from forks need approval before they run. Each workspace member is tested with only itself and the members it declares installed. See Test layout and CI below.

nautilus_trader v2 is pinned to an exact release, currently 2.0.0rc6.dev20260930.

That's a nightly wheel, not on PyPI, so the project adds Nautech's package index for this one package only and the lockfile records the exact wheel:

[[tool.uv.index]]
name = "nautech"
url = "https://packages.nautechsystems.io/simple"
explicit = true

[tool.uv.sources]
nautilus-trader = { index = "nautech" }

with "nautilus-trader[visualization]==2.0.0rc6.dev20260930" in the dependencies; the visualization extra brings plotly, which the tearsheet needs. CI installs from the lockfile. Nautilus is never built from source here: it needs more memory than the local machine has.

Retention on Nautech's index (nautilus README, "Retention policies"):

  • develop wheels (.devYYYYMMDD+run, a CI run number) keep only the most recent build, so the next merged commit deletes them. Never pin one. The earlier pin 2.0.0rc6.dev20260927+18861 was one, and CI got a 404 the next day.
  • nightly wheels (.devYYYYMMDD) keep the 30 most recent publication dates, so roughly a month.
  • PyPI releases are permanent.

A published sbt2 can't carry this index to its users, so releases to PyPI need a stable nautilus from PyPI (see Library). A nightly pin also expires. Before it ages out, move to a PyPI release, or bump to a newer nightly through the upgrade steps below.

The nautilus-bump step of the live pipeline opens a build(deps) pull request whenever Nautech publishes a newer nightly (never a develop build), so CI checks the upgrade before the pinned nightly ages out. Dependabot can't: it doesn't read the explicit index and can't tell nightlies from develop builds.

Upgrading: release candidates still change the API (rc5 → rc6 renamed methods and fee-model arguments), so upgrades are deliberate. Bump the pin, run uv run pytest -m characterization first, then the golden-run tests (uv run pytest -m golden). A golden run includes funding, so a nautilus change to funding settlement fails CI.

Test layout and CI

Each package carries its own tests; sbt2-backend/tests/ holds the ones that span packages or test the repo.

Backend commands below run from sbt2-backend/, or through make test-backend at the repository root. make test-protocol runs the protocol's Python and Rust tests, and make check runs every check of the repository.

Tree Tests
sbt2-backend/packages/sbt2-core/tests/ assets, data, run, spec, strategy, results, cli and realdata. They run a test-local copy of the moving-average crossover (crossover.py) and other toy strategies, never sbt2.strategies.
sbt2-backend/packages/sbt2-strategies/tests/ The example strategies: their parameters, and their bar handlers in a backtest engine on synthetic 1-minute bars (bar_engine.py).
sbt2-backend/packages/sbt2-server/tests/ The server: the token check and the router as units; the handshake, health, the Hello gate, errors, the outgoing queue, offloading and interleaved result streams against a real server on a free localhost port with a websockets client; the results area against a parquet store of a short nautilus run and a small price catalog, checked against sbt2-core's own metrics, curve and tearsheet; the sbt2-server command. Tests are synchronous and drive each scenario with asyncio.run, so no pytest plugin is needed.
sbt2-backend/tests/ golden (core running the shipped strategies, the end-to-end contract of the sbt2 distribution), characterization (pins nautilus and systemd, not an sbt2 package), container (the server's image and compose.yaml), run (scripts calling batches), scripts, the workspace smoke test test_packages.py, and the helpers the core tests share with them (launchers.py, plotted.py).

sbt2-papers has no tests tree until it has hand-written code; the smoke test imports sbt2-server's generated sbt2.protocol.v1.envelope_pb2.

uv run pytest runs all four trees (testpaths). The --update-golden option and the check that every test carries exactly one level marker (unit, integration or e2e) live in the workspace plugin sbt2-backend/tests/pytest_sbt2.py, loaded with -p, so they apply to every tree. pytest imports test files with --import-mode=importlib, which leaves sys.path alone; the folders holding helper modules are listed in its pythonpath. Batch children inherit the parent's sys.path, so they import test strategies by the same paths (run_strategies:BuyThenSell).

Each member has a dev dependency group with only pytest and pytest-cov; the workspace's dev group keeps every tool for local work.

CI runs one test job per member: uv sync --locked --package <member> installs only that member and what it declares, so a dependency the member uses but doesn't declare fails its job. The job runs the member's tests with coverage measured on that package alone, from the member's own coverage config, and fails under 85%. It uploads coverage and test results to Codecov under a flag named after the package (sbt2-core, sbt2-strategies, sbt2-server). A separate job runs sbt2-backend/tests/ on the whole workspace and uploads their results under the tests flag; the characterization tests run in their own job under the characterization flag. The protocol's Python tests and its Rust golden tests run in two more jobs, so both components are checked on every pull request. The image job builds the server's Containerfile, with sbt2-backend/ as its context, using buildah, and pushes it nowhere. On a pull request it runs only when the Containerfile, .dockerignore, uv.lock, a pyproject.toml or the backend workflow changes, since only these can break the build; pushes to main, tags and manual runs always build it. It runs in an unprivileged step container, which can neither mount overlays nor set up namespaces, so buildah uses the vfs storage driver and chroot isolation. Its cache volume keeps the base images and the build's uv cache between runs; the built image is removed after each build, since vfs stores a full copy of every layer.

Parallel steps share the checkout, so each Python step keeps its environment and reports in its own existing /cache volume: UV_PROJECT_ENVIRONMENT=/cache/venv, COVERAGE_FILE=/cache/.coverage, /cache/junit.xml and /cache/coverage.xml. Because the environment and uv's cache share that volume, uv hard-links packages instead of copying them. This prevents one job's uv sync from removing another job's dependencies, coverage from combining unrelated suites, and jobs from overwriting each other's reports. Test selections and coverage thresholds stay the same.

Core and workspace coverage enable concurrency = ["multiprocessing"] to measure the children forked from the forkserver. The subprocess patch remains enabled for commands started outside multiprocessing. Without multiprocessing support, the child runner and order receiver execute successfully but their work is absent from the coverage report.

The batch failure tests also cover a running child that ignores SIGTERM: another child fails only after the resistant child confirms its signal handler is installed. Cleanup must kill the resistant child and start no further run, an edge case of the existing failure behavior.

Protocol checks

The protocol (Protocol on Architecture) lives in sbt2-protocol/, in this repository, and the repository's prek hooks check it: they are the official bufbuild/buf hooks pinned to a buf release (v1.73.0), and prek builds buf with the Go toolchain the hooks name, so neither a developer nor a CI image installs buf. Each hook names sbt2-protocol/ itself, because buf takes it as its module.

Hook Where Checks
buf format sbt2-protocol/ the .proto files are formatted as buf formats them, rewriting them if not
buf lint sbt2-protocol/ buf's STANDARD rules (buf.yaml): naming, _UNSPECIFIED zero values, the package matching its folder and versioned
buf breaking sbt2-protocol/ the FILE rules against sbt2-protocol/ on main: no change that breaks the wire, JSON or the code generated from it, so a breaking change needs sbt2.protocol.v2
generate-protocol sbt2-backend scripts/generate-protocol.sh regenerates sbt2-server's sbt2.protocol from sbt2-protocol/ aside, then replaces it, so the modules of a deleted .proto file go too; the hook runs on every commit and fails when that changed a committed file

The protocol's Python tests in sbt2-protocol/tests/ run through a root hook and a CI job. They check buf's descriptor set and decode and re-encode every fixture. Its Rust checks in sbt2-protocol/checks/rust/ do the same round trip with prost; they run in CI and through make test-protocol.

scripts/check-protocol-breaking.sh fetches a fresh snapshot of this repository's main into a temporary repository. It compares against the tracked sbt2-protocol/ files there. While main still holds the old submodule, it compares against that submodule's pinned commit. Missing baselines and fetch failures stop the check; no rule is skipped.

Newline and trailing-whitespace hooks check the protocol's text files. Binary .binpb fixtures and generated backend modules are excluded. The generated code is committed, so installing from git, an sdist or uv sync needs no protoc, no buf and no submodule; a clone needs no submodule step at all. The generators are dev dependencies pinned in sbt2-backend/uv.lock: grpcio-tools, whose bundled protoc writes the modules, and mypy-protobuf for the stubs. Generating runs offline, in about a second, so the hook can run on every commit; no build step calls an online service, since a rate-limited registry once blocked every commit. protoc's gencode version must not be newer than the floor of sbt2-server's protobuf dependency, because the generated modules refuse an older runtime. ruff and the whitespace hooks skip the generated folder; pyright checks it like any other module, except for reportIncompatibleVariableOverride: mypy-protobuf's stubs narrow Message.DESCRIPTOR, which typeshed declares as a union, and mypy-protobuf turns that rule off for generated code too.

Nautilus characterization tests

Characterization tests record how nautilus behaves today on the pinned wheel, whether or not that behavior is documented. They live in sbt2-backend/tests/characterization/nautilus/ under the characterization pytest marker. They use a few synthetic ticks, need no network and run in their own CI job.

Each test protects a design decision. When one fails after an upgrade, the decision it names has to be re-checked before the pin moves.

Test file Pinned behavior Protects
test_funding_settlement.py Funding = rate × mark notional, recorded as a PositionAdjusted (FUNDING); never settles without interval; settles only when ts_event is an exact multiple of interval since the Unix epoch; a position opened at the funding timestamp pays it, one opened 1 ns later doesn't; reversing a netting position moves its funding adjustments into a position snapshot Native funding in Crypto perpetuals, the carry ledger in Results, funding ingest in Source Bybit
test_venue.py A venue without an explicit fee_model is refused; positions are valued at mark price by default and at quotes with use_mark_prices=False Venue profiles in Spec, valuation in Run
test_liquidation.py With liquidation on, a position that falls below maintenance margin is closed when quotes stream, and never when only trades and mark prices do Liquidation off by default and the pre-flight liquidation check in Crypto perpetuals and Run
test_portfolio_snapshots.py The default equity curve snapshots at start, every UTC midnight and end; equity_curve=False turns it off; snapshot_interval_ms samples only while a position is open Equity from portfolio snapshots in Run
test_internal_bars.py INTERNAL time bars built from trades close on the right edge, include a trade exactly on the boundary, and never emit the trailing partial bar Bars built at run time in Data
test_catalog.py Files are named after the start/end passed to the writer (after the first and last timestamps without them); adjacent day files form one covered interval; a missing day is reported exactly; there is no Python funding writer; a hand-written funding file in nautilus's schema reads back unchanged Day files and coverage in Data
test_empty_day.py The typed writers write no file for an empty list; a zero-row day file named with the day's bounds counts as covered, adds no rows, and BacktestNode streams across it Zero-row empty days in Data
test_atomic_write.py A typed write goes to <file>.parquet#N and is renamed when complete; a write killed midway leaves no covered day, and its leftover temporary file neither counts as coverage nor blocks the next write Day-atomic ingest in Data
test_backtest_node.py BacktestNode streams quotes, mark prices and funding from the catalog and the venue settles the funding; with dispose_on_completion=False the cache and snapshots stay readable; the default disposal silently empties the engine cache execute in Run, the funding round trip
test_brackets.py On 1-minute bars and on trades: a bracket's filled exit cancels the other; a take-profit fills at its price, a stop-loss at its trigger on a bar and at the trade's price on a gap; modify_order moves a resting stop or a bracket's stop-loss and changes a resting limit's price and quantity; a partially filled entry releases its exits for the full entry quantity, and its remainder stays open and fills unprotected after an exit fills Brackets and stops in Strategy

Findings from these tests that the design depends on:

  • Funding timestamps must be exact. A record that is off its epoch-aligned interval boundary, even by 1 ns, is never charged and nothing reports it. See Source Bybit.
  • Funding lives in position snapshots too. After a netting reversal, the funding paid before the flip is only on the snapshot. See Run.
  • Nautilus's typed writers are atomic. Ingest needs no staging area of its own; it only mirrors the rename for the files it writes with pyarrow. See Data.
  • dispose_on_completion=False is required. With the default, the run returns normally but the cache has no account, positions or funding adjustments. See Run.
  • A partially filled bracket entry is left unprotected. Nautilus doesn't cancel the entry's remainder when an exit fills, so a strategy entering with a limit cancels it itself. See Strategy.

systemd characterization tests

They pin how a transient systemd scope, started over D-Bus with StartTransientUnit around a running PID, caps a run's memory, which --launcher systemd relies on (see Memory cap in Run). They talk to systemd with jeepney, as SystemdScope does. They live in sbt2-backend/tests/characterization/systemd/ under the characterization and systemd markers. They're excluded by default and in CI, because they need a systemd user session and its D-Bus session bus, which the CI container doesn't have: uv run pytest -m systemd.

sbt2-backend/packages/sbt2-core/tests/run/test_systemd_scope.py runs batch itself on SystemdScope, under the systemd marker too: a run under its cap is stored, a run over it raises OutOfMemoryError, and its scope is reset. The two over-cap tests use a test strategy that allocates and writes 256 MiB under a 100 MiB cap; shared forkserver imports no longer make an ordinary strategy exceed that cap. Their error and scope-cleanup assertions stay the same. Its no-session test isn't marked, and runs in CI. The batch, CLI and golden tests without the systemd marker leave their children uncapped, and some replace the batch's private spawn of a child with scripted children, forked from the same forkserver. sbt2-backend/tests/run/test_script_batches.py runs a script that calls batch with and without the if __name__ == "__main__": guard.

Pinned behavior Protects
A child over MemoryMax with MemorySwapMax=0 is killed with SIGKILL (return code −9), and its scope's Result is oom-kill The out-of-memory error in Run
A PID placed into a scope by StartTransientUnit is in the scope's cgroup once the start job reports done batch sending the order only after cap returns
A child under the cap runs to completion, and its scope disappears within 5 s Runs within their cap are unaffected
A killed scope stays loaded until ResetFailedUnit batch reading Result, then resetting
With MemoryMax alone, a child over the cap survives by swapping (skipped on hosts without swap) MemorySwapMax=0 in batch
Without a user session bus, jeepney opens no connection and raises KeyError The up-front session check in SystemdScope.check

Container tests

sbt2-backend/tests/container/test_image.py builds the server's image from the Containerfile once per session, with podman and --format docker so that the HEALTHCHECK is kept, and runs it: GET /health answers 200 from a server that isn't root, a websockets client presenting the token gets a Welcome carrying the package version, a container without SBT2_SERVER_TOKEN exits with status 2 naming it, and the engine marks a container healthy within the start period. They're under the container marker, excluded by default and in CI, because they need podman, which the CI step container doesn't have: uv run pytest -m container tests/container. CI checks only that the image builds (the image job above). sbt2-backend/tests/container/test_compose_file.py reads compose.yaml with PyYAML and checks its one service, volumes, port, token and health check; it isn't marked, and runs in CI with the other backend tests.

Golden-run tests

They live in sbt2-backend/tests/golden/ under the golden pytest marker and run in CI with the other backend tests.

  • Data: their own synthetic catalog (golden_catalog.py), separate from the one the execute tests use. Two perps over five days, each with a trade and a mark price every minute at the same price, and funding every 8h on the epoch-aligned boundaries, cycling through positive and negative rates. BTCUSDT zigzags every 20h on a slow uptrend. ETHUSDT zigzags every 16h on a slow uptrend, then on the last day slides 200 USDT a minute for four minutes from 06:30.
  • Candles: the golden catalog also holds a flat 1-minute candle a minute at the trades' price, for the *_candles.toml specs, each the same spec as its trades twin with bars = "candles". They settle the same funding but fill differently: at the last candle's close, in chunks of a quarter of the candle's volume, and a stop gapped through by a candle fills between its trigger and the candle's price (on trades it fills at the gapping trade's price). The ma_cross candles run also enters an hour later, since the trades run gets a one-trade bar at the data start.
  • Specs: all run on the real bybit_linear venue profile, so a change to that profile's fees changes the golden numbers too. The strategies are the examples in sbt2.strategies (the sbt2-strategies distribution).
    • ma_cross.toml runs the moving-average crossover (MovingAverageCross, long/short, reversing on every cross) on BTCUSDT.
    • bracket_risk.toml runs BracketRisk on BTCUSDT and ETHUSDT: whenever an instrument is flat, it enters a market-entry bracket on the side of the price against its moving average, sized with nautilus's FixedRiskSizer. The spec sets every [risk] key, with engine limits loose enough to deny nothing and a 6% drawdown_limit. ETH's slide gaps through a long's stop-loss, so the guard trips at 07:00 on the last day and exits the market; BTC and ETH both fill take-profits and stop-losses before that.
  • Run: through sbt2 run, as a user would, with each run in a plain child process instead of a systemd scope.
  • Checked: the summary's headline metrics, the carry ledger, the fills and the equity snapshots, against the spec's .json file, plus the drawdown trip time for a run that trips. Strings (money, prices, timestamps) must match exactly and floats to a relative 1e-9. Every run must settle at least one funding payment. The bracket runs are also checked for not degenerating: both instruments trade, at least one take-profit and one stop-loss fill, and the guard trips.
  • Updating: after an intended change, uv run pytest -m golden --update-golden rewrites the JSON, and the diff shows what moved. A diff that isn't explained by the change is a bug.

Real-data tests

They live in sbt2-backend/packages/sbt2-core/tests/realdata/ under the realdata pytest marker. They're excluded by default and in CI, because they need a month of real data: uv run pytest -m realdata.

  • Data: the gitignored data/ folder in sbt2-backend/. The run's pre-flight downloads and ingests the days it lacks, so the first run fetches the month (2.6 GB raw, 2.1 GB catalog). Results go to a throwaway folder.
  • Spec: btcusdt_month.toml runs the core tests' copy of MovingAverageCross (crossover.py) with its default params (1h bars, 10/30, 0.1 BTC) on Bybit BTCUSDT for January 2025, on the bybit_linear venue profile.
  • Run: through sbt2 run, as a user would.
  • Checked: the carry ledger has one payment at every funding settlement from the first fill to the end of the month (89 payments), and each one equals −position × mark × rate, to within 1e-8 USDT. The position is rebuilt from the fills, and the mark is the last MarkPriceUpdate at or before the settlement.

Measured on a 20-core, 23 GB machine (January 2025 plus warm-up, 33 days, 76.1M trades):

Step Wall time Peak RSS
sbt2 download 29 s 0.3 GB
sbt2 ingest 2 min 28 s 6.8 GB, on the busiest day (see Data)
sbt2 run (71.7M engine iterations) 54 s, about 1.5M trades/s 1.2 GB

btcusdt_month_candles.toml runs the same month with bars = "candles", under the same checks:

Step Wall time Peak RSS
sbt2 download --type Bar 3.4 s 0.3 GB
sbt2 ingest --type Bar 0.7 s 0.2 GB
sbt2 run 1.3 s 0.4 GB

Both are fast enough to keep building bars at run time (see Decisions). Before liquidation was turned off for runs without quotes, the same run took 2 min 27 s and logged 18.5M warnings (see Crypto perpetuals).

⚠️ **GitHub.com Fallback** ⚠️