Data sources

The framework is source-agnostic, just as it is asset-class-agnostic (see Asset classes). A source adapter plugs into Data: download (where files live, how to fetch them) and ingest (how to parse them). Everything downstream of the catalog only sees nautilus types and never knows which source the data came from.

The adapter interface

Each adapter subclasses Source, the abstract base class in data/sources/base.py. Other modules get one with sbt2.core.data.sources.source(name, known_gaps_file) and never import an adapter.

A subclass implements what is specific to its source:

Member	What it gives
name	class variable: the source's list in known_gaps.toml, e.g. "bybit"
data_types	the nautilus types served as one raw file per UTC day (e.g. TradeTick, FundingRateUpdate, MarkPriceUpdate)
day_file(symbol, data_type, day)	a RawFile for that day
instrument_snapshot(symbol, taken_on)	a RawFile for today's instrument spec, named with the date it's taken on
instrument_id(symbol)	the source's nautilus InstrumentId format
symbol(instrument_id)	the inverse of instrument_id, so Run can fetch a spec's instruments by symbol
parse(path, data_type, instrument)	a raw day file's records as nautilus types, in time order, for ingest; the instrument gives precisions and anything else the parser needs
parse_instrument(path)	the instrument of a snapshot, with ts_init at the start of the day it was taken on

The base owns what every source shares:

Member	What it gives
Source.listing(entries)	the source built from its list in known_gaps.toml, each entry turned into a Gap through instrument_id and the source's names for its data types
known_gaps	the gaps below, as (instrument id, data type, day)
served(names)	the served data types with those nautilus names, all of them when none are given; an unserved name raises UnsupportedDataTypeError listing what the source serves
is_known_gap(symbol, data_type, day)	whether that day is a known gap, so download and ingest skip it

A subclass is constructible from its known gaps alone, since listing builds it that way. The known gaps file names a data type by its nautilus name (TradeTick) unless the source overrides _listed_data_type, as Bybit does with its channel names (trades).

A RawFile is a path under the raw root plus its origin: either a URL that download fetches itself (a 404 means missing), or a coroutine returning the file's bytes, for data the source fetches from an API; it raises MissingAtSourceError when the source has nothing. Download stays source-agnostic either way: it owns scheduling, retries, .part files and the missing-days summary.

A source ships inside the package that provides it (sbt2 itself for Bybit) and registers under the sbt2.sources entry point, so another package can add a source without a release of sbt2.

Known gaps

Sources sometimes lack a day (e.g. a missing dump file). Each source keeps a known-gaps list in the data root's known_gaps.toml, one list per source (e.g. bybit = [{ symbol = "BTCUSDT", data = "trades", day = 2020-03-25 }]): (instrument, data type, day) entries confirmed as unavailable at the source. A missing file means no known gaps; known_gaps(file) gives every source's, for sbt2 data status. The file is edited by hand or from the GUI: listed_gaps(file) lists the entries with their source, and add_known_gaps and remove_known_gaps change them, each write validated (a known source and data type) and atomic (see Config area on Architecture). Run's coverage check allows these days and records them in the run's summary; any other missing day still fails the run. A gap is added to the list deliberately, never automatically. A day the source serves with no rows is not a gap: ingest writes it as a zero-row file (see Data).

Source	Status	Page
Bybit (linear perps: L2, trades, funding, mark price)	first adapter	Source Bybit