Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.7.0 — read its documentation.

Home › Embeddings

A transformer does not read text; it reads token ids. This namespace turns one into the other, for the three sub-word algorithms the models in use are built on, and it reproduces HuggingFace tokenizers and sentencepiece closely enough that the ids match theirs.

Getting them to match matters more than it sounds: a model fed ids from the wrong tokenizer returns vectors that are confidently wrong rather than an error.

The answer is whichever the model was trained with — this is not a choice you get to make. The model's own files say which, and which loader reads them:

flowchart TD
    A["What did the model ship?"] --> B["vocab.txt"]
    A --> C["spiece.model"]
    A --> D["tokenizer.json"]
    A --> E["vocab.json + merges.txt"]

    B --> W["WordPieceTokenizer<br/>VocabTxtLoader"]
    C --> C1{"Trained with<br/>byte_fallback?"}
    C1 -->|no| S["SentencePieceTokenizer<br/>SentencePieceModelLoader"]
    C1 -->|yes| X["Refused at load, by design<br/>(the Unigram pipeline does not reproduce it)"]

    D --> D1{"What does<br/>model.type say?"}
    D1 -->|WordPiece| W2["WordPieceTokenizer<br/>TokenizerJsonLoader.LoadWordPiece"]
    D1 -->|Unigram, byte_fallback set| X
    D1 -->|Unigram, no byte_fallback| S2["SentencePieceTokenizer<br/>TokenizerJsonLoader.LoadUnigram"]
    D1 -->|BPE| P["BpeTokenizer<br/>TokenizerJsonLoader.LoadBpe"]

    E --> P2["BpeTokenizer<br/>BpeFilesLoader"]

A tokenizer.json does not say which loader to call — its model.type does. The three Load… methods each assert it and refuse a file declaring another, so reaching for the wrong one fails with a message naming the mismatch rather than producing ids that look plausible.

byte_fallback means something different on each path it can appear on. Python resolves an uncovered character into <0x..> byte pieces where these tokenizers would otherwise emit the unknown piece — silently ignoring the flag would return confidently wrong vectors, so the Unigram lineage (spiece.model, or a tokenizer.json declaring model.type: "Unigram") refuses a checkpoint that declares it, unconditionally: that pipeline does not reproduce it. The BPE lineage does: TokenizerJsonLoader.LoadBpe resolves an uncovered symbol into the <0xXX> byte pieces the flag promises, which is what lets Llama-2 and Mistral v0.1 — the SentencePiece-BPE lineage tracked at #175 and scoped by decision 0005 §3 — load at all, and refuses only a vocabulary that declares the flag without carrying all 256 pieces (decision 0007).

The same routing, as a table:

All three implement ISubwordTokenizer, so code that only encodes can be written once against that.

WordPiece splits a word into the longest pieces its vocabulary holds, marking every piece after the first with a continuation prefix — ##ize is "ize, continuing a word". A word it cannot cover at all becomes a single unknown token, not a sequence of partial ones.

SentencePiece treats the text as a stream and encodes the space itself, as ▁. That is why its tokens carry a leading ▁ and why it needs no pre-tokenizer: word boundaries are inside the vocabulary rather than assumed by a regex.

BPE starts from characters and applies a ranked list of merges in order. Byte-level BPE — what GPT-2 and Llama-3 use — maps bytes to printable characters first, which is what lets it round-trip any input exactly, emoji and broken UTF-8 included.

Encoding one string gives a TokenizationResult. Feeding a model wants more than that: a rectangular batch, padded, with an attention mask and the model's own special tokens. BatchEncoder does that, driven by EncodingOptions and a SpecialTokenTemplate, and hands back an EncodedBatch.

The template has to match the model and the vocabulary: asking for SpecialTokenTemplate.Bert against a vocabulary with no [CLS] is refused at construction rather than encoded into something the model will misread.

• Semantic search with embeddings — the guide, end to end.
• ONNX inference — Lodestar.Onnx, what consumes the ids this namespace produces.
• Python → C# equivalence — every tokenizers call and its counterpart.