Home › Embeddings › Tokenization

A BPE model: the vocabulary, the ranked merges, and the flags that decide how they are applied.

public sealed record BpeVocabulary

Properties — Vocab maps token to id and Merges is the ranked MergePair list. ByteLevel selects byte-level spelling. AddPrefixSpace prepends a space to every text. IgnoreMerges short-circuits to a whole-piece vocabulary lookup. FuseUnk collapses adjacent unknown tokens into one. ByteFallback resolves an uncovered symbol into <0xXX> byte pieces instead, and LoadBpe requires the vocabulary to carry all 256 of them when it is set. EndOfWordSuffix and ContinuingSubwordPrefix are the classic-BPE markers. UnkToken is the fallback. PreTokenizerPattern, NoPreTokenizer and PreSplit decide what the merge loop sees. NormalizationForms are applied first. PrefixTokens and SuffixTokens are what the file's post_processor wraps a sequence in — ["<s>"] and empty for Llama-2 and Mistral v0.1 — and they are the two lists a caller pairs with a pad token of their own to build a SpecialTokenTemplate, since the file states no such token. Count is the vocabulary size.

Example — a byte-level model with three merges.

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["Ġ"] = 0, ["t"] = 1, ["o"] = 2, ["k"] = 3, ["e"] = 4, ["n"] = 5,
    ["to"] = 6, ["ken"] = 7, ["token"] = 8, ["Ġtoken"] = 9, ["ke"] = 10,
};
var merges = new List<MergePair> { new("t", "o"), new("k", "e"), new("ke", "n") };
var model = new BpeVocabulary(vocab, merges)
{
    ByteLevel = true,
    PreTokenizerPattern = BpePatterns.Gpt2,
    PreSplit = null,
};

int size = model.Count;  // => 11
int rules = model.Merges.Count;  // => 3

Remarks — sixteen properties because a tokenizer.json has that many knobs and getting any of them wrong changes the ids. The ones that surprise:

• PreSplit = null is not the same as NoPreTokenizer = true. The first says "no Split step ahead of ByteLevel", which is stock GPT-2; the second says "no pre-tokenizer at all", so the merge loop sees the whole text including its spaces.
• IgnoreMerges makes a whole piece present in the vocabulary win over any merging. Llama-3 sets it, and without it the same file tokenizes differently.
• AddPrefixSpace changes every first token of every text. It is a property of the model, not a preference.
• ByteFallback needs the vocabulary to already carry all 256 <0xXX> pieces; LoadBpe refuses a file that sets the flag without them rather than let it degrade silently.

Vocab, Merges and the other collections are taken as given, not copied: a caller casting one back to its collection and writing to it changes this record and what it equals, and can change its hash (#1305). BpeTokenizer copies them at construction, so a tokenizer already built is unaffected.

Applies to — net10.0, netstandard2.0.

See also — BpeTokenizer, MergePair, BpeSplitStep, BpePatterns.