Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.8.0 — read its documentation.

Home › Embeddings › Tokenization

Byte-level and classic BPE — GPT-2, Llama-3, Qwen2 — and the only tokenizer here that decodes.

public sealed class BpeTokenizer : ISubwordTokenizer

Constructor — takes a BpeVocabulary. It throws ArgumentNullException when the vocabulary is null; ArgumentException when a token a merge or the unknown token names is not in the vocabulary, a byte-level one declares a continuing prefix, the split is malformed, a normalization form is undefined, byte fallback lacks a piece, a Metaspace escape meets a normalized added token, a list is missing, a symbol or added token is null, an id is negative, the merges pass 2^29 (#1435), or a split or pre-tokenizer pattern does not parse (#1494); and InvalidOperationException when the vocabulary's keys need more trie slots than one array holds (#1393).

Example — three merges turning five characters into two tokens, and back.

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["Ġ"] = 0, ["t"] = 1, ["o"] = 2, ["k"] = 3, ["e"] = 4, ["n"] = 5,
    ["to"] = 6, ["ken"] = 7, ["token"] = 8, ["Ġtoken"] = 9, ["ke"] = 10,
};
var merges = new List<MergePair> { new("t", "o"), new("k", "e"), new("ke", "n") };
var model = new BpeVocabulary(vocab, merges)
{
    ByteLevel = true,
    PreTokenizerPattern = BpePatterns.Gpt2,
    PreSplit = null,
};
var tokenizer = new BpeTokenizer(model);

TokenizationResult encoded = tokenizer.Encode("token");
string roundTripped = tokenizer.Decode(encoded.Ids);

int pieces = encoded.Tokens.Count;  // => 2

Remarks — encoding starts from characters and applies the ranked merges in order: t o k e n becomes to k e n, then to ke n, then to ken. The result is two tokens, and no rule joins them because none is listed.

It is the only one of the three that can Decode, and byte-level is why: the vocabulary covers all 256 byte values through printable stand-ins, so well-formed text round-trips exactly, emoji and mixed scripts alike. Two limits: an unpaired surrogate is not well-formed and throws on Encode, and a declared normalizer — NFC, say — changes the text before it is encoded, so decoding returns the normalized text. WordPiece and SentencePiece have thrown information away by then.

Applies to — net10.0, netstandard2.0.

See also — BpeVocabulary, MergePair, ISubwordTokenizer.