Lodestar.Embeddings 0.6.0. This page is frozen at that release. Read the current documentation for what main says now. A link to a decision or a migration page follows main, and leaves the archive.

Ids back to text, exactly.

public string Decode(IReadOnlyList<int> ids, bool skipSpecialTokens = false)
public string Decode(ReadOnlySpan<int> ids, bool skipSpecialTokens = false)

Parameters — ids are the ids to turn back into text. skipSpecialTokens drops control tokens rather than rendering them.

Returns — string, the text those ids encode.

Example — a round trip.

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["Ġ"] = 0, ["t"] = 1, ["o"] = 2, ["k"] = 3, ["e"] = 4, ["n"] = 5,
    ["to"] = 6, ["ken"] = 7, ["token"] = 8, ["Ġtoken"] = 9, ["ke"] = 10,
};
var merges = new List<MergePair> { new("t", "o"), new("k", "e"), new("ke", "n") };
var model = new BpeVocabulary(vocab, merges)
{
    ByteLevel = true,
    PreTokenizerPattern = BpePatterns.Gpt2,
    PreSplit = null,
};
var tokenizer = new BpeTokenizer(model);

TokenizationResult encoded = tokenizer.Encode("token");
string text = tokenizer.Decode(encoded.Ids);  // => token

Exceptions — ArgumentOutOfRangeException when an id falls outside the vocabulary. Decoding cannot silently skip one, since the caller would get back a shorter text than it asked for with nothing said about it. Nothing else on this path throws: a byte sequence that is not well-formed UTF-8 becomes U+FFFD rather than an exception, under whichever rule the file's own shape calls for. On the byte-level path it is one U+FFFD per maximal invalid subpart, which is what decision 0023 settled. On a byte_fallback file's run of byte pieces it is one U+FFFD per byte of the run — HuggingFace's own ByteFallback rule, measured and reproduced by decision 0063.

Remarks — byte-level BPE round-trips exactly, and that is the property that makes decoding worth having: the vocabulary covers all 256 byte values through printable stand-ins, so emoji, mixed scripts and even malformed UTF-8 come back as they went in.

The SentencePiece-BPE lineage is the one qualified case, and only for a file declaring neither a decoder this package undoes nor byte_fallback. Where the model declares the whitespace escape — a Metaspace pre-tokenizer or a Prepend + Replace normalizer, which TokenizerJsonLoader.LoadBpe reads — that escape is an encode-side transform in this package, and for such a file a Metaspace decoder block is accepted without being applied. So the text comes back with its replacement symbols in place of the spaces, and Decode(Encode(x)) is not x. Decision 0062 records it. For a file declaring byte_fallback, the chain is reproduced instead: a bare ByteFallback decoder undoes the byte pieces alone, and Llama-2's own Sequence of [Replace, ByteFallback, Fuse, Strip] undoes the byte pieces and the whitespace escape together, so Decode(Encode(x)) is x again for such a file. Decision 0063 has the measurements.

skipSpecialTokens is what you want when showing a generated sequence to a person, and not what you want when comparing against a reference that includes them.

Neither WordPieceTokenizer nor SentencePieceTokenizer offers this, which is why ISubwordTokenizer does not declare it — a promise two of the three cannot keep.

Applies to — net10.0, netstandard2.0.

See also — BpeTokenizer.Encode.