Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.7.0 — read its documentation.

Home › Embeddings › Tokenization

Ids back to text, exactly.

public string Decode(IReadOnlyList<int> ids, bool skipSpecialTokens = false)
public string Decode(ReadOnlySpan<int> ids, bool skipSpecialTokens = false)

Parameters — ids are the ids to turn back into text. skipSpecialTokens drops control tokens rather than rendering them.

Returns — string, the text those ids encode.

Example — a round trip.

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["Ġ"] = 0, ["t"] = 1, ["o"] = 2, ["k"] = 3, ["e"] = 4, ["n"] = 5,
    ["to"] = 6, ["ken"] = 7, ["token"] = 8, ["Ġtoken"] = 9, ["ke"] = 10,
};
var merges = new List<MergePair> { new("t", "o"), new("k", "e"), new("ke", "n") };
var model = new BpeVocabulary(vocab, merges)
{
    ByteLevel = true,
    PreTokenizerPattern = BpePatterns.Gpt2,
    PreSplit = null,
};
var tokenizer = new BpeTokenizer(model);

TokenizationResult encoded = tokenizer.Encode("token");
string text = tokenizer.Decode(encoded.Ids);  // => token

Exceptions — ArgumentOutOfRangeException when an id falls outside the vocabulary. Decoding cannot silently skip one, since the caller would get back a shorter text than it asked for with nothing said about it. Nothing else on this path throws: a byte sequence that is not well-formed UTF-8 becomes U+FFFD rather than an exception, under whichever rule the file's own shape calls for. On the byte-level path it is one U+FFFD per maximal invalid subpart, which is what decision 0007 settled. On a byte_fallback file's run of byte pieces it is one U+FFFD per byte of the run — HuggingFace's own ByteFallback rule, measured and reproduced by decision 0007.

Remarks — byte-level BPE round-trips exactly, and that is the property that makes decoding worth having: the vocabulary covers all 256 byte values through printable stand-ins, so emoji, mixed scripts and even malformed UTF-8 come back as they went in.

The SentencePiece-BPE lineage is the one qualified case, and only for a file declaring neither a decoder this package undoes nor byte_fallback. Where the model declares the whitespace escape — a Metaspace pre-tokenizer or a Prepend + Replace normalizer, which TokenizerJsonLoader.LoadBpe reads — that escape is an encode-side transform in this package, and for such a file a Metaspace decoder block is accepted without being applied. So the text comes back with its replacement symbols in place of the spaces, and Decode(Encode(x)) is not x. docs/equivalence.md's Metaspace rows records it. For a file declaring byte_fallback, the chain is reproduced instead: a bare ByteFallback decoder undoes the byte pieces alone, and Llama-2's own Sequence of [Replace, ByteFallback, Fuse, Strip] undoes the byte pieces and the whitespace escape together, so Decode(Encode(x)) is x again for such a file. Decision 0007 has the measurements.

skipSpecialTokens is what you want when showing a generated sequence to a person, and not what you want when comparing against a reference that includes them.

Neither WordPieceTokenizer nor SentencePieceTokenizer offers this, which is why ISubwordTokenizer does not declare it — a promise two of the three cannot keep.

Applies to — net10.0, netstandard2.0.

See also — BpeTokenizer.Encode.