Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.8.0 — read its documentation.

Home › Embeddings › Tokenization

WordPieceTokenizer

Longest-match sub-word encoding with a continuation prefix — BERT's tokenizer.

public sealed class WordPieceTokenizer : ISubwordTokenizer

Constructor — takes a vocabulary, an unknown token, a continuation prefix, a per-word character cap and a lowercase flag; or a WordPieceVocabulary that carries all of them. Either throws ArgumentNullException when the vocabulary, or the unknown token or the continuation prefix passed alone, is null; ArgumentException when the unknown token is not among the entries or an id is negative — and, from a WordPieceVocabulary, when it lacks its entries, unknown token, continuation prefix or added tokens, or holds an added token with no string; ArgumentOutOfRangeException when maxCharsPerWord is negative, which the reference's unsigned count cannot be (#1505); and InvalidOperationException when the keys need more trie slots than one array holds (#1393).

Example — one word that splits into two pieces, one that does not.

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["[UNK]"] = 0, ["token"] = 1, ["##ize"] = 2, ["text"] = 3,
};
var tokenizer = new WordPieceTokenizer(
    vocab, unkToken: "[UNK]", continuationPrefix: "##", maxCharsPerWord: 100, lowercase: true);

TokenizationResult encoded = tokenizer.Encode("tokenize text");

string continuation = encoded.Tokens[1];  // => ##ize
int count = encoded.Ids.Count;  // => 3

Remarks — the algorithm is greedy longest-match within a word: take the longest vocabulary entry that prefixes what is left, mark every piece after the first with the continuation prefix, repeat. tokenize becomes token + ##ize because token is the longest entry that starts it.

The important failure is all-or-nothing: a word no sequence of pieces can cover becomes one unknown token, not a run of partial ones. Half a word is not a useful signal, and BERT agrees.

maxCharsPerWord is a guard rather than a tuning knob — a word longer than it becomes the unknown token without being attempted, which keeps a pathological input from costing quadratic time. It counts code points, as max_input_chars_per_word does, so a surrogate pair counts once.

Applies to — net10.0, netstandard2.0.

See also — WordPieceVocabulary, ISubwordTokenizer, the embeddings guide.

Members

Member	What it does
WordPieceTokenizer.Encode	Tokens and ids for one string.
WordPieceTokenizer.EncodeToIds	The ids alone, Encode's with the token strings left behind.
WordPieceTokenizer.TryGetId	The id of a token, if the vocabulary holds it.