Home › Embeddings › Tokenization

WordPieceVocabulary

The vocabulary and the settings that decide how it is read.

public sealed record WordPieceVocabulary

Properties — Vocab maps token to id. UnkToken is the token a word that cannot be covered becomes. ContinuationPrefix marks a piece that continues a word, ## in BERT. Lowercase says whether text is folded before matching. AddedTokens are the literal matches applied first. BasicTokenization runs BERT's BasicTokenizer ahead of WordPiece; VocabTxtLoader sets it, as TokenizerJsonLoader.LoadWordPiece does for a stock BERT file, and off, text is split as pre_tokenizers.Whitespace() splits it. PrefixTokens and SuffixTokens are what a tokenizer.json's post_processor wraps a sequence in — ["[CLS]"] and ["[SEP]"] for BERT, empty when it declares none — the two lists a caller pairs with a pad token to build a SpecialTokenTemplate. Count is how many entries the vocabulary holds.

Example — loading one from the vocab.txt a model ships.

using System.Text;
using Lodestar.Embeddings.Persistence;
using Lodestar.Embeddings.Tokenization;

var bounds = new ArtifactLoadOptions();
byte[] file = Encoding.UTF8.GetBytes("[UNK]\ntoken\n##ize\ntext");

WordPieceVocabulary vocabulary = VocabTxtLoader.Load(
    new MemoryStream(file), bounds, unkToken: "[UNK]", continuationPrefix: "##", lowercase: true);

int count = vocabulary.Count;  // => 4

Remarks — a vocab.txt is one token per line and the id is the line number, which is why loading it needs no ids and why editing such a file by inserting a line renumbers everything after it. That is a real way to break a model quietly.

The settings travel with the vocabulary rather than with the tokenizer because they are properties of the file: a vocabulary trained lowercase cannot be read case-sensitively, and pairing it with the wrong ContinuationPrefix produces tokens that exist nowhere in it.

Vocab, AddedTokens and the other collections are taken as given, not copied: a caller casting one back to its collection and writing to it changes this record and what it equals, and can change its hash (#1305). WordPieceTokenizer copies them at construction, so a tokenizer already built is unaffected.

Applies to — net10.0, netstandard2.0.

See also — WordPieceTokenizer, AddedToken.

Members

Member	What it does
WordPieceVocabulary.Equals	Value equality over the entries and settings.
WordPieceVocabulary.GetHashCode	A hash consistent with it.