Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.8.0 — read its documentation.

Home › Embeddings › Loading vocabularies

TokenizerJsonLoader.LoadWordPiece

Reads the WordPiece model a tokenizer.json declares.

public static WordPieceVocabulary LoadWordPiece(Stream source, ArtifactLoadOptions options = null)
public static WordPieceVocabulary LoadWordPiece(string path, ArtifactLoadOptions options = null)

Parameters — source is a readable stream, never disposed here; path is the file to read. options bounds what will be accepted and defaults to ArtifactLoadOptions's own defaults.

Returns — WordPieceVocabulary, with the continuation prefix and the unknown piece read from the file.

Exceptions — FileNotFoundException or IOException from the path overload, when the file is missing or cannot be read (#1504). ArgumentNullException for a null source or path. InvalidDataException when the file declares a different model, declares a pipeline this package does not reproduce, or exceeds a bound in options — the message names what was refused and why.

Example — a stock BERT tokenizer.json, all-MiniLM-L6-v2's for one.

using Lodestar.Embeddings.Persistence;
using Lodestar.Embeddings.Tokenization;

WordPieceVocabulary vocab = TokenizerJsonLoader.LoadWordPiece("tokenizer.json");
var template = new SpecialTokenTemplate(vocab.PrefixTokens, vocab.SuffixTokens, "[PAD]");

Remarks — A stock HuggingFace BERT tokenizer.json loads as BERT's BasicTokenizer. Its default BertNormalizer — text cleaned, CJK padded, accents stripped exactly when lowercasing — followed by a BertPreTokenizer sets BasicTokenization, the pipeline VocabTxtLoader.Load gives a vocab.txt. A BertPreTokenizer after any other normalizer is refused, and so is a BertNormalizer doing more than lowercasing ahead of a Whitespace pre-tokenizer: those steps are reproduced as one unit or not at all.

The post_processor — TemplateProcessing's single template, or BertProcessing and RobertaProcessing as cls $A sep — lands in PrefixTokens and SuffixTokens. The file's truncation and padding are accepted and not read: they are call settings, and EncodingOptions carries the length and the template instead, with a pad token the vocabulary does not.

Unlike VocabTxtLoader.Load, there is no lowercase parameter: the file says.

Applies to — net10.0, netstandard2.0.

See also — VocabTxtLoader.Load, TokenizerJsonLoader.LoadWordPieceAsync, TokenizerJsonLoader.