Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.7.0 — read its documentation.

Home › Embeddings › Loading vocabularies

VocabTxtLoader

Reads a BERT-style vocab.txt: one token per line, the id being the line number.

public static class VocabTxtLoader

Example — the route for a stock BERT checkpoint.

using Lodestar.Embeddings.Persistence;
using Lodestar.Embeddings.Tokenization;

WordPieceVocabulary vocab = VocabTxtLoader.Load("bert-base-uncased/vocab.txt", lowercase: true);
var tokenizer = new WordPieceTokenizer(vocab);

Remarks — this is the route for stock BERT, not a fallback. The vocabulary it returns has WordPieceVocabulary.BasicTokenization set, so the tokenizer runs BERT's BasicTokenizer ahead of WordPiece as transformers.BertTokenizer does: control characters dropped, CJK ideographs padded, each punctuation character a token of its own, and accents stripped when lowercase is set. café in an uncased checkpoint is cafe, not [UNK]. A HuggingFace BERT tokenizer.json declares the same steps as a BertPreTokenizer and a full BertNormalizer, which TokenizerJsonLoader.LoadWordPiece still refuses. Decision 0005 has why the route changed.

The format records nothing but the tokens, so everything else is a parameter — Load has the three that matter and why lowercase is the dangerous one.

Reference behaviour is transformers.BertTokenizer's vocabulary loading, including two quirks of the Python loop it reproduces deliberately; docs/equivalence.md's loader row names them.

Applies to — net10.0, netstandard2.0.

See also — TokenizerJsonLoader, ArtifactLoadOptions, the persistence index.

Members

Member	What it does
VocabTxtLoader.Load	Reads a vocab.txt into a WordPiece vocabulary.
VocabTxtLoader.LoadAsync	The same, asynchronously.