Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.7.0 — read its documentation.

Home › Embeddings › Tokenization

BatchEncoder

Strings in, a padded rectangular batch with an attention mask out — what a model actually wants.

public sealed class BatchEncoder

Constructor — BatchEncoder(ISubwordTokenizer tokenizer, EncodingOptions? options = null). The template's special tokens are resolved against the tokenizer's vocabulary here, so a mismatch is refused at construction rather than encoded into something the model misreads.

Properties — Options is the EncodingOptions it was built with.

Example — two texts of different lengths, padded to one rectangle.

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["[UNK]"] = 0, ["token"] = 1, ["##ize"] = 2, ["text"] = 3,
    ["[CLS]"] = 4, ["[SEP]"] = 5, ["[PAD]"] = 6,
};
var tokenizer = new WordPieceTokenizer(
    vocab, unkToken: "[UNK]", continuationPrefix: "##", maxCharsPerWord: 100, lowercase: true);

var encoder = new BatchEncoder(tokenizer, new EncodingOptions
{
    Template = SpecialTokenTemplate.Bert,
    MaxLength = 8,
});

EncodedBatch batch = encoder.EncodeBatch(["text", "tokenize text"]);

int rows = batch.Count;             // => 2
int width = batch.SequenceLength;   // => 5

Remarks — the batch is as wide as its longest sequence, and the shorter ones are padded to match. That is what makes it a rectangle a model can consume, and it is why EncodedBatch.Lengths exists: the padding has to be ignored downstream, and the attention mask is how the model is told to.

Asking for a template the vocabulary cannot satisfy throws at construction:

The tokenizer's vocabulary has no token '[CLS]', which the special-token template requires.

That is deliberate. The alternative — encoding without the token, or with the unknown token in its place — produces a batch the model accepts and misinterprets, which is the failure that costs a day.

Applies to — net10.0, netstandard2.0.

See also — EncodingOptions, SpecialTokenTemplate, EncodedBatch.

Members

Member	What it does
BatchEncoder.Encode	One string to ids, template applied.
BatchEncoder.EncodeBatch	Many strings to one padded rectangle.
BatchEncoder.EncodeAll	Every text as its own row of ids, unpadded.
BatchEncoder.Pad	A window of encoded rows, laid out as one rectangle.