Home › Embeddings › Tokenization

BatchEncoder

Strings in, a padded rectangular batch with an attention mask out — what a model actually wants.

public sealed class BatchEncoder

Constructor — BatchEncoder(ISubwordTokenizer tokenizer, EncodingOptions? options = null). The template's special tokens are resolved against the tokenizer's vocabulary here, so a mismatch is refused at construction rather than encoded into something the model misreads. It throws ArgumentNullException when tokenizer is null, and ArgumentException when the vocabulary lacks one of the template's tokens, the template or one of its members is null, BatchSize or MaxLength is below 1, Truncation is not a declared strategy, or MaxLength leaves no room for the template's tokens.

Properties — Options is the EncodingOptions it was built with.

Example — two texts of different lengths, padded to one rectangle.

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["[UNK]"] = 0, ["token"] = 1, ["##ize"] = 2, ["text"] = 3,
    ["[CLS]"] = 4, ["[SEP]"] = 5, ["[PAD]"] = 6,
};
var tokenizer = new WordPieceTokenizer(
    vocab, unkToken: "[UNK]", continuationPrefix: "##", maxCharsPerWord: 100, lowercase: true);

var encoder = new BatchEncoder(tokenizer, new EncodingOptions
{
    Template = SpecialTokenTemplate.Bert,
    MaxLength = 8,
});

EncodedBatch batch = encoder.EncodeBatch(["text", "tokenize text"]);

int rows = batch.Count;             // => 2
int width = batch.SequenceLength;   // => 5

Remarks — the batch is as wide as its longest sequence, and the shorter ones are padded to match. That is what makes it a rectangle a model can consume, and it is why EncodedBatch.Lengths exists: the padding has to be ignored downstream, and the attention mask is how the model is told to.

Asking for a template the vocabulary cannot satisfy throws at construction:

The tokenizer's vocabulary has no token '[CLS]', which the special-token template requires.

That is deliberate. The alternative — encoding without the token, or with the unknown token in its place — produces a batch the model accepts and misinterprets, which is the failure that costs a day.

Applies to — net10.0, netstandard2.0.

See also — EncodingOptions, SpecialTokenTemplate, EncodedBatch.

Members

Member	What it does
BatchEncoder.Encode	One string to ids, template applied.
BatchEncoder.EncodeBatch	Many strings to one padded rectangle.
BatchEncoder.EncodeAll	Every text as its own row of ids, unpadded.
BatchEncoder.Pad	A window of encoded rows, laid out as one rectangle.