Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.8.0 — read its documentation.

Home › Embeddings › Tokenization

SpecialTokenTemplate

Which tokens wrap a sequence, and what pads it — per model family.

public sealed record SpecialTokenTemplate

Properties — PrefixTokens go before the text and SuffixTokens after. PadToken fills the rectangle. SpecialTokenCount is how many of the budget they consume. Bert, Roberta, T5 and None are the four ready-made templates.

Example — BERT costs two tokens of the budget; none costs none.

using Lodestar.Embeddings.Tokenization;

int bert = SpecialTokenTemplate.Bert.SpecialTokenCount;  // => 2
int none = SpecialTokenTemplate.None.SpecialTokenCount;  // => 0

Remarks — the template must match the model, not the tokenizer family: a RoBERTa model and a BERT model can share a WordPiece-shaped vocabulary and still expect different wrappers. Getting it wrong produces a batch the model accepts and reads differently, which is the failure mode this whole namespace exists to make hard.

It must also match the vocabulary, and that is checked: constructing a BatchEncoder whose vocabulary lacks [CLS] throws rather than substituting the unknown token.

None is for models that want none — many sentence-transformer exports do their own wrapping — and for measuring what the text alone encodes to.

PrefixTokens and SuffixTokens are taken as given, not copied: a caller casting one back to its list and writing to it changes this template and what it equals, and can change its hash (#1305). BatchEncoder reads them at construction, so an encoder already built is unaffected.

Applies to — net10.0, netstandard2.0.

See also — BatchEncoder, EncodingOptions.

Members

Member	What it does
SpecialTokenTemplate.Equals	Value equality over the tokens.
SpecialTokenTemplate.GetHashCode	A hash consistent with it.