Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.8.0 — read its documentation.

Home › Embeddings › Tokenization

BatchEncoder.Encode

One string to ids, with the template applied.

public long[] Encode(string text)

Parameters — text is the string to encode.

Returns — long[], the ids with the template's prefix and suffix tokens in place. long because that is what ONNX models take.

Example — [CLS], the text, [SEP].

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["[UNK]"] = 0, ["token"] = 1, ["##ize"] = 2, ["text"] = 3,
    ["[CLS]"] = 4, ["[SEP]"] = 5, ["[PAD]"] = 6,
};
var tokenizer = new WordPieceTokenizer(
    vocab, unkToken: "[UNK]", continuationPrefix: "##", maxCharsPerWord: 100, lowercase: true);

var encoder = new BatchEncoder(tokenizer, new EncodingOptions
{
    Template = SpecialTokenTemplate.Bert,
});

long[] ids = encoder.Encode("text");

int length = ids.Length;  // => 3
long first = ids[0];      // => 4

Exceptions — ArgumentNullException when text is null. ArgumentException when the sequence exceeds EncodingOptions.MaxLength and Truncation is None. Measured: with MaxLength = 3 and TruncationStrategy.None a four-token text throws; the same text with TruncationStrategy.Right is cut and returns. ArgumentException also when the tokenizer refuses text: a SentencePiece model refuses a lone surrogate, as neither reference can be handed one (#1324). InvalidDataException when a SentencePiece model's precompiled charsmap points at a replacement it does not contain, which is found only when an encode walks the map.

Remarks — three ids for a one-token text: [CLS] is 4, text is 3, [SEP] is 5. The template is the difference between this and ISubwordTokenizer.Encode, which gives what the text is rather than what the model expects around it.

No padding happens here — there is nothing to pad to. Truncation does, when MaxLength is set and the text is longer, and the special tokens are counted against that budget rather than added on top of it.

Applies to — net10.0, netstandard2.0.

See also — BatchEncoder.EncodeBatch, SpecialTokenTemplate.