Home › Embeddings › Tokenization

PrecompiledNormalizer.Normalize

Apply the model's folding to one string.

public string Normalize(string text)

Parameters — text is the string to fold.

Returns — string, the text after the charsmap's replacements.

Example — full-width characters folded as the model expects.

using Lodestar.Embeddings.Tokenization;

// charsMap is the precompiled_charsmap blob from the model's own spiece.model.
byte[] charsMap = File.ReadAllBytes("spiece.model");

PrecompiledNormalizer normalizer = PrecompiledNormalizer.FromCharsMap(charsMap);

string folded = normalizer.Normalize("Ｈｅｌｌｏ");

Exceptions — EncoderFallbackException when the text holds a lone surrogate, which UTF-8 cannot carry (#1502). ArgumentNullException when text is null. ArgumentException when text encodes to more UTF-8 bytes than one array holds. InvalidDataException when the charsmap points at a replacement it does not itself contain. That is a defect in the model file rather than in the input, and it surfaces here because this is where the trie is walked.

Remarks — the replacements are the model's, not a standard's. Two models can fold the same input differently and both be right, because each was trained on its own folding — which is why this is a per-model artifact rather than a call to string.Normalize.

It runs before tokenization, so a piece in the vocabulary is spelled in normalized form, and looking one up with unnormalized text can fail to match.

A normalizer read by TokenizerJsonLoader applies the same charsmap as tokenizers does rather than as sentencepiece does: by grapheme, a cluster under six UTF-8 bytes replaced whole by the shortest rule matching its start. The two readings part on decomposed text — "pho\u031b\u0309" is phở from a .model and phơ from a tokenizer.json (#1260).

Applies to — net10.0, netstandard2.0.

See also — PrecompiledNormalizer.FromCharsMap, SentencePieceTokenizer.