Embeddings Lodestar.Embeddings - CyrilB1531/lodestar GitHub Wiki

Home › Embeddings

Lodestar.Embeddings

Everything around an embedding model except running it. Sub-word tokenizers loaded from the files a Hugging Face model ships (WordPiece, SentencePiece unigram, BPE and byte-level BPE, from tokenizer.json, vocab.txt or a SentencePiece model); batch encoding with padding, truncation and attention masks; mean pooling and L2 normalization; a SIMD k-nearest-neighbour EmbeddingIndex; and .npy reading and writing. It has no native dependency: running an ONNX encoder is Lodestar.Onnx's job.

Install

dotnet add package Lodestar.Embeddings

Example

using Lodestar.Embeddings.Tokenization;

var vocab = new Dictionary<string, int>(StringComparer.Ordinal)
{
    ["[UNK]"] = 0, ["token"] = 1, ["##ize"] = 2, ["text"] = 3,
};
var tokenizer = new WordPieceTokenizer(
    vocab, unkToken: "[UNK]", continuationPrefix: "##", maxCharsPerWord: 100, lowercase: true);

TokenizationResult encoded = tokenizer.Encode("tokenize");
int pieces = encoded.Tokens.Count;   // 2: "token", "##ize"

Parity

Token ids are replayed against Hugging Face tokenizers and sentencepiece on the vocabularies of real models; pooling and search against numpy. docs/equivalence.md maps each Python call to its C# counterpart, with every deliberate divergence.

Dependencies

A core package (decision 0003), built for net10.0 and netstandard2.0:

  • Lodestar.Abstractions 0.2.1 or later

Documentation