Embeddings - CyrilB1531/lodestar GitHub Wiki
Development build. This page describes
main, not a released package. The latest published Lodestar.Embeddings is 0.7.0 — read its documentation.
Embeddings
Start here
Namespaces
- Loading vocabularies —
Lodestar.Embeddings.Persistence— A tokenizer is only correct if its vocabulary is the model's own. - Pooling —
Lodestar.Embeddings.Pooling— A transformer returns one vector per token. - Vector search —
Lodestar.Embeddings.Search— Embedding a corpus is the expensive half and it happens once. - Tokenization —
Lodestar.Embeddings.Tokenization— A transformer does not read text; it reads token ids.