Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.7.0 — read its documentation.

Home › Embeddings › Tokenization

PrecompiledNormalizer.GetHashCode

A hash consistent with that equality.

public int GetHashCode()

Returns — int, derived from the charsmap bytes.

Example — equal normalizers hash alike.

using Lodestar.Embeddings.Tokenization;

// charsMap is the precompiled_charsmap blob from the model's own spiece.model.
byte[] charsMap = File.ReadAllBytes("spiece.model");

PrecompiledNormalizer first = PrecompiledNormalizer.FromCharsMap(charsMap);
PrecompiledNormalizer second = PrecompiledNormalizer.FromCharsMap(charsMap);

bool consistent = first.Equals(second) && first.GetHashCode() == second.GetHashCode();

Remarks — the same bytes that Equals compares feed the hash, which is the contract. A charsmap is hundreds of kilobytes, so this is not free — cache the normalizer rather than rebuilding it, which is what a vocabulary already does by holding one.

Applies to — net10.0, netstandard2.0.

See also — PrecompiledNormalizer.Equals.