Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.7.0 — read its documentation.

Home › Text › Vectorization

Hash a corpus into the fixed columns.

public CsrMatrix Transform(IEnumerable<string> documents)

Parameters — documents is the corpus to vectorize. Nothing is learned from it.

Returns — CsrMatrix, one row per document and exactly NumFeatures columns, normalized by HashingVectorizerOptions.Norm.

Exceptions — ArgumentNullException when documents is null. ArgumentException when documents holds a null document.

Example — the width is the option, not the corpus.

using Lodestar.Abstractions;
using Lodestar.Text.Vectorization;

var hv = new HashingVectorizer(new HashingVectorizerOptions { NumFeatures = 16 });

CsrMatrix first = hv.Transform(["the cat eats"]);
CsrMatrix second = hv.Transform(["an entirely different corpus about boats"]);

int width = first.ColumnCount;      // => 16
int alsoWidth = second.ColumnCount; // => 16

Remarks — no state carries between calls, so two corpora vectorized separately are directly comparable — the same term lands in the same column both times, and on another machine too. That is the property the count vectorizers cannot offer without saving and shipping a fitted model.

A bucket two terms reached with opposite signs sums to 0 and stays stored, as scikit-learn's sum_duplicates leaves it: a TfidfTransformer fitted on the matrix counts it in the column's document frequency, and CountVectorizerOptions.Binary turns it into 1 with the rest.

A row can hold negative values when AlternateSign is on, which is the default. That is not a defect: it is what makes two colliding terms tend to cancel rather than sum, and it means a CsrMatrix from here is the one place in this namespace where RowL1Norm's absolute values matter.

Applies to — net10.0, netstandard2.0.

See also — HashingVectorizer.FitTransform, HashingVectorizerOptions.