Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.7.0 — read its documentation.

Home › Text › Vectorization

Learn the vocabulary and frequencies, and weight the same corpus.

public CsrMatrix FitTransform(IEnumerable<string> documents)

Parameters — documents is the corpus, both learned from and weighted.

Returns — CsrMatrix, one row per document, weighted and normalized by TfidfOptions.Norm.

Exceptions — ArgumentNullException when documents is null. ArgumentException when documents holds a null document. InvalidOperationException when the corpus yields no term at all — every document empty or made only of stop words — when MinDf and MaxDf leave no term, or when MaxDf corresponds to fewer documents than MinDf: scikit-learn raises ValueError for each, and so there is no model of zero columns.

Example — the whole corpus in one call, which is the usual way in.

using Lodestar.Abstractions;
using Lodestar.Text.Vectorization;

string[] docs = ["the cat eats", "the dog eats", "the cat and the dog"];

CsrMatrix weighted = new TfidfVectorizer().FitTransform(docs);

int rows = weighted.RowCount;         // => 3
int columns = weighted.ColumnCount;   // => 5

Remarks — equivalent to Fit then Transform on the same corpus, in one enumeration rather than two. It is not equivalent to fitting one corpus and transforming another, and the difference is not cosmetic here: the document frequencies would come from the wrong corpus.

Applies to — net10.0, netstandard2.0.

See also — TfidfVectorizer.Fit, TfidfTransformer.FitTransform, CsrMatrix.