Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.7.0 — read its documentation.

Home › Text › Vectorization

Learn the vocabulary and the document frequencies from a corpus.

public TfidfVectorizer Fit(IEnumerable<string> documents)

Parameters — documents is the corpus to learn from, enumerated once.

Returns — TfidfVectorizer, the same instance, so a call can be chained.

Exceptions — ArgumentNullException when documents is null. ArgumentException when documents holds a null document. InvalidOperationException when the corpus yields no term at all — every document empty or made only of stop words — when MinDf and MaxDf leave no term, or when MaxDf corresponds to fewer documents than MinDf: scikit-learn raises ValueError for each, and so there is no model of zero columns.

Example — fit on training documents, weight a later one with those frequencies.

using Lodestar.Abstractions;
using Lodestar.Text.Vectorization;

var tv = new TfidfVectorizer();
tv.Fit(["the cat eats", "the dog eats"]);

CsrMatrix weighted = tv.Transform(["the cat sleeps"]);
int columns = weighted.ColumnCount;  // => 4

Remarks — two things are learned here where CountVectorizer.Fit learns one: the vocabulary, and the document frequency of each term in it. Both come from this corpus, which is what makes fitting on training data the correct order — a term's rarity is a property of the corpus it was measured on, and measuring it on the test set leaks information that will not exist at prediction time.

sleeps was never seen and is dropped, exactly as it would be by a count vectorizer.

Applies to — net10.0, netstandard2.0.

See also — TfidfVectorizer.Transform, TfidfVectorizer.FitTransform.