Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.7.0 — read its documentation.

Home › Text › Vectorization

Learn the vocabulary and count the same corpus, in one pass.

public CsrMatrix FitTransform(IEnumerable<string> documents)

Parameters — documents is the corpus, both learned from and counted.

Returns — CsrMatrix, one row per document and one column per learned term.

Exceptions — ArgumentNullException when documents is null. ArgumentException when documents holds a null document. InvalidOperationException when the corpus yields no term at all — every document empty or made only of stop words — when MinDf and MaxDf leave no term, or when MaxDf corresponds to fewer documents than MinDf: scikit-learn raises ValueError for each, and so there is no model of zero columns.

Example — the whole corpus at once.

using Lodestar.Abstractions;
using Lodestar.Text.Vectorization;

string[] docs = ["the cat eats", "the dog eats", "the cat and the dog"];

CsrMatrix counts = new CountVectorizer().FitTransform(docs);

int rows = counts.RowCount;         // => 3
int stored = counts.NonZeroCount;   // => 10

Remarks — equivalent to Fit then Transform on the same corpus, and not equivalent to fitting one corpus and transforming another. It exists because that is the common case and because doing it in one pass avoids enumerating the corpus twice — which matters when the corpus is a lazy sequence read from disk.

The fit is kept, so the vectorizer can go on to transform further corpora afterwards.

Applies to — net10.0, netstandard2.0.

See also — CountVectorizer.Fit, CountVectorizer.Transform, CsrMatrix.