Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.7.0 — read its documentation.

Home › Text › Vectorization

CountVectorizer

Term counts over a vocabulary learned from the corpus — the equivalent of sklearn.feature_extraction.text.CountVectorizer.

Fitting learns which terms exist and fixes their column order; transforming counts them. Column 7 means the same term in every row, and GetFeatureNames can say which.

public sealed class CountVectorizer

Constructor — CountVectorizer(CountVectorizerOptions? options = null). The default options are scikit-learn's defaults, so new CountVectorizer() is CountVectorizer(). It throws ArgumentOutOfRangeException when MinDf or MaxDf is negative, not finite or a fraction above 1, and ArgumentException when NgramRange descends, the one range scikit-learn refuses too, or when Analyzer is not an AnalyzerKind member. The token pattern is checked after the document-frequency bounds and the n-gram range, where 0.7.0 compiled it, whatever the analyzer: a null one throws ArgumentNullException with the parameter named pattern, and one .NET refuses as written its RegexParseException, a leading (?u) included, ahead of the analyzer check — a plain ArgumentException on a runtime without that type, as 0.7.0 raised there (#1645, #1656).

Example — three documents, and the vocabulary they produce.

using Lodestar.Abstractions;
using Lodestar.Text.Vectorization;

string[] docs = ["the cat eats", "the dog eats", "the cat and the dog"];

var cv = new CountVectorizer();
CsrMatrix counts = cv.FitTransform(docs);

int features = counts.ColumnCount;  // => 5

Remarks — the vocabulary is sorted by code point, as scikit-learn's is, so the column order depends only on the terms and not on the order the documents arrived in. Two fits over the same corpus give the same matrix, and a corpus shuffled before fitting gives the same matrix too. Code point and not UTF-16 unit: 𠮟る follows ｶﾀｶﾅ, as it does in Python, although its leading surrogate sorts below U+FF76.

What decides the counts is CountVectorizerOptions rather than anything here — the token pattern that drops single-letter words, the stop words, the n-gram range. This class only applies them.

To weight rare terms above common ones, TfidfVectorizer is this followed by TfidfTransformer. To skip the vocabulary entirely, HashingVectorizer trades naming for not having to hold one.

Applies to — net10.0, netstandard2.0.

See also — CountVectorizerOptions, CsrMatrix, TfidfVectorizer, the vectorization guide.

Members

Member	What it does
CountVectorizer.Fit	Learn the vocabulary, and return the same instance.
CountVectorizer.FitTransform	Learn it and count the same corpus in one pass.
CountVectorizer.GetFeatureNames	The term each column stands for, in column order.
CountVectorizer.Load	Read a fitted vectorizer back.
CountVectorizer.LoadAsync	The same, without blocking.
CountVectorizer.Save	Write a fitted vectorizer out.
CountVectorizer.SaveAsync	The same, without blocking.
CountVectorizer.Transform	Count a corpus against the vocabulary already learned.