Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.6.0 — read its documentation.

Home › Text › Vectorization

TfidfTransformer

Counts in, TF-IDF weights out — the equivalent of sklearn.feature_extraction.text.TfidfTransformer.

Takes a CsrMatrix of counts from anywhere and reweights it, so a term appearing in every document counts for little and one appearing in a few counts for a lot.

public sealed class TfidfTransformer

Constructor — TfidfTransformer(TfidfOptions? options = null), whose defaults are scikit-learn's.

Properties — Idf is the inverse document frequency learned per column, computed by fitting whether or not TfidfOptions.UseIdf is set; reading it before fitting throws InvalidOperationException.

Example — counts from a CountVectorizer, weighted afterwards.

using Lodestar.Abstractions;
using Lodestar.Text.Vectorization;

string[] docs = ["the cat eats", "the dog eats", "the cat and the dog"];
CsrMatrix counts = new CountVectorizer().FitTransform(docs);

CsrMatrix weighted = new TfidfTransformer().FitTransform(counts);

double rowLength = weighted.RowL2Norm(0);  // => 1

Remarks — this exists separately from TfidfVectorizer because counts do not have to come from text. A matrix built by hand, loaded from a file, or produced by another library can be weighted here, and that is the case the combined vectorizer cannot serve.

Where the counts do come from text, TfidfVectorizer is this and a CountVectorizer in one pass, and is the shorter path.

Applies to — net10.0, netstandard2.0.

See also — TfidfOptions, TfidfVectorizer, CsrMatrix, the Python equivalence table.

Members

Member	What it does
TfidfTransformer.Fit	Learn the document frequencies from a count matrix.
TfidfTransformer.FitTransform	Learn them and weight the same matrix.
TfidfTransformer.Transform	Weight a matrix using frequencies already learned.