Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.6.0 — read its documentation.

Home › Text

ran, running and runs are three spellings of one idea, and an index that stores them as three terms cannot match a query that uses the fourth. A stemmer cuts each word down to a shared key so they collide on purpose.

Lodestar.Text.Stemming holds sixteen stemmers, one static class each, all with the same single method: a string in, a string out.

flowchart TD
    A["What language is the text?"] --> B["English"]
    A --> C["Arabic, Danish, Dutch, Finnish, French,<br/>German, Hungarian, Italian, Norwegian,<br/>Portuguese, Romanian, Russian, Spanish, Swedish"]
    B --> D{"Matching an existing<br/>Porter index?"}
    D -->|"no — this is a new index"| E["EnglishSnowballStemmer"]
    D -->|"yes"| F["PorterStemmer"]
    C --> G["the Snowball stemmer<br/>for that language"]

Language picks the stemmer, and for fifteen of the sixteen that is the whole decision. A stemmer is built from one language's suffix rules and has nothing sensible to say about another's: GermanSnowballStemmer.Stem applied to French returns something, and that something is noise.

English has two, and they are the same algorithm a generation apart. PorterStemmer.Stem is Martin Porter's 1980 original; EnglishSnowballStemmer.Stem is his own later revision, published as Snowball and universally called Porter2.

Porter2 is the one to reach for. It is Porter's own later revision, so where the two disagree it is the original being corrected. Over the 86 words both are pinned on, they disagree six times, and Porter2 trims less in five of the six:

generalization is the clearest of them: Porter trims all the way to gener, a fragment short enough that several unrelated words reach it. Porter2 stops at general, which still names something.

The reason to choose the original anyway is compatibility, not quality. An index built by Porter has to be queried by Porter, and a corpus already stemmed one way cannot be searched the other. That is the whole of the case for it.

using Lodestar.Text.Stemming;

string old = PorterStemmer.Stem("generalization");    // => gener
string current = EnglishSnowballStemmer.Stem("generalization");  // => general

A stem is a key, not a word. musico, musica, musicos and musicas all stem to music, and Spanish cantar stems to cant — which is not Spanish. The output is meant to be compared against other output, never shown to a reader.

This is what separates stemming from lemmatisation, which returns the dictionary form and needs a dictionary to do it. Nothing here carries one: these are rule engines, small and fast, and they are wrong on irregular words by construction. Lodestar ships no lemmatiser.

The consequence for a search index is that both sides must be stemmed by the same stemmer — the documents when they are indexed, the query when it arrives. Stem one and not the other and the keys never meet.

• Input is lowercased first. The algorithms are defined on lowercase, so Running and running give the same stem, and the result is always lowercase.
• A null word is refused, with ArgumentNullException. An empty string is not: it comes back empty.
• Each is a static class with no state, so all fifteen are safe to call from any number of threads at once.
• Each is checked word for word against a Python reference, and the corpora are in tests/oracles. Where a stem looks wrong, it is wrong in the same way the reference is. That reference is nltk for thirteen of the fourteen and snowballstemmer for Hungarian — decision 0006 has why.

• Python → C# equivalence — the nltk call each of these replaces.
• From string to vector — where a stemmer sits in a pipeline.
• decisions/0006 — the one place a stemmer here follows nltk over the published algorithm.