Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.6.0 — read its documentation.

Home › Text

How much do two pieces of text have in common, when where it appears does not matter? Every type on this page answers that. They cut each input into q-grams, count them as a bag, and divide the grams the two share by something — and the something is the only place they disagree.

Two conventions run through the whole namespace, and knowing them saves reading every entry.

• A q-gram is a run of qval consecutive characters, and qval is 1 by default, which makes the grams individual characters. Repeats count: "apple" holds p twice, and a bag that holds it once shares only one of them. This is textdistance's reading, and the reason a caller who wants words rather than characters has to split and compare the pieces themselves.
• Every member takes a TextElement saying what counts as one character. The default, TextElement.Utf16Unit, is .NET's own unit and agrees with Python for every character in the Basic Multilingual Plane; outside it — emoji, rare ideographs — one character is two UTF-16 units and the two disagree on purpose. Pass TextElement.CodePoint for Python's answer. The reasoning is in decision 0001.

Comparing text position by position — how many edits turn one string into the other, whether two names are spelled alike — is a different question, answered by Lodestar.Text.Distances and not by anything here.

Every measure on this page divides |A∩B|, the grams the two bags share, by a denominator of its own. That is the entire difference between them, and it is enough to predict which will disagree with which.

Two consequences are worth having before you choose.

Jaccard and SorensenDice rank identically. Dice = 2·Jaccard / (1 + Jaccard), which rises with Jaccard over the whole of [0, 1], so sorting candidates by one produces the order the other would. Choosing between them changes the number a threshold has to be set against, never which match wins.

Tversky is the other four in disguise. α = β = 1 is Jaccard, α = β = 0.5 is SorensenDice, and the asymmetric settings are what the other four cannot express: α = 1, β = 0 charges only for what the first input holds alone, which asks "is A contained in B" rather than "do A and B agree".

flowchart TD
    A["Comparing two bags of grams"] --> B{"Are the two roughly<br/>the same length?"}
    B -->|yes| C["Any of them agree closely.<br/>Jaccard is the usual default"]
    B -->|no| D{"Is the shorter one supposed to be<br/>a fragment of the longer?"}
    D -->|yes, and a fragment<br/>should score full marks| E["Overlap"]
    D -->|yes, but the extra material<br/>should still cost something| F["Cosine"]
    D -->|no, the gap is real<br/>disagreement| G{"Do the two sides deserve<br/>the same penalty?"}
    G -->|yes| H["Jaccard, or SorensenDice<br/>for a gentler number"]
    G -->|no — one direction matters| I["Tversky, with α ≠ β"]

The five measures above compare two inputs exactly. A corpus of a million documents holds half a trillion pairs, and no exact measure survives that. The four types below trade exactness for a sketch that fits in a few hundred bytes and a lookup that never sees most pairs.

Set or bag is the choice that matters, and it is not a matter of taste. MinHash takes a minimum, which is idempotent, so a token repeated changes nothing; SimHash sums weights, so it does. Deduplicating near-identical documents wants the bag; comparing sets of tags or shingles wants the set.

Both are estimates, and both return candidates rather than answers — which is why Jaccard above stays the right call whenever the two inputs are in hand and small enough to compare directly.

Two empty inputs share nothing and disagree about nothing, and all five answer 1 — the reference libraries' choice, kept here so a ported comparison does not change at the boundary. One empty input against a non-empty one answers 0 everywhere except Tversky, whose weights can make the denominator vanish; its entry says when.

See also — distances for position-sensitive comparison, the Python equivalence table, and Lodestar.Fuzzy for scorers that do the token splitting for you.