performance - CyrilB1531/lodestar GitHub Wiki
One comparison per capability: this library against the library a reader would otherwise
reach for, with that library's version, the machine the pair ran on and the window the run
took. Both sides are checked to return the same answers before either is timed — the
agreement checks, the corpora and every command are
bench/README.md's
subject, and this page carries only what came out.
A before/after of this repository's own code is not here. It is an argument about one commit, and it belongs in the pull request that made it.
- A ratio above 1 means this library is faster, unless the table's own header says otherwise; each table states its direction.
- A named machine is not a shared runner. Every row below names the processor, the operating system and the runtime it ran on. Where a row comes from a hosted runner or a session container it says so, and is read as a ratio: such a machine's absolutes are not comparable between nights, or with another machine's.
- A cross-language row is processor time as well as wall time where the Python side can use more than one core, because a wall-clock lead over a multi-threaded reference is a different claim from a lead per core.
- Every baseline here is single-threaded unless the row says otherwise.
Levenshtein.Distance against rapidfuzz
Levenshtein.Distance against rapidfuzz
3.14.6 on Python 3.12, through the cross-language harness over bench/corpus/pairs.json — the same
committed corpus and the same ns/pair methodology on both sides. Method, corpora and which bucket
reaches which kernel:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores (BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime, AVX-512. Window: one harness run per side on 2026-09-14.
| Alphabet, length | rapidfuzz | Lodestar | |
|---|---|---|---|
| Latin, 128 | 994.2 ns | 536.4 ns | 1.85× C# faster |
| Latin, 512 | 8,711.5 ns | 7,535.7 ns | 1.16× C# faster |
| CJK, 128 | 1,671.8 ns | 1,245.1 ns | 1.34× C# faster |
| CJK, 512 | 14,632.2 ns | 11,393.5 ns | 1.28× C# faster |
Ahead on both alphabets at both lengths. The Latin buckets take Myers' blocked kernel two words at a time; the CJK buckets leave Latin-1 once their affixes are trimmed — 993 of 1,000 pairs at 128 and all 1,000 at 512 — and take the row-major kernel with the side table that carries symbols above U+00FF. Buckets of 8 and 64 do not reach the blocked kernel at all; at those lengths rapidfuzz pays per-call interop overhead and this package is further ahead.
Indel is len(a) + len(b) - 2·LCS, so what runs is
Lcs.SubsequenceLength — and that is also
what fuzz.ratio, every process.extract and every blocking deduplication pass runs. Same corpus,
same harness and the same rapidfuzz 3.14.6 as the table above.
Machine: the AMD Ryzen 7 8700G named above. Window: the harness's compare-indel, one run per side on 2026-09-13, the C#
side published once over three rounds; the range below is the three rounds' spread.
| Alphabet, length | rapidfuzz | Lodestar | |
|---|---|---|---|
| Latin, 128 | 258.2 ns | 229.3 – 230.7 ns | 1.12× C# faster |
| Latin, 512 | 2,712.8 ns | 3,085 – 3,103 ns | 1.14× Python faster |
| CJK, 128 | 1,127.8 ns | 776 – 817 ns | 1.38× C# faster |
| CJK, 512 | 9,423.5 ns | 5,222 – 5,277 ns | 1.79× C# faster |
Latin 512 is the one row behind, and what it pays there is the four-word group's stored carry and the equality table's clear; the CJK rows, which take a different route, are ahead by more the longer the input gets.
TfidfVectorizer against ML.NET 5.0.0's FeaturizeText
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, 2026-09-14, one run per class. The
class and its agreement check:
bench/README.md
section 15.
| Documents | Lodestar | FeaturizeText |
Ratio |
|---|---|---|---|
| 200 | 5.057 ms, 5.13 MB | 28.875 ms, 28.80 MB | 5.71 |
| 1,000 | 22.294 ms, 24.92 MB | 244.044 ms, 325.26 MB | 10.95 |
This is not like-for-like, and section 15 of bench/README.md counts the difference.
FeaturizeText adds character n-grams, so it produces 70,307 non-zero features at 200 documents
where this package stores 7,996, and 351,217 at 1,000 against 39,974. Per non-zero feature produced,
that is 0.63 μs against 0.41 μs at 200 documents, ML.NET ahead, and 0.56 μs against 0.69 μs at 1,000.
The claim is the sparse representation, not a faster kernel.
Bm25Index against LuceneSharp.Core 26.8.4415's
BM25Similarity, by Bm25Benchmarks: one query term, the top ten, over a seeded corpus of 500
terms and 40 tokens a document. Method:
bench/README.md
section 21.
Both sides return the same ten documents in the same order, checked at 1,000 and at 20,000
documents before either was timed. This package's K1 is 1.5 and Lucene's 1.2; with one term and
documents of equal length the ranking follows the term's frequency alone, whatever K1 is.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, 2026-09-14 — the query rows from the
run that measured both sides after #751
replaced Top's sort with a bounded heap, the text-to-ranking rows from the run before it, which
that change does not touch because the row is the vectorizer's.
| Documents | Row | Lodestar | LuceneSharp | Faster |
|---|---|---|---|---|
| 1,000 | query only | 1.223 μs, 424 B | 1.943 μs | Lodestar, 1.6× |
| 20,000 | query only | 15.166 μs, 424 B | 10.614 μs | Lucene, 1.4× |
| 1,000 | text to ranking | 9,280.8 μs, 21,362.8 KB | 4,451.2 μs, 1,346.0 KB | Lucene, 2.1× |
| 20,000 | text to ranking | 228,889 μs ± 23,916, 418,588.8 KB | 86,301 μs, 21,897.3 KB | Lucene, 2.7× |
The query splits by size. What is left at 20,000 documents is the pass over every document's score, 20,000 comparisons against the heap's root; Lucene never reads a document without the term, and skips postings blocks whose best score cannot enter its top ten, as its design has it. The query no longer allocates in proportion to the corpus.
The text-to-ranking row is the vectorizer's, not the index's. Each phase, split with a
Stopwatch on the same machine and corpus, lowest of several runs:
| Documents | CountVectorizer.FitTransform |
new Bm25Index(counts) |
Bm25Index.Score |
|---|---|---|---|
| 1,000 | 9.39 ms, 20.4 MB | 0.360 ms, 0.46 MB | 1.3 μs |
| 20,000 | 175.6 ms, 399.6 MB | 3.233 ms, 8.97 MB | 10.1 μs |
For a caller who already holds the matrix — one who has vectorized for something else, TF-IDF
or a classifier — building Bm25Index and answering q queries costs 0.36 ms + 1.22 μs·q at
1,000 documents against Lucene's 4.45 ms + 1.94 μs·q, cheaper at every q. At 20,000 documents
it is 3.23 ms + 15.2 μs·q against 86.3 ms + 10.6 μs·q, which crosses near 18,000 queries. From
text Lucene stays ahead from the first query. These crossings mix Stopwatch build figures with
BenchmarkDotNet query figures, so read them as orders of magnitude.
No .NET package publishes a BK-tree, so the alternative is the loop a caller would otherwise
write: BkTreeBenchmarks.LengthFilteredScan, a linear scan that skips any word whose length
already rules it out, then calls
Levenshtein.Distance on what survives.
Both arms materialise and sort the same shape of result — a List<BkTreeMatch> ordered by
distance ascending — so neither pays a cost the other is exempt from. 20,000 words per shape, 200
queries drawn from the corpus. Building the tree runs in [GlobalSetup], which BenchmarkDotNet
excludes from every measured iteration: the scan has no structure to build, so a reader weighing
the k = 1 win should price the build in separately.
python3 bench/corpus/generate_dictionary.py
dotnet run -c Release --project bench/Lodestar.Text.Benchmarks -- --filter '*BkTree*'Machine: 4-core Intel Xeon Processor @ 2.10GHz (BenchmarkDotNet's own header; the physical CPU
behind this virtualized host is not otherwise identified), Ubuntu 24.04.4 LTS, .NET SDK 10.0.111,
.NET 10.0.11 runtime — a hosted session container, not a dedicated benchmark machine, so these
ratios are directional rather than exact. Window: one BenchmarkDotNet run, default job,
2026-09-02, no other load on the container, 6 min 16 s across 16 benchmarks.
| Method | Radius | Shape | Mean | Ratio | RatioSD | Allocated | Alloc Ratio |
|---|---|---|---|---|---|---|---|
LengthFilteredScan |
1 | clustered | 153.01 ms | 1.00 | 0.02 | 27.25 KB | 1.00 |
TreeWithinDistance |
1 | clustered | 90.91 ms | 0.59 | 0.02 | 103.75 KB | 3.81 |
LengthFilteredScan |
1 | uniform | 161.09 ms | 1.00 | 0.02 | 23.86 KB | 1.00 |
TreeWithinDistance |
1 | uniform | 84.03 ms | 0.52 | 0.02 | 116.56 KB | 4.88 |
LengthFilteredScan |
2 | clustered | 223.35 ms | 1.00 | 0.01 | 103.44 KB | 1.00 |
TreeWithinDistance |
2 | clustered | 352.15 ms | 1.58 | 0.02 | 260.13 KB | 2.51 |
LengthFilteredScan |
2 | uniform | 234.14 ms | 1.00 | 0.03 | 54.65 KB | 1.00 |
TreeWithinDistance |
2 | uniform | 315.79 ms | 1.35 | 0.06 | 193.09 KB | 3.53 |
LengthFilteredScan |
3 | clustered | 279.42 ms | 1.00 | 0.02 | 949.90 KB | 1.00 |
TreeWithinDistance |
3 | clustered | 489.30 ms | 1.75 | 0.04 | 1366.63 KB | 1.44 |
LengthFilteredScan |
3 | uniform | 286.40 ms | 1.00 | 0.02 | 741.56 KB | 1.00 |
TreeWithinDistance |
3 | uniform | 476.27 ms | 1.66 | 0.07 | 1153.80 KB | 1.56 |
LengthFilteredScan |
4 | clustered | 327.14 ms | 1.00 | 0.03 | 5113.56 KB | 1.00 |
TreeWithinDistance |
4 | clustered | 569.75 ms | 1.74 | 0.06 | 7216.20 KB | 1.41 |
LengthFilteredScan |
4 | uniform | 326.21 ms | 1.00 | 0.01 | 5514.13 KB | 1.00 |
TreeWithinDistance |
4 | uniform | 584.62 ms | 1.79 | 0.07 | 7964.50 KB | 1.44 |
Ratio is Mean(TreeWithinDistance) / Mean(LengthFilteredScan), BenchmarkDotNet's own baseline
column: below 1 the tree is faster, above 1 it is slower than never having built it. Only the
k = 1 rows are below 1, and the gap widens with radius rather than closing — 0.52/0.59
(uniform/clustered) at k = 1 against 1.79/1.74 at k = 4.
Pruning is not the whole cost. An instrumented run in the same window counted raw distance
computations, tree over scan, at 0.33/0.34 (k = 1), 0.79/0.86, 0.92/0.95 and 0.96/0.93 (k = 4):
the tree does compute fewer distances at every radius. What that count cannot see is per-node
traversal — a Dictionary<int, Node> lookup, a Stack<Node> push and a call through the stored
Func<string, string, int> metric delegate, against the scan's one array read and one integer
subtraction through a static method the JIT inlines. That fixed cost is cheap at k = 1, where the
tree visits a third as many nodes as the scan has candidates, and dominates from k = 2, where it
visits most of the corpus one node at a time instead of one array element at a time.
The allocation column carries the same story: the tree starts at 3.81× to 4.88× the scan at
k = 1, where a Stack<Node> costs more than a 20-word hit list, and falls to 1.41× to 1.44×
by k = 4 as the shared result list comes to dominate both sides.
docs/guides/dictionary-lookup.md carries the reader-facing version: the
tree is worth using at k = 1, and a length-filtered scan is the better answer past it.
Levenshtein.Distance against Fastenshtein 1.0.12, Quickenshtein 1.5.1 and F23.StringSimilarity 7.0.1
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, 2026-09-14, one run per class. The
class and its agreement check:
bench/README.md
section 15.
| Length | Lodestar | Fastenshtein | Quickenshtein | F23.StringSimilarity |
|---|---|---|---|---|
| 8 | 15.71 ns, 0 B | 46.11 ns, 56 B | 48.27 ns, 0 B | 103.42 ns, 128 B |
| 64 | 178.57 ns, 0 B | 3,339.85 ns, 280 B | 727.63 ns, 0 B | 5,443.03 ns, 576 B |
| 512 | 7,097.59 ns, 0 B | 239,725.19 ns, 2,072 B | 19,810.70 ns, 0 B | 452,681.32 ns, 4,160 B |
Ahead of all three at every length: 2.8× to 4.1× against Quickenshtein, the closest, and 2.9× to 63.8× against the other two. Neither Lodestar nor Quickenshtein allocates.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, 2026-09-14, one run per class. The
class and its agreement check:
bench/README.md
section 15.
| Operation | Lodestar | FuzzySharp | Faster |
|---|---|---|---|
Fuzz.Ratio |
62.18 ns, 0 B | 129.69 ns, 80 B | Lodestar, 2.09× |
Fuzz.PartialRatio |
452.36 ns, 0 B | 5,767.01 ns, 160 B | Lodestar, 12.75× |
Fuzz.TokenSetRatio |
653.21 ns, 1,448 B | 1,159.41 ns, 1,944 B | Lodestar, 1.78× |
Fuzz.WRatio |
1,346.20 ns, 2,760 B | 2,938.48 ns, 3,128 B | Lodestar, 2.18× |
Ahead on all four, and allocating less on each.
Full method, and the check that both sides return the same ids:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0 runs of TokenizerIncumbentBenchmarks, default
job, on 2026-09-13, encoding all 5 000 documents of the corpus per operation.
Microsoft.ML.Tokenizers 2.0.0. spiece_30k.model is a unigram model.
| Model | Lodestar | Microsoft.ML.Tokenizers | Allocated, Lodestar | Allocated, Microsoft.ML.Tokenizers |
|---|---|---|---|---|
WordPieceTokenizer |
20.26 ms | 29.96 ms | 8.71 MB | 3.55 MB |
SentencePieceTokenizer |
26.21 ms | 29.51 ms | 5.44 MB | 3.09 MB |
Both encoders run ahead of the incumbent, and the ids agree with it over the whole corpus. Each allocates more than the incumbent does: what is left is the result lists, the normalized text and, for WordPiece, the lowercased copy — a matched token is the vocabulary's own string rather than a new one.
Both encoders find every piece starting at a position in one walk of a double-array trie, rather than probing a hash table once per candidate substring.
What the trie costs the loader, which decision 0004 makes the product. Measured with a
Stopwatch over the same two vocabularies on the same machine, not by BenchmarkDotNet:
| Constructor | First call | Warm |
|---|---|---|
new SentencePieceTokenizer(vocabulary) |
35.1 ms | 5.6–7.3 ms |
new WordPieceTokenizer(vocabulary) |
31.8 ms | 6.6–8.1 ms |
Construction is 6–7 ms warm and about 35 ms on the first call in a process, paid once per
tokenizer; where the first call's extra time goes was not measured apart. Reading
spiece_30k.model itself takes 14 ms on the same run.
Full method, and the check that both sides return the same ids:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, on 2026-09-14, of BpeBenchmarks,
BpeScalingBenchmarks and TokenizerIncumbentBenchmarks. Every document of the corpus is encoded
per operation, with tokenizer_30k_bpe.json's byte-level model. Microsoft.ML.Tokenizers 2.0.0.
| Row | Mean | Allocated |
|---|---|---|
BpeTokenizer |
58.33 ms | 28.47 MB |
SentencePieceTokenizer, the Unigram baseline |
17.34 ms | 5.43 MB |
CodeGenTokenizer, Microsoft.ML.Tokenizers |
148.73 ms | 59.08 MB |
Lodestar against it, TokenizerIncumbentBenchmarks
|
57.95 ms, 2.57× faster |
Byte-level BPE is 2.57× faster than the incumbent, and 3.36× the cost of this package's own unigram baseline, which is the cheaper model rather than the faster implementation. The ids agree with the incumbent's over the whole corpus. Decision 0005 was taken on these numbers.
The largest single step was a hash function. The merge ranks were a Dictionary<long, int>
keyed by the two ids packed into a long, and long.GetHashCode is the exclusive or of the two
halves, so every pair whose ids share an exclusive or collided — (1, 2) with (2, 1), (0, 3)
among them. An open-addressing table with a Fibonacci hash removed two thirds of what was left.
What compiling the split pattern costs. Matching it over the corpus takes 17 ms interpreted and
5 compiled. Code generation moves to the tokenizer's first encode, about 2 ms, measured with a
Stopwatch; building the tokenizer stays at 5 to 6 ms warm either way. The netstandard2.0 build
compiles it too, and that cost was not measured on .NET Framework.
One token with no split point, BpeScalingBenchmarks:
| Length | Mean | Allocated |
|---|---|---|
| 512 | 10.20 μs | 7.48 KB |
| 1,024 | 21.49 μs | 14.53 KB |
| 2,048 | 48.49 μs | 28.58 KB |
| 4,096 | 152.17 μs | 56.63 KB |
The step from 2,048 to 4,096 costs 3.1×, where the others cost about 2×. It is one step, not a
curve: a Stopwatch taken further measured each doubling from 4,096 to 32,768 characters at 2.0
to 2.2×.
VectorMath.Dot against TensorPrimitives, by vector width (issue #754)
tensor-primitives (bench/README.md section 14) on the machine below, pinned to one core, three
conditions interleaved over five runs of nine. Median of the five medians, in ms, over
10,000 × 384 floats. Ratios above 1 mean TensorPrimitives is faster.
| row |
Vector512 on |
Vector512 off |
AVX-512 off |
|---|---|---|---|
ours_dot_knn |
0.863 | 1.232 | 0.955 |
tp_dot_knn |
6.222 | 3.860 | 3.946 |
ours_cosine_knn |
1.803 | 2.055 | 1.814 |
tp_cosine_knn |
10.795 | 6.130 | 6.363 |
ours_one_sweep |
0.563 | 0.545 | 0.351 |
tp_one_sweep |
0.456 | 0.387 | 0.369 |
index_search |
3.734 | 3.787 | 3.692 |
| dot ratio | 0.14 | 0.32 | 0.24 |
| cosine ratio | 0.17 | 0.34 | 0.29 |
Our kernel is ahead on the kNN pattern at every width, 3–7×, and one long sweep is at parity.
Turning the 512-bit path off halves TensorPrimitives' time without changing the direction.
Decision 0004
records it against the hosted runner's opposite reading in
0004.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 16 logical and 8 physical cores, Ubuntu 26.04.1 LTS, .NET 10.0.12 runtime. 2026-09-16.
The nightly's compare-persistence of 2026-09-12 has Lodestar at 0.13–0.34× numpy on every
uncompressed index row. Most of that is not code on either side: it is whose allocator keeps a
freed 15–20 MB block's pages. glibc raises its mmap threshold after the first free of a block
that size, so a Harness.Measure loop of np.load refills warm pages; the .NET GC decommits a
dead large-object region, so every iteration of the C# loop page-faults its 15 MB again.
Conditions. AMD Ryzen 7 8700G, 8 cores / 16 threads, Ubuntu 26.04.1, .NET 10.0.12, numpy
2.5.3 on Python 3.12, a workstation shared with other sessions under the machine lock, one-minute
load average 2.2–4.1. The probes are a Harness.Measure-shaped Stopwatch loop, median of five
rounds, and are not BenchmarkDotNet; read their ratios.
Each side, with its allocator's retention turned off or on:
| row | default | retention flipped | flip |
|---|---|---|---|
np.load, BytesIO
|
1.073 ms | 7.276 ms |
MALLOC_MMAP_THRESHOLD_=65536, pages fresh |
np.load, file |
1.728 ms | 2.798 ms | the same |
EmbeddingIndex.Load, memory |
4.409 ms | 2.236 ms |
DOTNET_GCRetainVM=1, pages kept |
EmbeddingIndex.Load, MemoryStream
|
5.207 ms | 3.177 ms | the same |
EmbeddingIndex.Load, file |
6.484 ms | 4.348 ms | the same |
15.36 MB memcpy into a fresh float[]
|
4.531 ms | 1.239 ms | the same |
the save row's own new MemoryStream(capacity) plus one memcpy, no library code |
3.911 ms | 1.543 ms | the same |
-
numpy pays 6.8× on its load the moment its pages are fresh, and a
memcpyinto a freshfloat[]costs 3.7× one into kept pages. A caller who loads one index once pays those faults on both sides; only a loop of loads collects the difference, and the harness is a loop. -
The save rows charge the C# side for the harness's sink.
new MemoryStream(indexArtifact.Length)allocates and zeroes 20.6 MB inside the timed window: 3.911 ms of the 4.057 msembedding_index_savemeasures here, against 0.568 ms for the whole chunked base64 encode.io.BytesIO()grows on pages glibc kept. -
What remains with retention on both sides is the format's own work, which
decision 0001
already prices: a JSON scan of the 20.6 MB document with its 10 000 ids (0.581 ms), a base64
decode (0.760 ms into warm pages), and the finite scan numpy does not promise (0.235 ms).
embedding_index_ingest_npystays the like-for-like row.
Two levers inside that remainder were measured and not taken: a Vector512 exponent-bits
finite scan, 0.235 to 0.167 ms, is 1.5% of a load; decoding and scanning in L2-sized slices,
3.711 to 3.655 ms, is noise.
The artifact is base64 inside JSON, which spends eight bits to carry six, so it is about 1.33× the raw block. Deflate takes that back almost exactly — base64 is the one expansion a general-purpose coder undoes perfectly. The question is what the time costs.
Machine: Intel Core i7-4770S, .NET 10.0.10, a synthetic 4,000 × 384 index through the real Save
and Load. Window: warmed, median of 7, the five modes interleaved in one window, published
2026-08-20.
| bytes | × size | save | × save | load | × load | |
|---|---|---|---|---|---|---|
| plain | 8 231 006 | 1.000 | 10.2 ms | 1.00 | 7.9 ms | 1.00 |
gzip Fastest
|
6 257 079 | 0.760 | 270.8 ms | 26.67× | 56.5 ms | 7.19× |
gzip Optimal
|
6 151 764 | 0.747 | 382.1 ms | 37.62× | 46.6 ms | 5.92× |
brotli Fastest
|
6 074 449 | 0.738 | 37.4 ms | 3.68× | 40.8 ms | 5.19× |
brotli Optimal
|
6 069 780 | 0.737 | 122.1 ms | 12.02× | 38.3 ms | 4.87× |
The size claim holds exactly: 0.747 × 1.333 = 0.996 of the raw block. Level 9 gives the same
bytes as level 6, so there is nothing to tune, and deflate is dominated on all three axes —
brotli Fastest is smaller than gzip Optimal and writes it seven times faster.
Against numpy, compressed, from the nightly's own rows at 10,000 × 384 on a hosted runner — ratios only, per that page's warning, since a shared runner's absolutes are not comparable:
| operation | C# cpu | bytes | Python cpu | Python bytes |
|---|---|---|---|---|
embedding_index_save |
5.949 ms | 20 589 007 | 1.337 ms | 15 360 128 |
embedding_index_save_gzip |
456.995 ms | 15 251 458 | 638.992 ms | 14 022 374 |
embedding_index_load |
5.519 ms | 20 589 007 | 1.327 ms | 15 360 128 |
embedding_index_load_gzip |
81.774 ms | 15 251 458 | 72.368 ms | 14 022 374 |
-
Compressed, this package is ahead of numpy on the write and level with it on the read —
1.40× on
savez_compressed, 0.88× coming back. The deflate coder is the same on both sides, so what is compared is what each side hands it. - Compression closes most of the format gap. Uncompressed the artifact is 1.34× numpy's block, the expansion decision 0001 priced; compressed it is 1.09×.
-
The price grows with the artifact, which is the opposite of what would make it worth paying:
0.741× the size for 76.8× the save and 14.8× the load on the corpus, against 37.62× and 5.92× for
Optimalon the 8 MB synthetic index. The indexes big enough for 26 % of a disk to matter are the ones where compressing costs most.
dotnet run -c Release --project bench/Lodestar.Text.Benchmarks -- --filter '*BatchEmbedding*' --inProcessRead this before quoting the ratio. The model is tiny_embedder.onnx: one
Gather node over a 64 × 4 table, because weights are never committed
(CONTRIBUTING.md) and a real encoder is a hundred megabytes. Its arithmetic is
free. So what is measured is the per-sequence cost that batching removes — graph
dispatch, thread-pool wake-up, tensor wrapping — and none of the matrix
multiplication a real encoder adds to both sides. This is an upper bound on the
speed-up, not the speed-up.
Machine: Intel Core i7-4770S (Haswell, 4 physical cores), Ubuntu 24.04, .NET 10.0.10, X64 RyuJIT
AVX2. Window: four runs in one window — two per target, alternating — published 2026-08-06; the
run's own date was not recorded. Full job, [MemoryDiagnoser], InProcessEmitToolchain. Corpus
of 1 to 61 words per text, sub-batch 8. UnitLoop is the baseline — one Embed
call per text, which is what the guide's three lines amounted to before
EmbedBatch existed. Two runs, both shown, because BenchmarkDotNet's ±
describes dispersion inside one process and not reproducibility across processes.
| Texts | UnitLoop |
EmbedBatch |
ratio | EmbedBatchBucketed |
ratio | allocated vs baseline |
|---|---|---|---|---|---|---|
| 1 | 9.9 / 11.0 µs | 10.6 / 10.8 µs | 1.06 / 0.99 | 10.0 / 11.1 µs | 1.00 / 1.01 | 1.12 |
| 8 | 168 / 180 µs | 105 / 107 µs | 0.62 / 0.60 | 101 / 106 µs | 0.60 / 0.59 | 0.95 |
| 32 | 628 / 672 µs | 360 / 381 µs | 0.57 / 0.57 | 346 / 349 µs | 0.55 / 0.52 | 0.94 / 0.91 |
| 128 | 2 439 / 2 602 µs | 1 482 / 1 460 µs | 0.61 / 0.56 | 1 370 / 1 356 µs | 0.56 / 0.52 | 0.94 / 0.91 |
Batching removes about 40 % of the wall clock from 8 texts upward and stays there — 0.56–0.62 across every pairing — as the per-call overhead is amortized over the whole sub-batch. At a single text there is nothing to amortize and the two paths are a wash: 1.06 in one run, 0.99 in the next, which is a way of saying this benchmark cannot tell them apart there rather than that either wins.
Bucketing is a different story, and the honest answer is smaller. It engages only when the corpus spans more than one sub-batch, so the rows at 1 and 8 above run the identical code in both columns — they are the control, and what they differ by is this harness's noise floor: 1–3 % on seven of the eight control measurements taken here, with one outlier at 5.7 %. At 128 texts bucketing is ahead by 5.8–7.5 % in all four pairings, and ahead at 32 in all four as well. The sign is consistent where the magnitude alone would not be decisive. What is decisive is the allocation column, which is counted rather than sampled: 1 764 KB → 1 697 KB at 128 texts, in every run. That is padding genuinely not written. On a model doing real work that padding would be matrix multiplication not performed, and the time would follow; this model cannot show it, so the claim stops here.
The two builds are level here, and that is the structurally correct answer. The same benchmark
against the netstandard2.0 assemblies puts that side 1.4–5.7 % ahead of net10 in all four
pairings (355 / 365 µs against 360 / 381 at 32 texts; 1 418 / 1 419 against 1 482 / 1 460 at 128) —
inside the harness's noise, but consistent in direction. Pooling guards its Vector<T> branch
with accumulator.Length >= Vector<float>.Count, and tiny_embedder.onnx has a hidden size of 4
(EMBEDDING_DIM in tools/build_tiny_models.py) against Vector<float>.Count of 8 under AVX2, so
on net10 the guard is false and both builds run the same scalar tail loop. This benchmark cannot
resolve a difference between the targets, and reports that instead of a number. Where the vector
path does engage it is worth 4×–7×, in VectorMath over 384–1024 dimensions
(bench/README.md section 2).
The one difference it does resolve is counted rather than timed: the unit-loop path allocates 0.6 %
more on netstandard2.0 (1 887 KB against 1 875 at 128 texts), identically in both runs, while the
two batch paths allocate byte for byte the same on both targets.
dotnet run -c Release --project bench/Lodestar.NetStandard.Benchmarks -- --filter '*BatchEmbedding*'--inProcess on the first command and not on the second is the point. The netstandard2.0 project
pins InProcessEmitToolchain in its Program.cs — it has to, or BenchmarkDotNet's generated
project re-resolves the ProjectReference and silently restores the net10.0 build — so the flag is
what puts the net10 side on the same toolchain. Without it the two commands measure the same code
two different ways.
Conditions. The four runs behind the table were taken back to back in one window, alternating
net10 and netstandard2.0, with the one-minute load average between 5.1 and 5.9 on 8 logical cores
— the editor's language servers and the session driving the runs are part of that load and cannot
be excluded from inside it. Both columns pay it equally, so the table is internally comparable; it
is not comparable to figures taken on this machine in a quieter state, and the ratios travel
between such sets while the absolute microseconds do not.
python bench/corpus/generate_metrics.py # writes bench/corpus/metrics/, git-ignored
.venv-oracles/bin/activate && python bench/python/bench_metrics.py
dotnet run -c Release --project bench/Lodestar.Text.Benchmarks -- compare-metrics
python bench/compare.py metricsSix operations — confusion_matrix, accuracy, precision_recall_f1_macro,
classification_report, roc_auc_binary, roc_auc_ovr_macro — over six shapes
(1 000 / 100 000 / 1 000 000 samples, 2 or 10 classes), on the same corpus files
on both sides. This is the merge gate for the branch, on processor time: every
row must be ≥ 1×, and it is.
Machine: Intel Core i7-4770S, .NET 10.0.10, against scikit-learn 1.9.0 / NumPy 2.5.1 on Python
3.12.3. Window: one run per side, published 2026-08-06.
Both sides measured back to back, Python first,
on a machine left to settle (one-minute load 1.52 at the Python start — below
this workstation's 1.9–2.3 floor, itself a permanent ~30–40 % background from
the desktop client, an editor and a browser). The C# side started 49 seconds
later, in the Python run's own wake, so its figures are the ones taken on the
busier machine and every ratio below is conservative rather than flattering;
bench/README.md records the full conditions.
| Operation | Lodestar ms | Python ms | wall | Lodestar cpu ms | Python cpu ms | cpu |
|---|---|---|---|---|---|---|
confusion_matrix_n1000_k2 |
0.009 | 1.028 | 117.98x | 0.009 | 1.028 | 117.97x |
accuracy_n1000_k2 |
0.001 | 0.546 | 618.32x | 0.001 | 0.546 | 618.33x |
precision_recall_f1_macro_n1000_k2 |
0.008 | 1.793 | 226.58x | 0.008 | 1.793 | 226.58x |
classification_report_n1000_k2 |
0.011 | 6.692 | 623.31x | 0.011 | 6.691 | 623.25x |
roc_auc_binary_n1000_k2 |
0.029 | 2.008 | 70.12x | 0.029 | 2.008 | 70.12x |
confusion_matrix_n1000_k10 |
0.009 | 1.051 | 120.64x | 0.009 | 1.051 | 120.64x |
accuracy_n1000_k10 |
0.001 | 0.541 | 622.03x | 0.001 | 0.541 | 622.08x |
precision_recall_f1_macro_n1000_k10 |
0.010 | 1.855 | 192.49x | 0.010 | 1.855 | 192.48x |
classification_report_n1000_k10 |
0.017 | 7.011 | 422.54x | 0.017 | 7.010 | 422.53x |
roc_auc_ovr_macro_n1000_k10 |
0.550 | 10.526 | 19.13x | 0.550 | 10.525 | 19.13x |
confusion_matrix_n100000_k2 |
0.964 | 15.791 | 16.39x | 0.964 | 15.791 | 16.39x |
accuracy_n100000_k2 |
0.190 | 5.519 | 29.01x | 0.190 | 5.518 | 29.01x |
precision_recall_f1_macro_n100000_k2 |
0.844 | 17.786 | 21.07x | 0.844 | 17.785 | 21.07x |
classification_report_n100000_k2 |
0.848 | 36.233 | 42.75x | 0.847 | 36.231 | 42.75x |
roc_auc_binary_n100000_k2 |
7.977 | 35.024 | 4.39x | 8.092 | 35.023 | 4.33x |
confusion_matrix_n100000_k10 |
1.059 | 16.109 | 15.20x | 1.059 | 16.108 | 15.21x |
accuracy_n100000_k10 |
0.296 | 5.519 | 18.66x | 0.296 | 5.519 | 18.66x |
precision_recall_f1_macro_n100000_k10 |
0.979 | 18.524 | 18.92x | 0.979 | 18.523 | 18.92x |
classification_report_n100000_k10 |
0.979 | 40.139 | 41.00x | 0.979 | 40.137 | 41.00x |
roc_auc_ovr_macro_n100000_k10 |
88.385 | 250.400 | 2.83x | 91.396 | 250.402 | 2.74x |
confusion_matrix_n1000000_k2 |
8.750 | 156.920 | 17.93x | 8.749 | 156.823 | 17.92x |
accuracy_n1000000_k2 |
2.045 | 51.599 | 25.23x | 2.045 | 51.596 | 25.23x |
precision_recall_f1_macro_n1000000_k2 |
8.701 | 164.332 | 18.89x | 8.701 | 164.330 | 18.89x |
classification_report_n1000000_k2 |
8.719 | 314.805 | 36.11x | 8.718 | 314.782 | 36.11x |
roc_auc_binary_n1000000_k2 |
95.219 | 364.420 | 3.83x | 95.684 | 364.384 | 3.81x |
confusion_matrix_n1000000_k10 |
9.916 | 156.707 | 15.80x | 9.915 | 156.699 | 15.80x |
accuracy_n1000000_k10 |
3.122 | 51.877 | 16.61x | 3.122 | 51.874 | 16.61x |
precision_recall_f1_macro_n1000000_k10 |
10.001 | 173.128 | 17.31x | 10.000 | 173.121 | 17.31x |
classification_report_n1000000_k10 |
9.865 | 352.364 | 35.72x | 9.864 | 352.349 | 35.72x |
Gate result: 29/29 operations at or above 1× on processor time. The
narrowest margin is 2.74×, on roc_auc_ovr_macro at n=100 000, k=10 — the
row the design brief flagged as the one most likely to need a radix-sort
rewrite of BinaryRoc. It did not: even the heaviest sort-bound row clears the
gate by a comfortable margin, so no algorithmic change was needed on this
branch.
Read this before quoting a single ratio. The rows at n=1 000 (70×–620×) are dominated by CPython's per-call interpreter overhead, not by the computation — a confusion matrix over 1 000 samples is sub-microsecond work on either side. The rows that carry the argument are the ones at n=100 000 and n=1 000 000, where the ratios settle to a more modest but still decisive 2.7×–43×.
Unlike the persistence comparison, wall and processor time agree here to within about 1% on every row (up to 3.4% on the single heaviest-cpu row): these metrics allocate little enough per call that .NET's background collector is never a factor, so there is no gap between the two columns to explain away.
Full breakdown, including the intra-C# and net10-vs-netstandard2.0 tiers and
where the two language sides do not do identical work, in
bench/README.md.
Balanced accuracy, Matthews correlation and Cohen's kappa (issue #93, Tasks
3–5) add three operations — balanced_accuracy, matthews, cohen_kappa —
run over all six shapes above, unweighted and with default label handling on
both sides, matching scikit-learn's balanced_accuracy_score,
matthews_corrcoef and cohen_kappa_score. Same corpus files, same harnesses,
same methodology as the table above — but measured in a separate window
from the original 29 rows, with its own load: uptime's one-minute average
was 19.70 just before the Python side started and 7.65 by the time
compare.py printed the numbers below (fifteen-minute average 13.2–14.9
throughout that window). That is nowhere near the 1.52 one-minute load the
paragraph above states for the original run, so these 18 rows should not be
read as sharing that sentence's conditions — only their own, given here.
| Operation | Lodestar ms | Python ms | wall | Lodestar cpu ms | Python cpu ms | cpu |
|---|---|---|---|---|---|---|
balanced_accuracy_n1000_k2 |
0.016 | 1.194 | 76.44x | 0.011 | 1.194 | 105.24x |
matthews_n1000_k2 |
0.017 | 2.216 | 134.27x | 0.012 | 2.216 | 192.06x |
cohen_kappa_n1000_k2 |
0.018 | 1.240 | 67.89x | 0.012 | 1.240 | 105.23x |
balanced_accuracy_n1000_k10 |
0.008 | 1.225 | 152.93x | 0.008 | 1.225 | 152.96x |
matthews_n1000_k10 |
0.008 | 2.258 | 282.30x | 0.008 | 2.258 | 282.33x |
cohen_kappa_n1000_k10 |
0.009 | 1.399 | 157.84x | 0.009 | 1.399 | 157.84x |
balanced_accuracy_n100000_k2 |
0.887 | 17.287 | 19.49x | 0.887 | 17.282 | 19.48x |
matthews_n100000_k2 |
0.884 | 34.733 | 39.28x | 0.884 | 34.712 | 39.26x |
cohen_kappa_n100000_k2 |
0.880 | 18.133 | 20.61x | 0.880 | 18.103 | 20.58x |
balanced_accuracy_n100000_k10 |
1.001 | 17.326 | 17.31x | 1.001 | 17.320 | 17.31x |
matthews_n100000_k10 |
0.996 | 36.312 | 36.46x | 0.996 | 36.307 | 36.45x |
cohen_kappa_n100000_k10 |
0.980 | 17.130 | 17.49x | 0.979 | 17.129 | 17.49x |
balanced_accuracy_n1000000_k2 |
9.087 | 166.698 | 18.35x | 9.085 | 166.690 | 18.35x |
matthews_n1000000_k2 |
9.003 | 350.953 | 38.98x | 9.003 | 350.762 | 38.96x |
cohen_kappa_n1000000_k2 |
9.032 | 186.455 | 20.64x | 9.032 | 185.697 | 20.56x |
balanced_accuracy_n1000000_k10 |
10.103 | 167.552 | 16.58x | 10.102 | 167.550 | 16.59x |
matthews_n1000000_k10 |
10.262 | 340.992 | 33.23x | 10.261 | 340.854 | 33.22x |
cohen_kappa_n1000000_k10 |
10.352 | 174.623 | 16.87x | 10.352 | 174.619 | 16.87x |
18/18 at or above 1× on processor time — the gate holds for these three
metrics too. The two narrowest are balanced_accuracy_n1000000_k10 at
16.59× and cohen_kappa_n1000000_k10 at 16.87×; every other row
clears 17×. As with the original 29, the busier the machine gets, the more
conservative (not flattering) a ratio above 1× is — and this window's load
average was roughly 5–13× the original run's, so these margins are, if
anything, understated relative to a quiet machine.
The eleven regression metrics landed for issue #92 add four benchmark
operations — mse, mae, median_ae, r2 — covering the four distinct cost
shapes among them: a squared mean, an absolute mean, a sort, and a two-pass
centred sum. The other seven metrics are one of those four with a different
arithmetic kernel and are not separately timed. They run over
y_true_real/y_pred_real, continuous targets drawn by a separate seeded
random generator and attached to each of the six existing corpus shapes,
independent of the classification columns those shapes already carry. The
generator inserting these draws would otherwise have shifted every
classification array after the insertion point, invalidating the 29 and 18
rows above. A before/after comparison of y_true[:10] on the regenerated
corpus confirmed it did not. Same corpus files, same harnesses, same
methodology as the tables above — but measured in yet another separate
window, with its own load: uptime's one-minute average was 8.05 just
before the Python side started (five/fifteen-minute: 11.95 / 14.25) and
6.05 by the time compare.py printed the numbers below (five/fifteen-minute:
7.15 / 11.07). That is well below the 16–23 one-minute load this session saw
at dispatch and while the code changes were being made, but still noticeably
busier than the 1.52 one-minute load recorded for the original 29 rows. So
these 24 rows should be read only under their own conditions, given here —
except the six median_ae rows marked †. Those come from a later
window described below, after MedianAbsoluteError's unweighted path was
rewritten.
Read the k suffix as a corpus file name, not as a workload. The
regression arrays are drawn from SeededRandom(SEED + 1_000 + n), which
depends on the sample count and not on the class count, so metrics_n1000_k2
and metrics_n1000_k10 carry byte-identical y_true_real. That is deliberate
— all four operations here are single-output, and k is a property of the
classification columns those files also hold — but it means the 24 rows below
are 12 distinct workloads, each measured twice. The pairs are useful for
exactly that: they bound the run-to-run spread. At n=1 000 000 the two members
agree to within 0.04× (mse 1.04× / 1.00×), while at n=1 000 the same
identical array gives 98.88× and 141.17× — a 43 % spread, which is what a
sub-millisecond mse measurement is worth on a machine at this load, and the
reason no conclusion on this page rests on an n=1 000 row.
| Operation | Lodestar ms | Python ms | wall | Lodestar cpu ms | Python cpu ms | cpu |
|---|---|---|---|---|---|---|
mse_n1000_k2 |
0.005 | 0.486 | 104.89x | 0.005 | 0.458 | 98.88x |
mae_n1000_k2 |
0.005 | 0.358 | 77.79x | 0.005 | 0.358 | 77.70x |
median_ae_n1000_k2† |
0.011 | 0.818 | 77.81x | 0.011 | 0.625 | 59.45x |
r2_n1000_k2 |
0.008 | 0.443 | 57.72x | 0.008 | 0.442 | 57.66x |
mse_n1000_k10 |
0.005 | 1.003 | 219.23x | 0.005 | 0.646 | 141.17x |
mae_n1000_k10 |
0.005 | 0.541 | 119.33x | 0.005 | 0.507 | 111.80x |
median_ae_n1000_k10† |
0.011 | 0.367 | 34.83x | 0.011 | 0.367 | 34.84x |
r2_n1000_k10 |
0.008 | 0.447 | 55.95x | 0.008 | 0.447 | 55.86x |
mse_n100000_k2 |
0.452 | 0.645 | 1.43x | 0.452 | 0.645 | 1.43x |
mae_n100000_k2 |
0.466 | 1.588 | 3.41x | 0.466 | 1.295 | 2.78x |
median_ae_n100000_k2† |
1.967 | 1.781 | 0.91x | 2.045 | 1.781 | 0.87x |
r2_n100000_k2‡ |
0.759 | 0.991 | 1.31x | 0.759 | 0.991 | 1.31x |
mse_n100000_k10 |
0.455 | 0.628 | 1.38x | 0.454 | 0.628 | 1.38x |
mae_n100000_k10 |
0.458 | 0.673 | 1.47x | 0.458 | 0.672 | 1.47x |
median_ae_n100000_k10† |
2.142 | 1.796 | 0.84x | 2.241 | 1.795 | 0.80x |
r2_n100000_k10‡ |
0.743 | 0.950 | 1.28x | 0.743 | 0.950 | 1.28x |
mse_n1000000_k2 |
5.013 | 5.226 | 1.04x | 5.008 | 5.220 | 1.04x |
mae_n1000000_k2 |
5.054 | 5.635 | 1.12x | 5.036 | 5.633 | 1.12x |
median_ae_n1000000_k2† |
18.365 | 16.375 | 0.89x | 18.708 | 16.360 | 0.87x |
r2_n1000000_k2‡ |
8.093 | 9.205 | 1.14x | 8.083 | 9.204 | 1.14x |
mse_n1000000_k10 |
4.983 | 4.989 | 1.00x | 4.982 | 4.983 | 1.00x |
mae_n1000000_k10 |
5.040 | 5.712 | 1.13x | 5.035 | 5.711 | 1.13x |
median_ae_n1000000_k10† |
18.094 | 16.282 | 0.90x | 18.163 | 16.259 | 0.90x |
r2_n1000000_k10‡ |
7.807 | 9.687 | 1.24x | 7.807 | 9.686 | 1.24x |
† The six median_ae rows were measured after WeightedPercentile's unweighted branch stopped
sorting the whole residual array, in a separate window from the other eighteen rows and under a
deliberately comparable load (one-minute average 6.62 falling to 6.52, against the table's own 8.05
to 6.05). They are superseded: #140 took a
further 38 % off median_ae at n = 1 000 000, so these rows measure a partition this package no
longer ships and the current one has not been re-measured against scikit-learn.
‡ The four r2 rows predate
#127: they measure the original sequential-sum
R2, not the Neumaier-compensated one this package ships, and are kept only so the pairing this
table relies on stays intact.
20 of 24 rows at or above 1× on processor time, which under the pairing above is 10 of 12
distinct workloads. The four below the gate are median_ae at n = 100 000 and n = 1 000 000 —
0.80× to 0.90× — and the cause is algorithmic rather than a run: scikit-learn's
median_absolute_error calls NumPy's median, which selects by introselect in expected O(n).
This package now selects the one or two order statistics the median needs with a median-of-three
quickselect, falling back to Array.Sort on the remaining range once partitioning exceeds a budget
proportional to log2(n) — the same worst-case guarantee NumPy relies on — so the two do the same
order of work and what is left reads as constant overhead: managed bounds checks, the Lomuto
partition's extra writes, no SIMD comparison loop. mse_n1000000_k10 is the narrowest passing row
at 1.00×, near enough to parity that a busier or quieter machine could tip it either way.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, 2026-09-14, one run per class. The
class and its agreement check:
bench/README.md
section 15.
| Samples | Request | Lodestar | ML.NET | Faster |
|---|---|---|---|---|
| 100,000 | the six numbers ML.NET returns | 4,744.17 μs, 998 B | 22,914.88 μs, 5,089,338 B | Lodestar, 4.84× |
| 100,000 | accuracy alone | 69.09 μs, 0 B | 22,143.43 μs, 5,089,337 B | Lodestar, 321.81× |
| 1,000,000 | the six numbers ML.NET returns | 89,431.29 μs, 0 B | 140,625.56 μs, 23,229,318 B | Lodestar, 1.57× |
| 1,000,000 | accuracy alone | 1,661.99 μs, 0 B | 144,519.97 μs, 23,229,318 B | Lodestar, 86.96× |
Ahead on every row, by less as the sample grows: 4.84× at 100,000 samples to 1.57× at a million for the full bundle. ML.NET computes the bundle whatever is asked, so accuracy alone costs a caller the six.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, 2026-09-14, one run per class. The
class and its agreement check:
bench/README.md
section 15, and the decomposition class in
section 16.
A 2,000 × 500 term-document matrix at 2% density, rank 20:
| Row | Mean | Ratio |
|---|---|---|
TruncatedSvd, over the sparse matrix |
17.58 ms | 1.00 |
Nmf, capped at 50 iterations |
118.72 ms | 6.76 |
| ML.NET's centred PCA, over the dense twin | 14.85 ms | 0.85 |
ML.NET's PCA is 1.18× faster, and it is a different decomposition: centred and dense against
uncentred and sparse. Section 16 of bench/README.md says what is checked instead of agreement, and
decision 0004 why the gap that matters is the explained variance, which ML.NET does not report.
Full method and what the pair does and does not compare:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: one BenchmarkDotNet run, default job, on 2026-09-13, 8 benchmarks. NumFlat
1.3.4. Both sides were checked to return the same spectrum before either was timed.
| Shape | PrincipalComponentVariance |
NumFlat PrincipalComponentAnalysis
|
NumFlat / Lodestar | Allocated, Lodestar | Allocated, NumFlat |
|---|---|---|---|---|---|
| 200 × 10 | 14.74 μs | 15.35 μs | 1.04 | 2.38 KB | 1.94 KB |
| 2,000 × 10 | 82.53 μs | 111.83 μs | 1.36 | 2.38 KB | 1.94 KB |
| 2,000 × 50 | 1,977.65 μs | 1,634.81 μs | 0.83 | 42.07 KB | 40.06 KB |
| 100 × 200 | 7,834.02 μs | 9,306.55 μs | 1.19 | 318.27 KB | 628.54 KB |
This package is faster on three shapes of four and slower on one, while NumFlat's row also
computes the eigenvectors and the mean. The one it loses is the one where the eigen solve
dominates: 50 × 50 is where a one-sided Jacobi, several sweeps of O(p³), falls behind a
tridiagonal solver. At 10 columns the Gram matrix is most of the work, and centring each row into
a buffer of one row's width keeps it in cache.
The wide block is solved through the 100 × 100 Gram matrix of its rows rather than the 200 × 200 one of its columns, which is where the 1.19 and half the allocation come from.
The path to these numbers is in
decision 0003: a
Jacobi solve over the whole centred block measured 13× slower than NumFlat at 2,000 × 50 before
the Gram route replaced it. Below net8.0 the second row is Meta.Numerics: NumFlat does not install there and ML.NET
reports no eigenvalue, and its section
has the numbers.
Full method, the two classes and why only one is like-for-like:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: one BenchmarkDotNet 0.14.0 run, default job, on 2026-09-14, 15 benchmarks.
NumFlat 1.3.4, Meta.Numerics 4.2.0. Every pair was checked to return the same centres before either
side was timed; the largest difference was zero.
Lloyd's iterations from the same centres — the like-for-like pair:
| Shape (rows × features × k) | Iterations | KMeans.Fit |
NumFlat KMeans.Update
|
NumFlat / Lodestar | Allocated, Lodestar | Allocated, NumFlat |
|---|---|---|---|---|---|---|
| 10,000 × 2 × 8 | 101 | 19.21 ms | 70.34 ms | 3.66 | 98.92 KB | 96.40 KB |
| 10,000 × 16 × 16 | 4 | 6.51 ms | 9.87 ms | 1.52 | 88.70 KB | 13.58 KB |
| 50,000 × 8 × 32 | 8 | 64.02 ms | 127.26 ms | 1.99 | 410.24 KB | 36.39 KB |
This package is ahead on every shape, 1.52× to 3.66×, and allocates more on the two wider ones:
its current and previous labels are one int per row each, 80 KB at 10,000 rows, where NumFlat's
allocation grows with the iteration count instead — which is why the 101-iteration row is level.
A whole fit, k-means++ included, under each library's own stopping rule:
| Shape | KMeans.Fit |
NumFlat KMeans
|
Meta.Numerics MeansClustering
|
NumFlat / Lodestar | Meta.Numerics / Lodestar |
|---|---|---|---|---|---|
| 10,000 × 2 × 8 | 0.41 ms | 3.10 ms | 1.69 ms | 7.49 | 4.09 |
| 10,000 × 16 × 16 | 3.93 ms | 21.53 ms | 54.62 ms | 5.48 | 13.90 |
| 50,000 × 8 × 32 | 19.98 ms | 260.88 ms | 408.07 ms | 13.06 | 20.42 |
Read this table as what a caller pays, not as a kernel ratio. On these well-separated blobs scikit-learn's scaled tolerance stops this package after one or two iterations, and neither incumbent reports how many it ran. The second table is the one that compares arithmetic; this one says that a default fit here is 4× to 20× cheaper, and part of that is when it decides to stop.
Meta.Numerics is the comparison that holds below net8.0, and the only one: NumFlat does not
install there.
Full method, the two classes and what the agreement check refused:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET 10.0.12 runtime, AVX-512. Window: one
BenchmarkDotNet 0.14.0 run, --job short, on 2026-09-16, 10 benchmarks. NumFlat 1.3.4,
Dbscan 3.0.0. Every shape was checked to return the same partition — cluster numbering
ignored — on all the libraries in its table before any was timed.
Planar, where all three compete:
| Shape (rows × features × blobs) | radius | Dbscan.Fit |
NumFlat DbScan.Fit
|
Dbscan 3.0.0 |
NumFlat / Lodestar | Dbscan / Lodestar |
|---|---|---|---|---|---|---|
| 5,000 × 2 × 5 | 1.0 | 63.70 ms | 287.31 ms | 157.27 ms | 4.51 | 2.47 |
| 20,000 × 2 × 10 | 1.0 | 889.93 ms | 9,716.85 ms | 2,365.60 ms | 10.92 | 2.66 |
Above two features, where only NumFlat follows:
| Shape | radius | Dbscan.Fit |
NumFlat DbScan.Fit
|
NumFlat / Lodestar |
|---|---|---|---|---|
| 10,000 × 8 × 8 | 3.0 | 413.80 ms | 1,536.70 ms | 3.71 |
| 5,000 × 16 × 8 | 5.0 | 193.20 ms | 518.30 ms | 2.68 |
Ahead on every shape, 2.47× to 10.92×, and allocating 2.89× to 5.46× less: 13.98 MB against
NumFlat's 66.59 MB and Dbscan's 40.40 MB at 5,000 planar points, 111.97 MB against 531.88 MB and
372.11 MB at 20,000.
Dbscan 3.0.0 has no row in the second table because it has no entry point there. Its Point
carries X and Y and nothing else, which is
decision 0004's whole
argument for writing this class — the gap is reach below net8.0, and the speed is a second
finding rather than the claim.
One measured defect in an incumbent, and it is not ours. NumFlat 1.3.4 marks a non-core sample
noise permanently when the ascending scan reaches it before any cluster has been grown, where the
original algorithm and scikit-learn let a later expansion take it as a border sample. Nine points
in one dimension separate them, and at sixteen features with a radius of 3 it costs 1,017 rows of
5,000 — where this package matches scikit-learn exactly. bench/README.md section 48 has the
reproducer and the alternative explanation that was tested and refused.
Full method, what was measured before any code and why the benchmark runs on blobs:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET 10.0.12 runtime, AVX-512. Window: one
BenchmarkDotNet 0.14.0 run, --job short, on 2026-09-16, 16 benchmarks, with nothing else
running — a first run that shared the machine with the oracle generator was discarded. Aglomera
1.1.1. Every combination was checked to return the same merge heights in order, and the same
partition at the cut, before either side was timed.
| samples | linkage | AgglomerativeClustering.Fit |
Aglomera GetClustering
|
Aglomera / Lodestar | allocated, Lodestar | allocated, Aglomera |
|---|---|---|---|---|---|---|
| 500 | ward | 2.03 ms | 48.91 ms | 24.14 | 1.04 MB | 64.19 MB |
| 500 | complete | 1.98 ms | 61.70 ms | 31.21 | 1.04 MB | 83.11 MB |
| 500 | average | 1.92 ms | 56.28 ms | 29.37 | 1.04 MB | 59.88 MB |
| 500 | single | 0.45 ms | 87.45 ms | 194.69 | 0.06 MB | 81.29 MB |
| 1,500 | ward | 17.44 ms | 1,110.66 ms | 63.70 | 8.97 MB | 573.62 MB |
| 1,500 | complete | 16.98 ms | 1,209.18 ms | 71.23 | 8.97 MB | 747.36 MB |
| 1,500 | average | 17.28 ms | 1,224.95 ms | 70.91 | 8.97 MB | 537.15 MB |
| 1,500 | single | 3.53 ms | 2,081.80 ms | 589.97 | 0.18 MB | 732.69 MB |
Ahead on every row, 24× to 590×, and the gap widens with the sample count — the three
nearest-neighbour-chain linkages go from 24–31× at 500 to 64–71× at 1,500. Single linkage holds no
distance matrix at all, 185 KB where Aglomera allocates 733 MB.
These rows are the only ones where the comparison is fair. Blobs have no tied merge heights; on
data that does, Aglomera breaks ties the other way and can build a different tree, and its Ward
height is reported as d²/2. The speed is a second finding — the reason this class exists is that
Aglomera's answer is not scikit-learn's.
Full method and what cannot be made to agree:
bench/README.md.
Machine: the AMD Ryzen 7 8700G named above, on 2026-09-16. BenchmarkDotNet 0.14.0, default job, one run. Five folds,
three classes at 60 / 30 / 10.
ML.NET's split is lazy, so it appears twice: the call, and the call followed by reading which rows each fold holds. Only the second is a split a caller can fit on, and it is the one that scales with the data — the construction is flat in the row count because it builds ten wrappers and stops.
| rows | operation | Lodestar | ML.NET 5.0.0 | ML.NET / Lodestar |
|---|---|---|---|---|
| 10,000 | five folds, constructed | 52.52 μs | 79.19 μs | 1.51 |
| 10,000 | five folds, rows read | 52.52 μs | 3,330.43 μs | 63.44 |
| 10,000 | train/test, constructed | 3.11 μs | 15.70 μs | 5.05 |
| 10,000 | train/test, rows read | 3.11 μs | 592.81 μs | 190.55 |
| 100,000 | five folds, constructed | 695.48 μs | 78.35 μs | 0.11 |
| 100,000 | five folds, rows read | 695.48 μs | 25,527.38 μs | 36.71 |
| 100,000 | train/test, constructed | 56.81 μs | 16.30 μs | 0.29 |
| 100,000 | train/test, rows read | 56.81 μs | 4,454.57 μs | 78.42 |
Stratifying costs 136.77 μs at 10,000 rows and 1,489.37 μs at 100,000 — 2.6× and 2.1× the plain fold
cut. ML.NET has no row to compare it against: CrossValidationSplit and TrainTestSplit never
stratify (dotnet/machinelearning#4396, open
since 2019).
Allocation, per operation: 273.94 KB against ML.NET's 149.07 KB at 10,000 rows for the fold cut — this returns every index of every fold, where ML.NET returns views that allocate again on each read, so the two columns count different things and the larger one is the one holding the answer.
Against scikit-learn 1.9.0 through compare-splitters, one run of each side, milliseconds per
split, best of five:
| n | operation | Lodestar |
scikit-learn, wall / cpu |
ratio, wall |
|---|---|---|---|---|
| 10,000 | KFold |
0.050 ms | 0.057 / 0.057 ms | 1.14 |
| 10,000 | StratifiedKFold |
0.135 ms | 0.445 / 0.445 ms | 3.30 |
| 10,000 | train/test | 0.003 ms | 0.073 / 0.073 ms | 25.70 |
| 100,000 | KFold |
0.798 ms | 2.145 / 2.145 ms | 2.69 |
| 100,000 | StratifiedKFold |
1.657 ms | 5.718 / 5.717 ms | 3.45 |
| 100,000 | train/test | 0.149 ms | 0.194 / 0.194 ms | 1.30 |
| 1,000,000 | KFold |
12.273 ms | 14.919 / 14.917 ms | 1.22 |
| 1,000,000 | StratifiedKFold |
20.507 ms | 48.187 / 48.178 ms | 2.35 |
| 1,000,000 | train/test | 0.517 ms | 1.323 / 1.323 ms | 2.56 |
The million-row rows move about ±20% run to run on this machine — KFold there read 10.0 ms once
and 12.0 to 12.4 ms in the four runs after, on unchanged code. The ratios below 100,000 rows are
stable to the third decimal across every run.
TrainTest writes the two index halves straight out: an identity read comes out ascending already,
so neither half is sorted and no order array is filled.
Full method, and why fixZero: false is passed:
bench/README.md.
Machine: the AMD Ryzen 7 8700G named above, on 2026-09-16. BenchmarkDotNet 0.14.0, one invocation per iteration,
five warmups and twenty iterations. Ten features per row; ML.NET's estimator is lazy, so it appears both
as a fit and as a fit whose values are read back.
| rows | operation | Lodestar | ML.NET 5.0.0 | ML.NET / Lodestar |
|---|---|---|---|---|
| 1,000 | min-max, fit + transform | 68.91 μs | 225.28 μs (fit only) | 3.27 |
| 1,000 | min-max, values read | 68.91 μs | 1,208.41 μs | 17.54 |
| 1,000 | robust, values read | 1,189.85 μs | 3,521.85 μs | 2.96 |
| 20,000 | min-max, fit + transform | 1,106.62 μs | 3,614.47 μs (fit only) | 3.27 |
| 20,000 | min-max, values read | 1,106.62 μs | 9,674.44 μs | 8.74 |
| 20,000 | robust, values read | 10,430.85 μs | 15,245.13 μs | 1.46 |
MaxAbsScaler costs 65.21 μs and 1,116.54 μs at the two sizes and has no ML.NET row: its
normalizers offer mean-variance, min-max, log-mean-variance, robust scaling, binning and L_p norm,
and none of them is "divide by the largest absolute value" — NormalizeMinMax(fixZero: true) comes
closest and is a different transform.
Both sides scale the same column to the same values, checked row by row to 1e-6 before either was
timed — single precision, which is what ML.NET's pipeline carries.
Where the cost is. RobustScaler is about nine times MinMaxScaler here and allocates twice as
much: a percentile has to order its column, so it sorts each feature once where the other two take a
single pass. numpy.percentile partitions rather than sorting, which is the same asymptotic work
with smaller constants and is where to look if that row ever needs to be cheaper. Allocation, at
20,000 rows: 1,563.96 KB for min-max against ML.NET's 1,133.80 KB, and 3,126.26 KB for robust
against 4,821.33 KB.
Full method, and the shape that had to be corrected first:
bench/README.md.
Machine: the AMD Ryzen 7 8700G named above, on 2026-09-16. BenchmarkDotNet 0.14.0, one invocation per iteration,
five warmups and twenty iterations. Twenty categories; ML.NET's estimator is lazy, so it appears as a fit
and as a fit whose values are read back.
| rows | operation | Lodestar | ML.NET 5.0.0 | ML.NET / Lodestar |
|---|---|---|---|---|
| 1,000 | one-hot, fit + transform | 136.37 μs | 195.66 μs (fit only) | 1.43 |
| 1,000 | one-hot, values read | 136.37 μs | 728.92 μs | 5.35 |
| 1,000 | impute, values read | 84.09 μs | 650.84 μs | 7.74 |
| 20,000 | one-hot, fit + transform | 3,672.62 μs | 2,083.19 μs (fit only) | 0.57 |
| 20,000 | one-hot, values read | 3,672.62 μs | 5,384.35 μs | 1.47 |
| 20,000 | impute, values read | 925.39 μs | 4,670.33 μs | 5.05 |
Encoders.Ordinal costs 133.73 μs and 2,875.13 μs at the two sizes — about three quarters of the
one-hot encoding, at a tenth of the allocation (16.70 KB against 165.24 KB at 1,000 rows), since
it produces one column rather than one per category. ML.NET's MapValueToKey is its counterpart and
is not measured here: it maps to a key type inside the pipeline rather than to a number a caller
holds.
The correction, stated because the first table was wrong. OneHotEncoding takes one named
column where this package's encoders take a matrix of however many features, so the first run
encoded four features here against one there — four times the work for the same row, reported as
2.8× slower at 20,000. One column on both sides is the comparison above. ReplaceMissingValues
does take a vector column, so the imputer rows compare four features against four.
Allocation is where the two differ most on the read: 165.24 KB against 413.76 KB at 1,000 rows,
and 3,282.43 KB against 1,212.64 KB at 20,000 — this package materialises every encoded column as a
double, where ML.NET's cursor yields rows one at a time and never holds the matrix.
Full method, correctness cross-check, and how Accord's 2017-era API names were resolved against
the restored package:
bench/README.md.
This section carries only the numbers, per the rule for where a fact belongs
(CLAUDE.md's "Where a fact belongs" table).
Machine: Intel Xeon Processor 2.80GHz, 1 CPU, 4 logical and 4 physical cores (BenchmarkDotNet's own
header), Ubuntu 24.04.4 LTS, .NET SDK 10.0.111, .NET 10.0.11 runtime — a hosted session container,
not a dedicated benchmark machine, so this row is indicative, not authoritative, the same
caveat every other container row in this document carries;
docs/guides/performance.md records a
case where a container read a full 3× slower than the dedicated machine on the same code, so treat
the ratios below as directional rather than exact. Window: one BenchmarkDotNet run, ShortRun
job — fewer iterations than the default, exact parameters in bench/README.md — 2026-09-05, no
other load on the container during the run; total run time 1 min 51 s across the 12 benchmarks
(6 pairs × 2 sample sizes). The short job matters for reading the table: three iterations is enough
to see which side is faster by an order of magnitude, as every row below is, and not enough to
trust the last digit of a ratio.
| Method | SampleSize | Mean | Allocated |
|---|---|---|---|
LodestarWelchT |
100 | 1.322 μs | — |
AccordWelchT |
100 | 40.10 μs | 392 B |
LodestarMannWhitney |
100 | 11.52 μs | 8,944 B |
AccordMannWhitney |
100 | 58.71 μs | 23,336 B |
LodestarChiSquare |
100 | 380.0 ns | 200 B |
AccordChiSquare |
100 | 293.6 ns | 168 B |
LodestarWelchT |
10,000 | 52.21 μs | — |
AccordWelchT |
10,000 | 166.6 μs | 392 B |
LodestarMannWhitney |
10,000 | 5.078 ms | 880,312 B |
AccordMannWhitney |
10,000 | 14.67 ms | 2,241,217 B |
LodestarChiSquare |
10,000 | 379.7 ns | 200 B |
AccordChiSquare |
10,000 | 295.4 ns | 168 B |
Lodestar.Stats is faster on
TTest.Independent (30× at 100 samples, narrowing
to 3.2× at 10,000, as Accord's fixed per-call overhead is amortised over more work) and on
MannWhitney.Test (5.1× at 100, 2.9× at 10,000,
allocating 61-62% less at both sizes — both sides take the guarded asymptotic path at 10,000, past
MannWhitney's own 20_000-product exact-method bound). Those two MannWhitney rows predate
#711, after which the ranking no longer
allocates the pooled arrays; that allocation column measures a path this package no longer ships
and has not been re-measured against Accord. Accord is faster on
ChiSquare.Contingency (roughly 380 ns against 294 ns, flat with
sample size since a 2×2 table has four cells regardless of how many observations produced it) — the
one family where this package's richer result (Chi2ContingencyResult carries the expected-value
table; Accord's ChiSquareTest does not expose one) costs more than it buys at this shape.
The chi-square row has since reversed, on a different machine. The tail reads an integer or
half-integer degree of freedom up to 100 as a finite sum rather than iterating the incomplete
gamma's continued fraction, and
ChiSquare.Contingency reads 66.52 ns at 100 samples and
63.42 ns at 10,000 against Accord's 121.7 ns at both, at 168 B on both sides — AMD Ryzen 7
8700G, Ubuntu 26.04.1 LTS, .NET 10.0.12, BenchmarkDotNet 0.14.0 default job, 2026-09-13,
StatsBenchmarks and DistributionTailBenchmarks. Correctness against scipy is unchanged or
tighter: 60,001 erfc points over [-6, 27.5] agree to a relative 7e-15, and 22,500 chi2.sf
points at 1 to 250 degrees of freedom to 1.7e-13. The two machines are not comparable cell by
cell; what is comparable is each run's own pair.
Correctness, not just speed. All three families were checked against scipy on frozen
tests/oracles/stats_*.json corpus cases through both implementations; no case disagreed beyond
floating-point noise (the last one or two digits of a double, inside the 1e-9 tolerance
docs/equivalence.md already uses). bench/README.md has the three cases and the exact figures.
Full method, and the six p-values that differ with their causes:
bench/README.md.
Machine: the AMD Ryzen 7 8700G named above, on 2026-09-16. BenchmarkDotNet 0.14.0, default job, one run.
Meta.Numerics 4.2.0, MS-PL, netstandard2.0.
Eight test families. Ratio above 1 means this package is faster; every statistic was checked to agree before anything was timed.
| family | n = 100 | n = 10,000 | allocated, Lodestar / Meta.Numerics (n = 10,000) |
|---|---|---|---|
| Student t, pooled | 2.88 | 4.81 | 0 B / 104 B |
| Mann-Whitney | 1.47 | 5.25 | 0 B / 80,379 B |
| Kruskal-Wallis | 1.32 | 3.44 | 33 B / 120,702 B |
| Kolmogorov-Smirnov | 1.47 | 1.48 | 160,051 B / 80,371 B |
| One-way ANOVA | 3.46 | 3.21 | 32 B / 744 B |
| Wilcoxon signed rank | 0.95 | 1.18 | 80,056 B / 80,176 B |
| Fisher exact | 3.49 | 3.45 | 0 B / 944 B |
| χ² contingency | 5.95 | 5.94 | 168 B / 968 B |
The Kolmogorov-Smirnov row at n = 10,000 was re-measured on 2026-09-16 after #802 made Auto exact
there; it read 1.16 when Auto was asymptotic. The other rows are the original run's.
Two rows were costs in this package rather than differences in what the two libraries compute, and this comparison is what found them. Both are now ahead of Meta.Numerics:
| row | Lodestar | against Meta.Numerics |
|---|---|---|
FisherExact.Test, 2×2 table |
256 ns | 3.49× |
KolmogorovSmirnov.TwoSample, n = m = 100 |
1,544 ns | 1.47× |
- Fisher walked the hypergeometric probabilities through nine log-gammas per candidate table — three per binomial coefficient — with the denominator recomputed every iteration. Neighbouring probabilities differ by a ratio of four small integers, so the range now costs one exponential and O(range) multiplications, anchored at the mode. Still zero allocation.
-
Kolmogorov-Smirnov built an
(n+1)×(m+1)table even when the samples are the same size, whereDis always a whole number of steps of1/nand Hodges' exceedance probability is a closed form: O(n) multiplications against O(n·m) cells and n+1 row allocations. Allocation fell from 86,512 B to 1,610 B.
Neither is an approximation, and no default moved. ExactMethod.Auto chooses exactly what it
chose before; the exact branch is simply cheaper. The corpus gained an equal-size pair of 100 and a
100-against-99 pair — the second of which must take the table rather than the closed form — and
both replay scipy 1.18.1 at 1e-9. This package still returns the exact p-value where
Meta.Numerics returns an asymptotic one, and is now faster doing it.
The explained variance. Ratio above 1 means this package is faster.
| shape | PrincipalComponentVariance.Compute |
Meta.Numerics PrincipalComponentAnalysis
|
ratio | allocated |
|---|---|---|---|---|
| 200 × 10 | 15.42 μs | 565.95 μs | 36.70 | 2.38 KB / 329.71 KB |
| 2,000 × 10 | 86.25 μs | 69,772.02 μs | 808.97 | 2.38 KB / 31,414.98 KB |
| 2,000 × 50 | 1,953.85 μs | 337,025.46 μs | 172.50 | 42.07 KB / 32,054.48 KB |
Both report the same first component's variance fraction, checked to 1e-9 relative before either
was timed. Meta.Numerics refuses the fourth shape: 100 rows by 200 features raises
InsufficientDataException, where this package and NumFlat both answer — so the wide matrix has no
Meta.Numerics row at all rather than a slow one.
Full method, what the pair does and does not compare, and why MathNet.Numerics is not a candidate
here:
bench/README.md.
This section carries only the numbers, per the rule for where a fact belongs
(CLAUDE.md's "Where a fact belongs" table).
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.112, .NET 10.0.12 runtime — a
developer workstation shared with other checkouts of this repository, so this row is indicative,
not authoritative. Window: one BenchmarkDotNet run, ShortRun job (IterationCount=3,
WarmupCount=3, LaunchCount=1), 2026-09-10; run time 28.65 s across the 4 benchmarks
(2 pairs × 2 sample sizes). Four regressors throughout, on a seeded design.
| Method | SampleSize | Mean | Ratio | Allocated |
|---|---|---|---|---|
Lodestar_Ols |
100 | 49.75 μs | 1.00 | 68.84 KB |
Accord_Ols |
100 | 136.55 μs | 2.74 | 43 KB |
Lodestar_Ols |
10,000 | 3,700.63 μs | 1.00 | 6,490.19 KB |
Accord_Ols |
10,000 | 2,779.11 μs | 0.75 | 3,523.47 KB |
The two rows are not doing the same work, and that is the finding rather than a caveat.
Accord.Statistics exports no variance inflation factor anywhere in its 4 796 members — decision
0003's reading established that — so MultipleLinearRegressionAnalysis.Learn performs one solve.
OrdinaryLeastSquares.Fit
performs five at four regressors: the model, plus one auxiliary regression per regressor for
the VIFs, each with its own Householder QR over the full design.
Read that way the numbers are consistent. At 100 rows the fixed per-call overhead dominates and
this package is 2.7× faster despite doing five times the factorisation. At 10,000 rows the
O(mn²) work dominates instead, and doing it five times costs 1.33× Accord's single solve. The
allocation follows the same shape — 1.8× Accord's at both sizes — because each auxiliary
regression materialises a standardised copy of the design and its own Q.
So the honest summary is a shape rather than a winner: below roughly a thousand rows this package is faster and returns strictly more; above it, the extra diagnostic is what you are paying for. A caller who does not want VIFs has no way to say so today, and that is the obvious next measurement rather than a defect — the table is one call by design.
Full method, and why the untyped GeneralizedLinearRegression is the one driven:
bench/README.md.
Accord.Statistics 3.8.0 is archived. Its last package shipped on 2017-10-19 and the repository
was archived on 2020-11-18 (docs/migration/README.md). A lead over it is
a smaller claim than a lead over a maintained library, and a deficit against it is still a deficit.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime, AVX-512.
Window: one BenchmarkDotNet run, default job, on 2026-09-14, 8 benchmarks. A binomial model with
a logit link and an intercept, both sides given 100 iterations and a 1e-8 tolerance. The benchmark
project builds Lodestar.Stats 0.5.0 from source. Its normal quantile is inverted by Newton (#709)
on the erfc docs/guides/performance.md
describes. Lodestar.Stats.Regression's published floor, Lodestar.Stats 0.4.0, carries neither
yet.
| SampleSize | Regressors | Lodestar_Glm |
Accord_Glm |
Accord / Lodestar | Allocated, Lodestar | Allocated, Accord |
|---|---|---|---|---|---|---|
| 200 | 1 | 29.48 μs | 29.61 μs | 1.00 | 50.54 KB | 117.67 KB |
| 200 | 3 | 59.40 μs | 61.64 μs | 1.04 | 95.51 KB | 168.44 KB |
| 2,000 | 1 | 292.42 μs | 213.73 μs | 0.73 | 486.48 KB | 881.77 KB |
| 2,000 | 3 | 579.36 μs | 406.43 μs | 0.70 | 925.2 KB | 1,265.51 KB |
At 200 rows the two are level, and at 2,000 Accord is 1.4× faster. This package allocates less in every cell, by 1.4× to 2.3×.
The 200-row result moved, and one call moved it. On 2026-09-13, before the quantiles stopped
bisecting (issue #709), the same cells read
41.19 μs against Accord's 30.33 μs and 71.31 μs against 60.31 μs. The gap was 10.9 and 11.0 μs,
flat across the regressor count. That is the one
Distributions.NormalQuantile call the
intervals need, which Accord's GetWaldTest does not make, and it then cost about 11.5 μs. It now
costs 654 ns, and the two 200-row rows fell by 11.7 and 11.9 μs. The 2,000-row rows fell by 14.7 and
24.6 μs, where Accord's control rows moved by 3% to 4% the same way, so part of that is the window.
At 2,000 rows the gap is per iteration, not iteration count. The two stop on different
quantities: this package when the absolute change in deviance falls to the tolerance, statsmodels'
atol with rtol at zero; Accord's Run when the largest relative change in the coefficients
does. On these designs this package stops after 4 iterations and Accord after 4 to 6. So at
2,000 × 1, where both take 4, each of this package's iterations costs about 73 μs against Accord's
53 μs. This run does not attribute that difference. Each iteration here is a Householder
factorization of the weighted design, and reading its Q through IReadOnlyList<double> is one
cost issue #670 already names.
Both sides return the same fit, and the check that says so is not the timed call. Run once
outside BenchmarkDotNet on the same seeded designs:
| stopping rule | coefficients, relative | standard errors, relative | first p-values, relative |
|---|---|---|---|
the benchmark's 1e-8 on both sides |
1.4e-10 | 5.1e-7 | 8.8e-5 |
this package at 1e-13, Accord at 1e-15
|
1.4e-10 | 6.1e-11 | 4.3e-10 |
The coefficients agree at the oracle tolerance under either rule. The standard errors do not under the
benchmark's: each side computes its covariance from the IRLS weights of its own last iterate, and the
two last iterates differ because the stopping rules do. Tightened, every quantity agrees within 1e-9.
What this package computes beyond Accord's coefficients and standard errors — every interval, every
p-value, the deviance, the null deviance, the log-likelihood and the AIC — is in the timed call.
Full method and what agrees:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime, AVX-512.
Window: one BenchmarkDotNet 0.14.0 run, default job, on 2026-09-15. MathNet.Numerics 5.0.0, Accord.Statistics 3.8.0. Every pair returned the same slope before it
was timed.
| n | WeightedLeastSquares.Fit |
Math.NET | Math.NET / Lodestar | Allocated, Lodestar | Allocated, Math.NET |
|---|---|---|---|---|---|
| 200 | 6.650 μs | 9.865 μs | 1.48 | 3.35 KB | 59.08 KB |
| 2,000 | 54.601 μs | 54.718 μs | 1.00 | 17.41 KB | 509.88 KB |
| 20,000 | 0.591 ms | 0.923 ms | 1.56 | 158.41 KB | 5,033.56 KB |
| 200,000 | 5.718 ms | 11.072 ms | 1.94 | 1,564.58 KB | 50,035.09 KB |
Level at 2,000 rows and faster everywhere else, while computing the whole table — Math.NET returns the coefficients alone — and allocating 17× to 32× less.
| n | OrdinaryLeastSquares.Fit |
Math.NET | Accord | Math.NET / Lodestar | Allocated, Lodestar |
|---|---|---|---|---|---|
| 100 | 3.938 μs | 24.096 μs | 135.345 μs | 6.12 | 2.57 KB |
| 10,000 | 262.402 μs | 860.391 μs | 2,677.370 μs | 3.28 | 79.91 KB |
At 10,000 rows the robust covariances on top of that fit cost 0.628–0.631 ms for HC0 to HC3, and
6.27–6.31 μs at 100 rows (RobustCovarianceBenchmarks).
Full method and what agrees:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, on 2026-09-15, the run taken after GLS was
rebased on the shared least-squares pipeline of
#782.
MathNet.Numerics 5.0.0. Four regressors and an intercept, AR(1) errors at 0.6; each pair returned the same
slope within 1e-9 before it was timed.
| n | GeneralizedLeastSquares.Fit |
Math.NET, Cholesky and normal equations | Math.NET / Lodestar | Allocated, Lodestar | Allocated, Math.NET |
|---|---|---|---|---|---|
| 50 | 17.79 μs | 23.38 μs | 1.31 | 27.02 KB | 34.14 KB |
| 200 | 527.74 μs | 1,010.36 μs | 1.91 | 336.49 KB | 371.24 KB |
| 500 | 5.763 ms | 9.121 ms | 1.58 | 2,010.14 KB | 2,227.48 KB |
| 1,000 | 47.752 ms | 57.188 ms | 1.20 | 7,924.16 KB | 9,049.56 KB |
Faster than Math.NET at every size, while computing the whole table — standard errors, p-values, intervals, R², the F test and the VIFs — against a Math.NET path that stops at the coefficients. Math.NET exports no GLS, so its row is the one its users write: its Cholesky factor of the covariance, and the normal equations solved through it.
From 500 rows on, the covariance's Cholesky factor is the fit, n³/6 products, which is why the
ratio falls toward 1.20 as n grows rather than holding at the 1.9 it reaches at 200 rows.
Neither side is allocation-free, and neither can be: the factor is n² doubles, 8 MB at 1,000
rows, and each side holds one. Lodestar allocates 0.79× to 0.91× what Math.NET does.
Full method and what agrees:
bench/README.md.
No free .NET library fits this family at parity — Accord's IRLS is right only for canonical links, and commercial
libraries are not timed under a trial licence — so the incumbent is statsmodels 0.15.0, through the cross-language
harness. Machine: the AMD Ryzen 7 8700G named above, on 2026-09-15; the Python side is one run of
bench_stats.py on the same machine. Milliseconds per fit, best of five, α = 1, four regressors
and an intercept. The coefficients agree to 1.3e-14 relative or better with the same iteration
counts.
| n | GeneralizedLinearModel.Fit |
statsmodels, wall |
statsmodels, cpu |
statsmodels / Lodestar, wall |
|---|---|---|---|---|
| 1,000 | 0.404 ms | 1.437 ms | 1.437 ms | 3.56 |
| 10,000 | 4.136 ms | 5.604 ms | 5.603 ms | 1.35 |
| 100,000 | 45.520 ms | 74.138 ms | 1,174.297 ms | 1.63 |
The log-likelihood reads lnΓ(1/α) once rather than once per row — a few dozen logarithms per fit.
At 100,000 rows statsmodels reaches LAPACK through numpy's threads and spends sixteen times the
processor time it takes in wall clock; the wall column is still the comparison.
Full method and what agrees:
bench/README.md.
No free .NET library fits this family — Accord's generalized linear regression returned NaN coefficients on the
inverse link — so the incumbent is statsmodels 0.15.0, through the cross-language harness.
Machine: the AMD Ryzen 7 8700G named above. Window: BenchmarkDotNet 0.14.0 and the harness,
default job, on 2026-09-15; the Python side one run of bench_stats.py. Log link, four
regressors and an intercept; milliseconds per fit, best of five.
| n |
GeneralizedLinearModel.Fit, Gamma |
statsmodels, wall |
statsmodels, cpu |
statsmodels / Lodestar, wall |
|---|---|---|---|---|
| 1,000 | 0.371 ms | 1.795 ms | 1.794 ms | 4.84 |
| 10,000 | 3.397 ms | 6.634 ms | 6.633 ms | 1.95 |
| 100,000 | 43.016 ms | 90.541 ms | 1,429.046 ms | 2.10 |
The IRLS loop the other families share is unchanged by the Gamma link, and their classes sit
level with Accord or ahead of it: GlmBenchmarks' logistic fit reads 21.75 μs at 200 rows and one
regressor against Accord's 28.29 μs, 29.80 against 58.82 at three regressors, 211.34 against 209.32
at 2,000 rows and one regressor — level — and 287.29 against 398.97 at three. GlmPoissonBenchmarks
has no Accord counterpart and reads 191.5 to 197.6 μs across mean counts of 5 to 5,000,000, at
16.85 KB.
Full method and what agrees:
bench/README.md.
No free .NET library computes either covariance, so the incumbent is statsmodels 0.15.0 through the cross-language
harness. Machine: the AMD Ryzen 7 8700G named above, on 2026-09-15. The harness side is one run of compare-ols and one of
bench_stats.py. Four regressors and an intercept, four lags, clusters of 20 consecutive rows; milliseconds per fit,
best of five, each row including the VIFs on both sides.
| n | HAC, Lodestar | HAC, statsmodels wall / cpu |
ratio, wall | cluster, Lodestar | cluster, statsmodels wall / cpu |
ratio, wall |
|---|---|---|---|---|---|---|
| 1,000 | 0.097 ms | 1.508 / 1.507 ms | 15.61 | 0.035 ms | 1.537 / 1.537 ms | 43.37 |
| 10,000 | 0.995 ms | 6.683 / 6.683 ms | 6.72 | 0.352 ms | 6.892 / 6.891 ms | 19.56 |
| 100,000 | 10.646 ms | 74.696 / 1,189.086 ms | 7.02 | 4.326 ms | 76.800 / 1,223.598 ms | 17.75 |
What each covariance costs beside the ordinary fit, from HacClusterBenchmarks,
BenchmarkDotNet 0.14.0 default job:
| rows | Nonrobust | HC0 | HAC, 4 lags | Cluster |
|---|---|---|---|---|
| 100 | 3.94 μs, 2.59 KB | 5.49 μs, 8.58 KB | 10.92 μs, 12.73 KB | 4.81 μs, 9.54 KB |
| 10,000 | 259.9 μs, 79.93 KB | 556.3 μs, 472.72 KB | 1,212.5 μs, 863.67 KB | 489.1 μs, 565.00 KB |
HAC's filling runs once per lag, so four lags cost about five HC0 fillings.
HC0 and HC1 do not compute the leverages they never read. HC2 and HC3 do, and the two pairs sit
about 15 % apart at both sizes for that reason (RobustCovarianceBenchmarks: 5.52 and 5.60 μs
against 6.45 and 6.39 at 100 rows, 552 and 567 μs against 644 and 636 at 10,000).
Full method and what agrees:
bench/README.md.
Accord's GLM takes no offset, so the incumbent is statsmodels 0.15.0 through the cross-language
harness. Machine: the AMD Ryzen 7 8700G named above, on 2026-09-15; one run of compare-glm and one of bench_stats.py. A Poisson fit of the
corpus's counts, four regressors and an intercept, the exposure 1 + row % 3; milliseconds per fit, best of five.
| n | Lodestar |
statsmodels, wall / cpu |
ratio, wall |
|---|---|---|---|
| 1,000 | 0.287 ms | 1.659 / 1.658 ms | 5.79 |
| 10,000 | 3.299 ms | 6.546 / 6.545 ms | 1.98 |
| 100,000 | 45.520 ms | 70.576 / 1,099.885 ms | 1.55 |
What the exposure costs, from GlmOffsetBenchmarks, BenchmarkDotNet 0.14.0 default job:
| rows | Poisson | with an exposure | Allocated |
|---|---|---|---|
| 200 | 21.46 μs | 39.14 μs | 43.66 → 75.84 KB |
| 20,000 | 2,923.9 μs | 5,243.6 μs | 5,003.5 → 8,131.0 KB |
Most of the difference is the null deviance. With an offset it is a second, intercept-only IRLS fit, as the reference computes it; without one it is the deviance at the response mean.
The loop every fit shares is unaffected by the offset: GlmBenchmarks' logistic rows read
22.71 and 30.93 μs at 200 rows (one and three regressors) and 222.05 and 297.03 μs at 2,000, and
GlmPoissonBenchmarks 189.0 to 207.9 μs across mean counts of 5 to 5,000,000.
Full method and what agrees:
bench/README.md.
Machine: the AMD Ryzen 7 8700G named above, on 2026-09-16. BenchmarkDotNet 0.14.0, default job, one run. Three regressors
and an intercept,
three categories.
| rows | MultinomialLogit.Fit |
Accord LowerBoundNewtonRaphson, tolerance 1e-10
|
Accord / Lodestar | Allocated, Lodestar / Accord |
|---|---|---|---|---|
| 200 | 90.22 μs | 556.05 μs | 6.16 | 21.63 KB / 1,466.83 KB |
| 2,000 | 877.42 μs | 4,507.78 μs | 5.14 | 134.13 KB / 12,015.21 KB |
Accord's standard errors come from the lower-bound Hessian its algorithm iterates on, and are 33% to 43% from the ones
this fit and statsmodels report. The race compares coefficients only.
Against statsmodels 0.15.0 through compare-glm, one run of each side: the corpus's four regressors and an intercept,
the category the count modulo three, milliseconds per fit, best of five.
| n | Lodestar |
statsmodels, wall / cpu |
ratio, wall |
|---|---|---|---|
| 1,000 | 0.537 ms | 7.203 / 7.202 ms | 13.42 |
| 10,000 | 5.758 ms | 40.385 / 40.378 ms | 7.01 |
| 100,000 | 58.563 ms | 407.374 / 1,853.258 ms | 6.96 |
Part of the difference is the null log-likelihood: statsmodels refits the constant-only model with Nelder–Mead and
BFGS, where this fit takes the closed form that refit approximates (decision 0004).
Full method and what agrees:
bench/README.md.
Machine: AMD Ryzen 7 8700G w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
(BenchmarkDotNet's own header), Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 runtime,
AVX-512. Window: BenchmarkDotNet 0.14.0, default job, on 2026-09-15. Cortex.TimeSeries 1.1.0. Each
pair returned the same statistic before it was timed.
| n | function | Lodestar.Stats.TimeSeries |
Cortex.TimeSeries |
Cortex / Lodestar | Allocated, Lodestar | Allocated, Cortex |
|---|---|---|---|---|---|---|
| 200 | ADF, lag 4 | 6.987 μs | 15.819 μs | 2.26 | 20.68 KB | 16.18 KB |
| 200 | KPSS, level | 2.219 μs | 2.328 μs | 1.05 | 1.69 KB | 3.43 KB |
| 200 | decomposition, period 12 | 0.810 μs | 1.499 μs | 1.85 | 6.50 KB | 8.16 KB |
| 2,000 | ADF, lag 4 | 101.382 μs | 181.199 μs | 1.79 | 203.51 KB | 142.76 KB |
| 2,000 | KPSS, level | 59.554 μs | 60.223 μs | 1.01 | 15.75 KB | 31.55 KB |
| 2,000 | decomposition, period 12 | 8.253 μs | 15.391 μs | 1.86 | 62.75 KB | 78.48 KB |
The augmented Dickey-Fuller test is 1.79× to 2.26× faster than Cortex's. Its lag search fits
through
OrdinaryLeastSquares.Estimate,
which applies the reflections to the response and stops at the coefficients, their standard errors
and the residual sum of squares, rather than forming Q explicitly and pricing variance inflation
factors and a Student quantile per coefficient to hand back one t statistic. It allocates more than
Cortex at 2,000 points: the design and its column-major copy. The p-value stays what the test is
for — Cortex's is clamped at 0.01, where MacKinnon's surface gives 1.1e-9 to 9.5e-4 on the
regressions checked.
The decomposition is 1.85× faster: the moving average is a running
sum over each window's interior rather than a weighted pass over the whole window, O(n) against
O(n·period). KPSS is level, with half the allocation; its cost is the lagged products of the
long-run variance, which both sides compute.
Full method, and why the partial autocorrelations of the two libraries differ:
bench/README.md.
Machine: the AMD Ryzen 7 8700G named above. Window: one BenchmarkDotNet run, default job, on 2026-09-13,
12 benchmarks, 20 lags throughout.
| Method | SampleSize | Mean | Ratio | Allocated |
|---|---|---|---|---|
LodestarAutocorrelation |
200 | 18.582 μs | 1.00 | 1,000 B |
CortexAutocorrelation |
200 | 7.308 μs | 0.39 | 192 B |
LodestarPartialAutocorrelation |
200 | 18.829 μs | 1.01 | 1,192 B |
CortexPartialAutocorrelation |
200 | 7.696 μs | 0.41 | 768 B |
LodestarLjungBox |
200 | 8.407 μs | 0.45 | 720 B |
CortexLjungBox |
200 | 7.375 μs | 0.40 | 224 B |
LodestarAutocorrelation |
2,000 | 88.121 μs | 1.00 | 1,000 B |
CortexAutocorrelation |
2,000 | 76.709 μs | 0.87 | 192 B |
LodestarPartialAutocorrelation |
2,000 | 88.041 μs | 1.00 | 1,192 B |
CortexPartialAutocorrelation |
2,000 | 77.123 μs | 0.88 | 768 B |
LodestarLjungBox |
2,000 | 77.513 μs | 0.88 | 720 B |
CortexLjungBox |
2,000 | 76.915 μs | 0.87 | 224 B |
The gap is one function call. Lodestar's ACF and PACF are 11.3 and 11.1 μs slower at 200
points and 11.4 and 10.9 μs slower at 2,000 — constant while the series grows tenfold, which a
kernel difference could not be. Both compute the confidence band Cortex.TimeSeries does not,
through one NormalQuantile call, and the Ljung-Box pair, which has no band, is 1.14× at 200
points and level at 2,000. Subtracting the 11.5 μs quantile leaves 7.1 μs against Cortex's
7.3 at 200 points and 76.6 μs against 76.7 at 2,000. The allocation difference is the band and
the result record that carries it, and
#709 confirmed it by removing that cost:
without the bisection the pairs are level.
Full method and what agrees:
bench/README.md.
Nothing in .NET estimates a VAR, so the incumbent is statsmodels 0.15.0 through compare-var.
Machine: the AMD Ryzen 7 8700G named above, on 2026-09-16; one run of each side. A two-variable series at lag 2, milliseconds per fit, best of five.
| n | VectorAutoregression.Fit |
statsmodels, wall / cpu |
ratio, wall |
|---|---|---|---|
| 1,000 | 0.042 ms | 1.263 / 1.263 ms | 30.23 |
| 10,000 | 0.485 ms | 8.565 / 8.564 ms | 17.65 |
| 100,000 | 5.391 ms | 79.616 / 79.609 ms | 14.77 |
What the fit costs by shape, from VectorAutoregressionBenchmarks, BenchmarkDotNet 0.14.0 default job:
| observations | variables | lags | mean | Allocated |
|---|---|---|---|---|
| 500 | 2 | 1 | 14.76 μs | 60.53 KB |
| 500 | 2 | 4 | 49.28 μs | 132.84 KB |
| 500 | 5 | 1 | 72.24 μs | 206.76 KB |
| 500 | 5 | 4 | 422.41 μs | 590.05 KB |
| 5,000 | 2 | 1 | 244.97 μs | 587.95 KB |
| 5,000 | 2 | 4 | 829.14 μs | 1,293.21 KB |
| 5,000 | 5 | 1 | 911.36 μs | 2,000.01 KB |
| 5,000 | 5 | 4 | 4,701.02 μs | 5,548.08 KB |
Both axes multiply: a system of K variables at lag p fits K least squares over a design of 1 + K·p columns.
Measured 2026-09-10, on the one machine bench/README.md's GPU gate
says a GPU figure has to be published beside:
| GPU | NVIDIA GeForce RTX 5070 Ti, 15 877 MB, CUDA SM_120 |
| CPU | AMD Ryzen 7 8700G, 8 physical / 16 logical cores, AVX-512 |
| OS / runtime | Ubuntu 26.04.1 LTS, .NET SDK 10.0.401, .NET 10.0.12 |
| ILGPU | 1.5.3, CUDA backend |
| BenchmarkDotNet job |
--job short — 3 iterations, 3 warmups, 1 launch |
Read this before the numbers. Three caveats travel with every table below.
-
The baseline is single-threaded. bench/README.md's GPU gate asks a kernel to be priced against this
repository's own path, and none of those paths is parallel. A
Parallel.Forover eight cores would close a large part of every gain here, most of all Myers'. These are honest against the gate as written; they are not a claim about a parallel CPU implementation. -
--job shortis three iterations. TheGpuResidentrows sit at 0.7–7 % standard error, which carries a verdict. SomeGpuFromHostrows do not — one shows a confidence interval wider than its own mean — and those are read as orders of magnitude, not as figures. -
The first run of these benchmarks measured the wrong device, and the numbers below are the
second.
Context.GetPreferredDevice(preferCPU: false)returned an OpenCL runtime that executes on the processor — ILGPU enumeratedcpu-skylake-avx512-AMD Ryzen 7 8700Gas an OpenCL device beside the CUDA card — and a check on the accelerator type called it a GPU.GpuContextnow orders the devices itself and asks OpenCL for the device type, and the device's name is a column in every table so the mistake cannot repeat silently.
Tiled cosine + top-k, against EmbeddingIndex.Search
384 dimensions, top-10, seeded corpus. GpuResident is the gate row: query transfer and result
read-back are inside the measurement, the corpus is not.
| Documents | Queries | SimdBaseline |
GpuResident |
gain | GpuFromHost |
|---|---|---|---|---|---|
| 10 000 | 1 | 759.4 µs | 219.2 µs | 3.5× | 11.9 ms |
| 100 000 | 1 | 9.32 ms | 1.41 ms | 6.6× | 86.2 ms |
| 10 000 | 256 | 197.5 ms | 18.4 ms | 10.8× | 29.9 ms |
| 100 000 | 256 | 2 356.9 ms | 178.2 ms | 13.2× | 278.6 ms |
Clears the 5–10× gate at three sizes of four, and misses at the smallest corpus with a single query — which is where the crossing point belongs and why the sizes bracket it.
GpuFromHost is the argument for residency, and it is brutal. At 100 000 documents and one
query it is 9.25× slower than the CPU path: 147 MB of corpus uploaded to answer one question.
At 256 queries the same upload amortises and the row returns to 8.5× faster than the CPU. Below
roughly a hundred queries per corpus, a GPU package that uploads per call is a pessimisation.
Allocation is the secondary win: 434 KB against the SIMD path's 200 MB at the largest size, a ratio of 0.002, because the kernel returns k hits rather than a score per document.
Sparse-dense product, against CsrMatrix.Multiply
20 000 terms at 0.2 % density, double precision on both sides.
| Documents | Width | CpuBaseline |
GpuResident |
gain |
|---|---|---|---|---|
| 5 000 | 64 | 7.92 ms | 4.54 ms | 1.74× |
| 5 000 | 256 | 38.4 ms | 11.1 ms | 3.45× |
| 50 000 | 64 | 71.5 ms | 8.25 ms | 8.66× |
| 50 000 | 256 | 377.5 ms | 20.0 ms | 18.9× |
bench/README.md predicted this kernel might miss the gate on FP64, since a consumer card runs
double precision at a fraction of single. That was wrong, and the reason is worth keeping:
sparse-dense is bound by memory bandwidth, not by the double-precision units, so the reduced FP64
rate never becomes the constraint. It clears the gate at both realistic vectorizer sizes.
The cost it does carry is allocation — 5× the CPU path at the small sizes, from the staged upload and the downloaded result.
Myers bit-parallel edit distance, against Levenshtein.Distance
24-character pattern, 26-letter alphabet, seeded corpus.
| Texts | Text length | CpuBaseline |
GpuResident |
gain |
GpuFromHost, 2026-09-10 |
GpuFromHost, 2026-09-17 |
|---|---|---|---|---|---|---|
| 10 000 | 32 | 1 600.7 µs | 57.4 µs | 27.9× | 2 943 µs | 421 µs |
| 10 000 | 256 | 7 585.8 µs | 121.6 µs | 62.4× | 21 277 µs | 1.65 ms |
| 200 000 | 32 | 32.07 ms | 220.2 µs | 145.7× | 55.5 ms | 5.27 ms |
| 200 000 | 256 | 153.4 ms | 1.19 ms | 129.4× | 450.0 ms | 37.3 ms |
This was written expecting a failure and is the largest gain of the four. The reasoning behind the prediction — that Myers is bit-parallel on both sides, so the CPU already spends tens of nanoseconds per short pair — was true and did not settle it: the baseline is one thread and the kernel is tens of thousands, over pairs with no dependency between them. Caveat 1 above bites hardest here.
GpuFromHost runs 4.1× to 6.5× below the CPU path: renaming the batch is a 64 KB code table on
the host rather than a dictionary probe per character
(#853). That column was measured on 2026-09-17,
same machine and --job short, with a CpuBaseline of 1.73 ms, 8.08 ms, 34.4 ms and 162 ms in the
same window; the other columns are the 2026-09-10 run.
MinHash signatures, against MinHash.Signature
24 tokens per document over a vocabulary of 5 000, seeded corpus, the same coefficients on both sides. Three rows because the two sides divide the work differently: the CPU path hashes and minimises in one pass, while the kernel takes hashes the host already computed.
| Documents | Permutations | CpuBaseline |
GpuResident |
gain | GpuWithHashing |
gain |
|---|---|---|---|---|---|---|
| 5 000 | 64 | 63.2 ms | 0.95 ms | 66× | 49.6 ms | 1.27× |
| 5 000 | 128 | 79.8 ms | 1.94 ms | 41× | 50.8 ms | 1.57× |
| 50 000 | 64 | 637.1 ms | 15.1 ms | 42× | 503.2 ms | 1.27× |
| 50 000 | 128 | 779.6 ms | 23.2 ms | 34× | 537.4 ms | 1.45× |
GpuWithHashing is the row that matters, and it is the one that misses the gate. The
minimisation is 34× to 66× faster on the accelerator; a caller starting from tokens sees 1.27× to
1.57×, because hashing is most of the work and it stays on the host. This kernel clears the GPU gate on the part it took and misses it on the part a caller experiences.
Located rather than disappointing: the next move is to hash on the accelerator, which is a separate kernel because parity requires the first four bytes of SHA-1 little-endian and a device implementation agreeing bit for bit is its own piece of work. Until then the kernel earns its place for a caller who already holds hashes — sketching one corpus under several permutation sets, for instance, where hashing is paid once and minimisation many times.
Two sparse-dense products; RoundTripped is the baseline because the question is what an
intermediate costs when it crosses the bus, not whether the accelerator wins.
| Rows | Width | RoundTripped |
Chained |
gain | CpuBaseline |
|---|---|---|---|---|---|
| 2 000 | 32 | 1 234.7 µs | 552.6 µs | 2.23× | 1 795.9 µs |
| 2 000 | 128 | 5 372.1 µs | 1 830.1 µs | 2.94× | 8 325.9 µs |
| 20 000 | 32 | 1 990.5 µs | 1 602.4 µs | 1.24× | 6 324.3 µs |
| 20 000 | 128 | 6 288.9 µs | 3 490.4 µs | 1.80× | 25 519.8 µs |
bench/README.md predicted the gap would be roughly flat in Rows. It is not — it shrinks,
2.23× to 1.24× at width 32, and the arithmetic never predicted otherwise: a fixed transfer against
work that grows with Rows is a falling share of the total. The prediction in Width holds.
The reading for a caller inverts the usual intuition: residency is worth most where the work is smallest. A large job amortises a round trip on its own; a small one does not.