Tuning Speed and Memory

JAZMIN is designed to put memory first, then speed. Most of the gains come from how a file is laid out when it's written, and from asking only for what you need when it's read. This page is a checklist, then the reasoning.

Checklist

Do	Avoid
sortedBy on the column you look up by most	An index on that same column: statistics already find it, and writers leave it out
A sorted index on a second lookup column	Indexes on columns nobody filters on (they cost size and write time)
select only the columns you need	Reading whole rows of a wide table
Stream with for…of, for await, DeserializeEnumerable, await foreach	[...rows] / ToList() on millions of rows
One reader per thread, opened once and reused	Opening the file for every row
count() and columnArrays() for totals and charts	Building row objects just to count or sum them
Sort row ids before fetching many by position	Fetching rows by position in random order
priority: 'memory' on small servers or many open files	Many readers and writers each running their own threads on a 2-core container
Node: --max-semi-space-size=8 on memory-limited servers	Judging memory by default heap growth (mostly garbage awaiting collection)
compactIndexes once every reader is on 1.2	Large indexes on ids and dates in files only 1.2+ reads
reverse: true for the newest rows of a file sorted by time (1.7.0)	Counting the matches, then reading the last page
estimate() before a query a person typed (1.7.0)	Running an unknown query on millions of rows without warning

Let the file tell you: advise()

reader.advise({ columns }) (CLI: jazmin advise file.jzm --column account) reads only the chunk directories and statistics, and says how the layout serves lookups of the columns you name, with suggestions. Here a file sorted by time is looked up by account, then rewritten as advised:

import { open, write } from '@smithsoft-studios/jazmin';

const rows = Array.from({ length: 200_000 }, (_, i) => ({
  at: new Date(Date.UTC(2026, 0, 1) + i * 60_000),
  account: `ACC-${String((i * 7919) % 500).padStart(3, '0')}`,
  amount: (i % 1000) / 4,
}));
const columns = [
  { name: 'at', type: 'datetime', nullable: false },
  { name: 'account', type: 'string', nullable: false },
  { name: 'amount', type: 'float' },
];
write('by-time.jzm', rows, { columns, sortedBy: ['at'] });

const before = open('by-time.jzm');
const advice = before.advise({ columns: ['account'] });
console.log(`${advice.rows} rows in ${advice.chunks} chunks, sorted by ${advice.sortedBy}`);
for (const s of advice.suggestions) console.log('-', s);

// Follow the advice: sort by account, then time
const sorted = [...rows].sort((a, b) => (a.account < b.account ? -1 : a.account > b.account ? 1 : a.at - b.at));
write('by-account.jzm', sorted, { columns, sortedBy: ['account', 'at'] });

for (const file of ['by-time.jzm', 'by-account.jzm']) {
  const r = open(file);
  const { rows: found, chunksRead, bytesRead } = r.explain({ account: 'ACC-042' }, { analyze: true });
  console.log(`${file.padEnd(15)} one account: ${found} rows, ${chunksRead} chunks, ${Math.round(bytesRead / 1024)} KB read`);
  r.close();
}

200000 rows in 49 chunks, sorted by at
- The rows of one 'account' value lie in about 49.0 of 49 chunks: a lookup reads about 227 KB. Writing the file with sortedBy: ["account","at"] would put them in a few chunks.
by-time.jzm     one account: 400 rows, 49 chunks, 228 KB read
by-account.jzm  one account: 400 rows, 1 chunks, 1 KB read

The levers

1. Sort order (sortedBy): the cheapest index there is

Each chunk records the smallest and largest value of every column. When rows are sorted by a column, each value's rows sit in one or two chunks, so a lookup or range on that column reads just those chunks. No index is needed, the file doesn't grow, and the reader also skips whole chunks when paging (offset) and counts whole chunks without reading them.

• Sort by the column you look up by most, then by what you page or range over (['account', 'at']).
• Declare it with sortedBy: the reader then uses binary search, and updates keep the order.
• In access-controlled files, sort by the partition column first, so each partition's chunks are full.

2. Indexes, for a second way in

A sorted index on another column (a reference number in a file sorted by account) finds its rows directly. Indexes over millions of values are paged: a lookup reads a small directory and one page (1M unique values: 72 KB read instead of 7.6 MB). The reader uses an index only when it reads less than the scan would. A trigram index serves contains and icontains text search. From 1.4, text that is in most rows, spread through the file, is found by a scan instead: the index would name rows in almost every chunk.

3. Chunk size (chunkRows)

Default 4,096 rows	Smaller (512–1,024)	Larger (16,384+)
A good balance	Many lookups of a few rows each: less is decoded per lookup	Whole-file reads and archives: slightly better compression

advise() suggests a chunkRows when one value's rows are much fewer than a chunk holds.

4. select: decode only what you use

Inside a chunk each column is stored separately, and a query decodes only the columns it filters on and the ones it selects. On a 300-column file, summing 3 columns is about 10× faster than reading whole rows. In .NET, typed queries (Query<T>, AsQueryable<T>) read only the properties their type maps.

5. Memory or speed first (priority)

One setting decides what reading and writing favour where memory and speed pull apart. The file written and the rows read are the same whichever you choose.

const reader = open('statements.jzm', { priority: 'speed' });            // 'memory' | 'balanced' (default) | 'speed'
const writer = new JazminWriter('out.jzm', { columns, priority: 'memory' });

using var reader = JazminReader.Open(path, new JazminReadOptions { Priority = JazminPriority.Speed });
using var writer = JazminWriter.Create(path, columns, new JazminWriteOptions { Priority = JazminPriority.Memory });

	memory	balanced (default)	speed
.NET reads	The calling thread only	Up to 4 chunks decoded ahead	Up to 8 chunks ahead
.NET writes	The calling thread only	One thread per processor, up to 16	As balanced
Node writes	This thread only	Up to 2 worker threads	As balanced
Node reads	This thread	This thread	Next chunks decompressed on up to 4 worker threads
Export shapes over unsorted groups	Batches of 100,000 rows	Batches of 100,000 rows	Batches of 1,000,000 rows (1.3.0)
Export shapes across tables (1.3.0)	Linked tables up to 10,000 rows kept by key; others per 2,500 parents	Up to 100,000 rows; per 10,000 parents	Up to 1,000,000 rows; per 20,000 parents
Sorted indexes in a rewrite (1.7.0)	Past 1,000,000 entries, through temporary files	Past 2,000,000 entries	All held in memory

Measured on 200,000 rows × 300 columns: memory keeps .NET narrow queries to 49 MB (about 3× slower); speed makes Node queries of a few columns about 2.8× as fast and .NET about 30% faster, for 15–70 MB more. Full tables are in Performance and Benchmarks. An explicit maxDegreeOfParallelism (.NET MaxDegreeOfParallelism) wins over the priority.

6. Codec

deflate (default)	brotli	none
Fast to write, read everywhere	About 30% smaller, slower to write; browsers can't read it	For data that is already compressed, or when attacker-supplied text sits next to secrets

7. Node.js garbage collection

JAZMIN keeps little alive while it reads (about 11 MB for a full scan of a 300-column file); most of a Node process's memory is garbage V8 hasn't yet collected. On a memory-constrained server, cap it:

node --max-semi-space-size=8 render-statements.js
NODE_OPTIONS=--max-semi-space-size=8 npm start   # for any way of starting Node

On 1M × 300 columns, reading every row went from 204 MB to 98 MB with no measurable slowdown. Smaller values (1–4) collect so often that full scans ran about 40% slower, so 8 is a good default.

Large rewrites in bounded memory (1.7.0)

Two parts of a rewrite (write, update, compact) used to grow with the file: a new sort order, and the sorted indexes, which hold an entry for each value until the file is written (25,000,000 different values held 3.5–4 GB). Both now stay within a budget:

• Sorted indexes: past 1,000,000 entries (priority: 'memory') or 2,000,000 ('balanced'), the builder writes its sorted entries to temporary files and merges them into the index at the end. 'speed' holds them all, as before. The index is byte for byte the same.
• A new sort order sorts up to sortRunRows rows at a time (10,000, 25,000 or 100,000 by priority), and now merges at most 256 sorted runs at once; with more rows, each run holds more of them.
• The temporary files go to the system's temp folder, each encrypted with a key made for it and never kept, so no row is written there readable. They are deleted when the rewrite ends.

Measured on 25,000,000 rows with priority: 'memory' (peak memory, each in its own process):

Rewrite	JavaScript, before → 1.7.0	.NET, before → 1.7.0
A new index on a text column	4,762 → 729 MB	4,730 → 515 MB
A new sort order	4,627 → 714 MB	3,864 → 501 MB

Large rewrites need room in the temp folder for these files. Nothing changes for files below the thresholds.

Patterns

Open once, query many times. Opening reads only the catalog; statistics load per column on demand, and index pages are cached (up to 8 per index).

using var reader = JazminReader.Open("statements.jzm");
foreach (var account in accountsToRender)
{
    var lines = reader.Find(JazminFilter.Eq("account", account), new JazminQueryOptions { Select = ["date", "amount"] }).ToList();
    await RenderPdfAsync(account, lines);
}

Process every section in file order without an index or a lookup per section: stream the rows and group consecutive ones. Each chunk is read exactly once.

function* bySection(rows, column) {
  let current;
  let batch = [];
  for (const row of rows) {
    if (batch.length && row[column] !== current) {
      yield [current, batch];
      batch = [];
    }
    current = row[column];
    batch.push(row);
  }
  if (batch.length) yield [current, batch];
}
for (const [account, lines] of bySection(reader.rows(), 'account')) await renderPdf(account, lines);

Charts and totals with columnArrays(): one Float64Array per number or date column, about a tenth of the memory of row objects (200,000 rows: 3.3 MB instead of 30.7 MB).

Measure, don't guess: explain(filter, { analyze: true }) on a freshly opened reader shows the bytes, chunks and columns a query read. See Reading and Querying.