Recipes - SmithSoft-Studios/JAZMIN GitHub Wiki

Recipes

Complete solutions to common jobs. Each one runs as shown; the output under it is what it prints.

Statements: one file, one document per client

The classic: a large export holds every client's statement lines, and each client needs a PDF. Instead of re-reading a huge JSON file per client, or splitting it into thousands of files, convert once and read each section directly.

import { open, write } from '@smithsoft-studios/jazmin';

// Once, when the data arrives: lines sorted by client, so each client's lines sit in one or two chunks
const lines = [];
for (let c = 1; c <= 2000; c++) {
  for (let n = 1; n <= 25; n++) {
    lines.push({ client: `CL-${String(c).padStart(5, '0')}`, line: n, date: new Date(Date.UTC(2026, 2, n)), description: n === 1 ? 'Opening balance' : 'Purchase', amount: ((c * 31 + n * 7) % 900) / 4 });
  }
}
write('statements.jzm', lines, {
  columns: [
    { name: 'client', type: 'string', nullable: false },
    { name: 'line', type: 'int', nullable: false },
    { name: 'date', type: 'datetime' },
    { name: 'description', type: 'string' },
    { name: 'amount', type: 'float' },
  ],
  sortedBy: ['client', 'line'],
  chunkRows: 1024,
});

// A worker: open once, then one client at a time (or stream them all in order: see Tuning Speed and Memory)
const reader = open('statements.jzm');
const renderPdf = (client, rows) => `${client}: ${rows.length} lines, total ${rows.reduce((t, r) => t + r.amount, 0).toFixed(2)}`;
for (const client of ['CL-00001', 'CL-01234', 'CL-02000']) {
  const rows = [...reader.find({ client }, { select: ['date', 'description', 'amount'] })];
  console.log(renderPdf(client, rows));
}
const fresh = open('statements.jzm');
const { chunksRead, bytesRead } = fresh.explain({ client: 'CL-01234' }, { analyze: true });
console.log(`one client reads ${chunksRead} of ${fresh.chunkCount} chunks (${(bytesRead / 1024).toFixed(1)} KB)`);
CL-00001: 25 lines, total 762.50
CL-01234: 25 lines, total 3406.25
CL-02000: 25 lines, total 3093.75
one client reads 1 of 49 chunks (2.6 KB)

On a real 271 MB statement file (1.37M lines, 5,000 clients), this took one section from 1.3 s to 0.3–0.4 ms, all 5,000 from about 1.8 hours to 2 s, and peak memory from 1,659 MB to 73–85 MB. To make the PDFs from an HTML template stored in the file, see renderPdf in Server Side.

An order history per customer, from several tables (1.3.0)

Keep customers, orders, order lines and products in one file, as several tables, each sorted by its link to its parent. Then one export shape gives any customer's history as nested JSON, and the same shape exports every customer in one pass.

// not run: a web server, with shop.jzm written as in Export Shapes
import fs from 'node:fs';
import http from 'node:http';
import { open, toJSON } from '@smithsoft-studios/jazmin';

const customers = open('shop.jzm');                       // opened once, queried per request
const shape = JSON.parse(fs.readFileSync('history-shape.json', 'utf8'));
http.createServer((req, res) => {
  const id = Number(new URL(req.url, 'http://x').searchParams.get('customer'));
  res.setHeader('content-type', 'application/json');
  res.end(toJSON(customers, { shape, filter: { id } }));  // one customer: 0.08 s on 2.5M order lines
}).listen(8080);
  • Write the tables sorted by their links: orders by customer_id, id, lines by customer_id, order_id, line, and link the lines on both columns. A full export then reads each table once.
  • Small lookup tables (products, countries) are kept in memory by key while exporting.

A nightly encrypted export

A job exports a table every night for another team. The key lives in a secret manager; the file carries what it is in its metadata; the receiving side checks it before using it.

import { JazminKey, open, writeAsync } from '@smithsoft-studios/jazmin';

// In production: the key text comes from your secret manager, e.g. process.env.EXPORT_KEY
const keyText = process.env.EXPORT_KEY ?? JazminKey.generate().export();

async function* customersFromDatabase() { // stands in for a database cursor
  for (let id = 1; id <= 5000; id++) yield { id, name: `Customer ${id}`, country: ['ZA', 'NA', 'BW'][id % 3], balance: `${(id * 13) % 10000}.00` };
}

await writeAsync('customers-2026-10-07.jzm', customersFromDatabase(), {
  columns: [
    { name: 'id', type: 'int', nullable: false },
    { name: 'name', type: 'string' },
    { name: 'country', type: 'string' },
    { name: 'balance', type: 'decimal' },
  ],
  sortedBy: ['id'],
  key: JazminKey.parse(keyText),
  metadata: { source: 'crm', table: 'customers', exportedFor: 'finance', schemaVersion: 3 },
});

// The receiving side
const reader = open('customers-2026-10-07.jzm', { key: keyText });
if (reader.metadata.schemaVersion !== 3) throw new Error('Unexpected export version');
console.log(reader.metadata, reader.rowCount, 'rows');
console.log('ZA balances over 9,000:', reader.count({ country: 'ZA', balance: { gt: '9000' } }));
{
  source: 'crm',
  table: 'customers',
  exportedFor: 'finance',
  schemaVersion: 3
} 5000 rows
ZA balances over 9,000: 154
  • Encryption replaces a separate gzip-then-encrypt step: the file is compressed and authenticated as it's written.
  • Rotate the key when someone who knew it leaves: rotateKey(path, { key, newKey }) re-locks a file in under a second without decoding rows.

A client portal: one file, a key per client, with expiry

One file for all clients; each client's app opens it with their own key and sees only their rows, for a limited time.

import { JazminKey, open, write } from '@smithsoft-studios/jazmin';

const owner = JazminKey.generate(); // stays on your server
const clients = ['ACME', 'GLOBEX', 'INITECH'];
const keys = Object.fromEntries(clients.map((c) => [c, owner.createAccessKey()])); // deliver keys[c].export() to each client

write('invoices.jzm', clients.flatMap((client, i) => Array.from({ length: 4 }, (_, n) => ({
  client, invoice: `INV-${i + 1}0${n + 1}`, amount: (i + 1) * 100 + n, internalNote: `margin ${10 + n}%`,
}))), {
  key: owner,
  sortedBy: ['client'],
  access: {
    partitionBy: 'client',
    columnGroups: { internal: ['internalNote'] },
    grants: clients.map((c) => ({ key: keys[c], rows: [c], columns: ['*'], expiresIn: '90d', label: c })),
  },
});

// GLOBEX's app
const view = open('invoices.jzm', { key: keys.GLOBEX.export(), accessState: { dir: 'access-state' } });
console.log(view.columns.map((c) => c.name), `${view.rowCount} rows visible, ${view.hiddenRowCount} hidden`);
for (const row of view.rows()) console.log(row);
console.log('access ends in about', Math.round((new Date(view.access.expires) - Date.now()) / 86_400_000), 'days');
[ 'client', 'invoice', 'amount' ] 4 rows visible, 8 hidden
{ client: 'GLOBEX', invoice: 'INV-201', amount: 200 }
{ client: 'GLOBEX', invoice: 'INV-202', amount: 201 }
{ client: 'GLOBEX', invoice: 'INV-203', amount: 202 }
{ client: 'GLOBEX', invoice: 'INV-204', amount: 203 }
access ends in about 90 days

When a client leaves: revokeAccess(path, owner, key). To extend everyone for another quarter, rewrite the grants with update(path, { key: owner, grant: [...] }). See Access Control and Time-Limited Access.

A dashboard: monthly counts and totals without loading rows

count() answers from chunk statistics and indexes where it can, and columnArrays() gives typed arrays instead of row objects.

import { open, write } from '@smithsoft-studios/jazmin';

const start = Date.UTC(2025, 0, 1);
write('sales.jzm', Array.from({ length: 300_000 }, (_, i) => ({
  at: new Date(start + i * 105_000), // about 300 per day through 2025
  region: ['ZA', 'NA', 'BW'][i % 3],
  amount: ((i * 7919) % 5000) / 10,
})), { sortedBy: ['at'], columns: [
  { name: 'at', type: 'datetime', nullable: false },
  { name: 'region', type: 'string', index: 'sorted' },
  { name: 'amount', type: 'float' },
] });

const reader = open('sales.jzm');
const months = Array.from({ length: 12 }, (_, m) => [new Date(Date.UTC(2025, m, 1)), new Date(Date.UTC(2025, m + 1, 1))]);
const counts = months.map(([from, to]) => reader.count({ at: { gte: from, lt: to } }));
console.log('sales per month:', counts.join(', '));

const { rowCount, values } = reader.columnArrays({ region: 'NA', at: { gte: months[11][0] } }, { select: ['amount'] });
let total = 0;
for (const v of values.amount) total += v;
console.log(`NA in December: ${rowCount} sales, ${total.toFixed(2)} total (${values.amount.constructor.name})`);

const fresh = open('sales.jzm');
const { bytesRead, rows } = fresh.explain({ at: { gte: months[5][0], lt: months[5][1] } }, { analyze: true, select: ['at'] });
console.log(`June: ${rows} rows from ${Math.round(bytesRead / 1024)} KB of a ${fresh.chunkCount}-chunk file`);
sales per month: 25509, 23040, 25509, 24685, 25509, 24686, 25508, 25509, 24685, 25509, 24686, 25165
NA in December: 8388 sales, 2096781.00 total (Float64Array)
June: 24686 rows from 62 KB of a 74-chunk file

Look up by two different keys

Sort by the key used most, and index the second. Both lookups then read a page or two, never the whole file.

import { open, write } from '@smithsoft-studios/jazmin';

write('payments.jzm', Array.from({ length: 200_000 }, (_, i) => ({
  account: `ACC-${String(Math.floor(i / 40)).padStart(5, '0')}`, // 40 payments per account
  reference: `TX-${String((i * 7919) % 1_000_003).padStart(7, '0')}`, // unique, in no order
  amount: (i % 777) / 3,
})), {
  columns: [
    { name: 'account', type: 'string', nullable: false },
    { name: 'reference', type: 'string', index: 'sorted' },
    { name: 'amount', type: 'float' },
  ],
  sortedBy: ['account'],
});

for (const filter of [{ account: 'ACC-01234' }, { reference: 'TX-0759764' }]) {
  const r = open('payments.jzm');
  const { strategy, rows, chunksRead, indexPagesRead, bytesRead } = r.explain(filter, { analyze: true });
  console.log(JSON.stringify(filter), `-> ${rows} rows via ${strategy}: ${chunksRead} chunk(s), ${indexPagesRead} index section(s), ${Math.round(bytesRead / 1024)} KB`);
  r.close();
}
{"account":"ACC-01234"} -> 40 rows via scan: 1 chunk(s), 0 index section(s), 19 KB
{"reference":"TX-0759764"} -> 1 rows via index: 1 chunk(s), 2 index section(s), 43 KB

An API backed by a JAZMIN file

A read-only API over a large dataset: open the file once at start-up, translate query-string filters, and stream JSON back. Shown with Node's http; the same handler works in Express or Fastify.

import http from 'node:http';
import { open } from '@smithsoft-studios/jazmin';
// not run: starts a server

const reader = open('customers.jzm', { key: process.env.CUSTOMERS_KEY });

http.createServer((req, res) => {
  const url = new URL(req.url, 'http://localhost');
  const where = url.searchParams.get('where'); // e.g. ?where={"country":"ZA","balance":{"gt":"100"}}
  const limit = Math.min(Number(url.searchParams.get('limit') ?? 50), 500);
  const offset = Number(url.searchParams.get('offset') ?? 0);
  try {
    const filter = where ? JSON.parse(where) : null;
    const rows = [...reader.find(filter, { select: ['id', 'name', 'country', 'balance'], limit, offset })];
    res.writeHead(200, { 'content-type': 'application/json' });
    res.end(JSON.stringify({ total: reader.count(filter ?? undefined), rows }));
  } catch (e) {
    res.writeHead(400, { 'content-type': 'application/json' });
    res.end(JSON.stringify({ error: e.message })); // invalid filters name the problem: "unknown column 'x'"
  }
}).listen(8080);

Filters are checked against the file's columns, so a bad query string returns a clear 400. For many concurrent requests, give each worker its own reader. See Server Side.

.NET: a daily CSV drop into a queryable archive

Each day a partner drops a CSV. Convert it streaming, append it to a yearly archive, and query the archive with LINQ.

using System.Text;
using Jazmin;
using Jazmin.Query;
using Jazmin.Serialization;

var columns = new[]
{
    new JazminColumn("day", JazminType.DateTime) { Nullable = false },
    new JazminColumn("store", JazminType.String) { Indexes = [JazminIndexKind.Sorted] },
    new JazminColumn("sales", JazminType.Decimal),
};

for (int day = 1; day <= 5; day++)
{
    // The partner's CSV for the day
    var csv = new StringBuilder("day,store,sales\n");
    for (int s = 1; s <= 200; s++) csv.Append($"2026-10-0{day}T00:00:00Z,S{s:D3},{(s * day * 37) % 5000}.{s % 100:D2}\n");
    File.WriteAllText($"drop-{day}.csv", csv.ToString());

    // Convert with the archive's columns (read once, typed), then add it to the archive
    JazminConvert.FromCsvFile($"drop-{day}.csv", $"drop-{day}.jzm", columns: columns);
    using var drop = JazminReader.Open($"drop-{day}.jzm");
    var rows = drop.Rows().Select(r => (IReadOnlyDictionary<string, object?>)r.ToDictionary(kv => kv.Key, kv => kv.Value)).ToList();
    if (!File.Exists("archive.jzm"))
        using (var writer = JazminWriter.Create("archive.jzm", columns, new JazminWriteOptions { SortedBy = ["day"] }))
            foreach (var row in rows) writer.WriteRow(row);
    else
        JazminFile.Append("archive.jzm", new JazminAppend { Insert = rows, AutoCompact = new JazminAutoCompact(Appends: 30) });
}

using var archive = JazminReader.Open("archive.jzm");
Console.WriteLine($"{archive.RowCount} rows, {archive.AppendCount} appends");
var sales = archive.AsQueryable<DailySales>();
Console.WriteLine($"S042 total: {sales.Where(s => s.Store == "S042").Sum(s => s.Sales)}");
Console.WriteLine($"best day for S007: {sales.Where(s => s.Store == "S007").OrderByDescending(s => s.Sales).First().Day:yyyy-MM-dd}");

public class DailySales
{
    public DateTime Day { get; set; }
    public string Store { get; set; } = "";
    public decimal Sales { get; set; }
}
1000 rows, 4 appends
S042 total: 13312.10
best day for S007: 2026-10-05

JAZMIN 1.2 · file format 1.0 · Repository · npm · NuGet · MIT license · Examples in this wiki are run and checked by tools/check-examples.mjs

⚠️ **GitHub.com Fallback** ⚠️