Writing Files - SmithSoft-Studios/JAZMIN GitHub Wiki

Writing Files

There are three ways to write a JAZMIN file. All three produce the same format.

Way	JavaScript	.NET	Use when
One call	`write(path, rows, options)`, `JAZMIN.stringify(rows)`	`JazminConvert.SerializeObject(list)`, `new JazminSerializer(settings).Serialize(path, list)`	The rows are already in memory, or come from an iterator
Streaming writer	`new JazminWriter(path, options)` + `writeRow()` + `finish()`	`JazminWriter.Create(path, columns, options)` + `WriteValues()` / `WriteRow()` + `Dispose()`	Millions of rows, generated or read from somewhere else
Async	`writeAsync()`, `writer.writeRowsAsync()`	`writer.WriteRowsAsync()`, `WriteValuesAsync()`	Rows come from a database cursor or network stream (see Server Side)

Pass null as the path (JavaScript) or a Stream (.NET) to get the bytes instead of a file. When writing to a path fails part-way, the half-written file is deleted.

Schemas: inferred or declared

Without columns, the schema is inferred from the rows:

whole numbers become int, other numbers float;
strings become string (with detectDates: true, ISO-8601 strings become datetime);
Date becomes datetime, Buffer becomes binary, objects and arrays become json;
a column is nullable when any row leaves it out or holds null.

Declare the columns for anything you keep: you choose exact types (decimal for money), nullability, indexes, descriptions and attributes, and every row is checked against them.

import { inferSchema } from '@smithsoft-studios/jazmin';

const rows = [
  { id: 1, name: 'Ann', score: 9.5, joined: '2024-01-02T00:00:00Z', tags: ['a'] },
  { id: 2, name: 'Bob', score: 7, joined: '2024-03-04T00:00:00Z' },
];
for (const c of inferSchema(rows, { detectDates: true })) console.log(`${c.name}: ${c.type}${c.nullable ? ' (nullable)' : ''}`);

id: int
name: string
score: float
joined: datetime
tags: json (nullable)

Every column has:

Property	Default	Meaning
`name`	required	Any non-empty text; names are matched exactly in JavaScript, case-insensitively by the .NET serializer
`type`	required	`bool`, `int`, `float`, `decimal`, `string`, `datetime`, `binary` or `json` (Core Concepts)
`nullable`	`true`	`false` makes a `null` or missing value an error
`index`	none	`'sorted'`, `'trigram'` or both: `['sorted', 'trigram']`
`description`	none	Travels with the file; shown by the viewer and the CLI
`attributes`	none	Free-form details, such as `{ unit: 'kPa' }` or `{ currency: 'ZAR' }`

Rows are validated

A wrong type, a misspelt column or a null in a non-nullable column throws, naming the row and column. Nothing half-written is left behind.

import { JazminValidationError, write } from '@smithsoft-studios/jazmin';

const columns = [{ name: 'id', type: 'int', nullable: false }, { name: 'name', type: 'string' }];
const attempts = [
  [{ id: 1, name: 'Ann' }, { id: 'two', name: 'Bob' }],
  [{ id: 1, nme: 'Ann' }],
  [{ id: null, name: 'Ann' }],
];
for (const rows of attempts) {
  try {
    write(null, rows, { columns });
  } catch (e) {
    if (e instanceof JazminValidationError) console.log(e.message);
  }
}

Column 'id': expected an integer, got string
Row 0: unknown column 'nme'
Row 0: column 'id' is not nullable

Options

Option (JS / .NET)	Default	Effect
`columns` / the `columns` argument	inferred	The schema
`sortedBy` / `SortedBy`	none	Rows must arrive in this column order. Lookups and ranges on the first column skip straight to the right chunks, with no index; updates merge in place. A key can also ignore case or compare numbers by value (1.7.0, `{ column, nocase, natural }`; .NET `Sort`): see Text orders
`indexes` / `[JazminIndex]` or `Indexes`	none	Shorthand: `{ country: 'sorted', name: ['trigram'] }`
`codec` / `Codec`	`deflate`	`brotli`: smaller, slower to write. `none`: for data that is already compressed
`level` / `CompressionLevel`	balanced	Deflate 0–9, Brotli 0–11
`chunkRows` / `ChunkRows`	4,096	Rows per chunk. Smaller chunks suit many single-row lookups; larger ones suit whole-file reads
`chunkBytes` / `ChunkBytes`	1 MiB	A chunk is closed early once its raw data reaches this size (wide rows)
`compactIndexes` / `CompactIndexes`	off	Much smaller sorted indexes on numbers and dates. Readers need 1.2 or later (New in 1.2)
`metadata` / `Metadata`	none	Your own key/values: `{ source: 'crm', exportedAt: '...' }`
`key` or `password` / `Key` or `Password`	none	Encrypt the file (Encryption and Keys)
`kdfIterations` / `KdfIterations`	600,000	PBKDF2 iterations for passwords
`access` / `Access`	none	One file, many keys (Access Control)
`tables` / `Tables`	one table	Several tables in one file (Several Tables)
`files`, `package` / `Files`, `Package`	none	Embedded files and documents (Embedded Files and Documents)
`locks` / `Locks`	none	Shared files: passwords or 2FA for stored files and masked columns (1.7.0; locked files, Masked Columns)
`onProgress`, `signal` / `Progress`, `CancellationToken`	none	Progress and Stop (1.7.0; for rewrites)
`priority` / `Priority`	`balanced`	`memory` or `speed` first (Tuning Speed and Memory)
`maxDegreeOfParallelism` / `MaxDegreeOfParallelism`	by priority	Threads compressing chunks; 1 for the lowest memory

What the options do to the size

The same 200,000 orders, written seven ways:

import { write } from '@smithsoft-studios/jazmin';

const regions = ['ZA', 'NA', 'BW', 'KE'];
const rows = Array.from({ length: 200_000 }, (_, i) => ({
  id: i + 1,
  at: new Date(Date.UTC(2026, 0, 1) + i * 60_000),
  region: regions[i % 4],
  customer: `C-${(i * 7919) % 10_000}`,
  amount: ((i * 104729) % 100_000) / 100,
}));
const columns = [
  { name: 'id', type: 'int', nullable: false },
  { name: 'at', type: 'datetime', nullable: false },
  { name: 'region', type: 'string' },
  { name: 'customer', type: 'string' },
  { name: 'amount', type: 'float' },
];
const indexed = columns.map((c) => (['at', 'region', 'customer'].includes(c.name) ? { ...c, index: 'sorted' } : c));

const json = Buffer.byteLength(JSON.stringify(rows));
const size = (label, options) => {
  const bytes = write(null, rows, { columns, ...options }).length;
  console.log(label.padEnd(36), `${Math.round(bytes / 1024).toLocaleString('en-US').padStart(6)} KB`, `${(json / bytes).toFixed(0).padStart(3)}x smaller than JSON`);
};
console.log('JSON'.padEnd(36), `${Math.round(json / 1024).toLocaleString('en-US').padStart(6)} KB`);
size('deflate (default)', {});
size('brotli', { codec: 'brotli' });
size('none', { codec: 'none' });
size('deflate, sortedBy id', { sortedBy: ['id'] });
size('deflate, 3 sorted indexes', { columns: indexed });
size('deflate, 3 indexes, compactIndexes', { columns: indexed, compactIndexes: true });
size('brotli, 3 indexes, compactIndexes', { columns: indexed, compactIndexes: true, codec: 'brotli' });

JSON                                 18,575 KB
deflate (default)                     1,155 KB  16x smaller than JSON
brotli                                  796 KB  23x smaller than JSON
none                                  3,079 KB   6x smaller than JSON
deflate, sortedBy id                  1,155 KB  16x smaller than JSON
deflate, 3 sorted indexes             2,267 KB   8x smaller than JSON
deflate, 3 indexes, compactIndexes    1,175 KB  16x smaller than JSON
brotli, 3 indexes, compactIndexes       808 KB  23x smaller than JSON

sortedBy costs nothing (these rows were already in id order) and makes an index on id unnecessary.
Indexes can be the largest part of a file: here three sorted indexes nearly doubled it. With compactIndexes, the same indexes take about 20 KB instead of 1.1 MB, because times and ids that follow each other are stored as small differences. Index only the columns you search on, and use compactIndexes once every reader is on 1.2.
Brotli saves another 30% for archives; it is slower to write, and the browser reader can't read it.
Even uncompressed, the file is several times smaller than JSON: names are stored once and values in binary.

The streaming writer

import { JazminWriter, open } from '@smithsoft-studios/jazmin';

const writer = new JazminWriter('sensors.jzm', {
  columns: [
    { name: 'sensor', type: 'string', nullable: false },
    { name: 'at', type: 'datetime', nullable: false },
    { name: 'pressure', type: 'float', description: 'Line pressure', attributes: { unit: 'kPa' } },
  ],
  sortedBy: ['sensor', 'at'], // rows must arrive in this order
  metadata: { site: 'Plant 4', exportedBy: 'nightly-job' },
});
for (const sensor of ['S-01', 'S-02', 'S-03']) {
  for (let m = 0; m < 1440; m++) writer.writeRow({ sensor, at: new Date(Date.UTC(2026, 4, 1, 0, m)), pressure: 300 + (m % 60) });
}
writer.finish(); // required: writes the indexes, catalog and trailer

const reader = open('sensors.jzm');
console.log(reader.rowCount, 'rows, sorted by', reader.sortedBy, 'metadata', reader.metadata);
console.log(reader.columns[2]);
console.log(reader.count({ sensor: 'S-02', at: { gte: new Date('2026-05-01T12:00Z') } }), 'S-02 readings after noon');

4320 rows, sorted by [ 'sensor', 'at' ] metadata { site: 'Plant 4', exportedBy: 'nightly-job' }
{
  name: 'pressure',
  type: 'float',
  nullable: true,
  description: 'Line pressure',
  attributes: { unit: 'kPa' }
}
720 S-02 readings after noon

finish() is required. Until it runs, the file has no catalog. Call abort() to stop and delete it.
Memory is one chunk plus the index lists, whatever the row count.
Out-of-order rows in a sortedBy file throw a JazminValidationError naming the row.

In .NET, Dispose() finishes the file (unless a row failed validation), and WriteValues takes the values in column order, with no dictionary per row:

using Jazmin;
using System.Text.Json.Nodes;

var columns = new[]
{
    new JazminColumn("sensor", JazminType.String) { Nullable = false },
    new JazminColumn("at", JazminType.DateTime) { Nullable = false },
    new JazminColumn("pressure", JazminType.Float) { Description = "Line pressure", Attributes = new JsonObject { ["unit"] = "kPa" } },
};
var options = new JazminWriteOptions
{
    SortedBy = ["sensor", "at"],
    Metadata = new JsonObject { ["site"] = "Plant 4" },
};
using (var writer = JazminWriter.Create("sensors.jzm", columns, options))
{
    foreach (var sensor in new[] { "S-01", "S-02", "S-03" })
        for (int m = 0; m < 1440; m++)
            writer.WriteValues(sensor, new DateTime(2026, 5, 1, 0, 0, 0, DateTimeKind.Utc).AddMinutes(m), 300.0 + m % 60);
}

using var reader = JazminReader.Open("sensors.jzm");
Console.WriteLine($"{reader.RowCount} rows, sorted by {string.Join(", ", reader.SortedBy!)}, site {reader.Metadata["site"]}");
Console.WriteLine($"{reader.Columns[2]} ({reader.Columns[2].Description}, {reader.Columns[2].Attributes!.ToJsonString()})");

4320 rows, sorted by sensor, at, site Plant 4
pressure: float? (Line pressure, {"unit":"kPa"})

Writing from your own classes (.NET)

The serializer builds the schema from your class: see .NET Serializer and LINQ for the attributes ([JazminIndex], [JazminProperty], [JazminIgnore]) and settings.

using Jazmin;
using Jazmin.Serialization;

var settings = new JazminSerializerSettings
{
    Codec = JazminCodec.Brotli,
    Indexes = new() { ["Region"] = [JazminIndexKind.Sorted] }, // indexes without attributes
};
var orders = Enumerable.Range(1, 1000).Select(i => new Order { Id = i, Region = i % 2 == 0 ? "ZA" : "NA", Amount = i * 1.5m });
new JazminSerializer(settings).Serialize("orders.jzm", orders); // streams: the sequence is not collected first

using var reader = JazminReader.Open("orders.jzm");
Console.WriteLine(string.Join(", ", reader.Columns));
Console.WriteLine(string.Join(", ", reader.Indexes.Select(i => $"{i.Column} {i.Kind}")));

public class Order
{
    public int Id { get; set; }
    public string Region { get; set; } = "";
    public decimal Amount { get; set; }
}

Id: int, Region: string?, Amount: decimal
Region Sorted

Good to know

Write once, sorted. If you know how the file will be queried, sort the rows by that column and say so with sortedBy. It is the cheapest index there is. See Tuning Speed and Memory.
Decimals: pass them as strings in JavaScript ('1520.75'), or as decimal in .NET. A JavaScript number written to a decimal column is accepted, but a number like 0.1 + 0.2 is already inexact before it arrives.
Dates are stored in UTC with millisecond precision. In .NET, a DateTime of kind Local is converted to UTC; Unspecified is taken as UTC.
Big integers: JavaScript bigint values beyond ±2^53 are stored exactly, and come back as bigint.
Lists and objects are stored as JSON text by default. From 1.4, nested columns store them as columns of their own fields: smaller and faster to read, and filtered with any, all and match.
Changing a file later doesn't need a full rewrite by hand: see Updating, Appending and Compacting.