Export Shapes - SmithSoft-Studios/JAZMIN GitHub Wiki

Export Shapes: Nested JSON and XML

A JAZMIN file holds flat rows. Consumers often want something else: one entry per client, with that client's transactions and totals. An export shape is a small JSON template describing the output you want. The same shape works in JavaScript and .NET and gives the same result.

  • Checked up front. A shape that names a column the file doesn't have, or that your key can't see, is rejected before any data is read, with the place of the mistake: Shape at clients[].name: unknown or hidden column 'clientNme'.
  • Repeated details are written once. In a group of rows, a column gives the first row's value.
  • Memory stays low. Each list is a query on the file: only the columns the shape uses are decoded, and filters use indexes and statistics.

From flat rows to clients with their transactions

import { open, shapeSchema, toJSON, toXML, write } from '@smithsoft-studios/jazmin';

write('statements.jzm', [
  { client: 'C1', clientName: 'ABC Corp', address: '10 Test Street', date: new Date('2025-03-01'), amount: 1000 },
  { client: 'C1', clientName: 'ABC Corp', address: '10 Test Street', date: new Date('2025-03-04'), amount: 250.5 },
  { client: 'C2', clientName: 'Test Ltd', address: '2 Test Place', date: new Date('2025-03-02'), amount: 20000 },
], { sortedBy: ['client'], metadata: { title: 'March statements' } });

const shape = {
  title: { $meta: 'title' },              // a metadata member of the file
  clients: {
    $rows: {
      id: 'client',                       // a column: in a group, the first row's value
      name: 'clientName',
      address: 'address',
      balance: { $sum: 'amount' },        // totals over the group's rows
      lines: { $count: true },
      transactions: { $rows: { date: 'date', amount: 'amount' }, $sort: ['date'], $xmlItem: 'transaction' },
    },
    $groupBy: 'client',                   // one entry per client
    $xmlItem: 'client',
  },
  total: { $sum: 'amount' },
};

const reader = open('statements.jzm');
console.log(toJSON(reader, { shape, pretty: true }));
console.log(toXML(reader, { shape, filter: { client: 'C2' } })); // one client only
console.log(Object.keys(shapeSchema(reader, shape).properties)); // a JSON Schema of the output
{
  "title": "March statements",
  "clients": [
    {
      "id": "C1",
      "name": "ABC Corp",
      "address": "10 Test Street",
      "balance": 1250.5,
      "lines": 2,
      "transactions": [
        {
          "date": "2025-03-01T00:00:00.000Z",
          "amount": 1000
        },
        {
          "date": "2025-03-04T00:00:00.000Z",
          "amount": 250.5
        }
      ]
    },
    {
      "id": "C2",
      "name": "Test Ltd",
      "address": "2 Test Place",
      "balance": 20000,
      "lines": 1,
      "transactions": [
        {
          "date": "2025-03-02T00:00:00.000Z",
          "amount": 20000
        }
      ]
    }
  ],
  "total": 21250.5
}

<?xml version="1.0" encoding="UTF-8"?>
<export>
  <title>March statements</title>
  <clients>
    <client>
      <id>C2</id>
      <name>Test Ltd</name>
      <address>2 Test Place</address>
      <balance>20000</balance>
      <lines>1</lines>
      <transactions>
        <transaction>
          <date>2025-03-02T00:00:00.000Z</date>
          <amount>20000</amount>
        </transaction>
      </transactions>
    </client>
  </clients>
  <total>20000</total>
</export>

[ 'title', 'clients', 'total' ]

The same in .NET, with the same shape text:

using Jazmin;
using Jazmin.Formats;
using Jazmin.Query;
using System.Text.Json.Nodes;

using (var writer = JazminWriter.Create("statements.jzm",
    [new("client", JazminType.String), new("clientName", JazminType.String), new("amount", JazminType.Float)],
    new JazminWriteOptions { SortedBy = ["client"] }))
{
    writer.WriteValues("C1", "ABC Corp", 1000.0);
    writer.WriteValues("C1", "ABC Corp", 250.5);
    writer.WriteValues("C2", "Test Ltd", 20000.0);
}

var shape = JazminShape.Parse("""
{ "clients": { "$rows": { "id": "client", "name": "clientName", "balance": { "$sum": "amount" }, "lines": { "$count": true } },
               "$groupBy": "client" },
  "total": { "$sum": "amount" } }
""");
using var reader = JazminReader.Open("statements.jzm");
shape.Validate(reader); // optional: fail early with the place of any mistake
Console.WriteLine(shape.ToJson(reader));
Console.WriteLine(shape.ToJson(reader, JazminFilter.Eq("client", "C1")));

try { JazminShape.Parse("""{ "name": "clientNme" }""").Validate(reader); }
catch (JazminValidationException e) { Console.WriteLine(e.Message); }
{"clients":[{"id":"C1","name":"ABC Corp","balance":1250.5,"lines":2},{"id":"C2","name":"Test Ltd","balance":20000,"lines":1}],"total":21250.5}
{"clients":[{"id":"C1","name":"ABC Corp","balance":1250.5,"lines":2}],"total":1250.5}
Shape at name: unknown or hidden column 'clientNme'

What a shape can contain

In the shape Gives Where
"column" The column's value; in a set of rows, the first row's Anywhere
12, true, null That literal Anywhere
{ "$value": "text" } A constant, any JSON (use it for literal text) Anywhere
{ "$meta": "title" } A metadata member of the file Anywhere
{ ... } A nested object Anywhere
{ "$count": true } The number of rows In a set
{ "$sum": "c" }, { "$min": "c" }, { "$max": "c" } Total, smallest, largest (null if no values) In a set
{ "$rows": ... } A list In a set

A set is the whole file (or the export's filter), or one group of a $groupBy list. Inside a list without $groupBy, each item is one row, so aggregates and nested lists aren't allowed there.

List option Meaning
$filter Only rows matching this filter; enclosing lists' and groups' filters apply too
$groupBy One item per distinct value (or combination, given an array), in order of first appearance
$sort For example ["-amount", "date"]: - means descending; nulls first
$limit At most this many items
$xmlItem XML element name of each item (default item)

Sums of int and decimal columns are exact, even beyond 2^53. A decimal sum keeps the largest number of decimals among its inputs.

Typical shapes

import { open, toJSON, write } from '@smithsoft-studios/jazmin';

write('sales.jzm', [
  { client: 'C1', country: 'ZA', month: '2026-01', amount: 100 },
  { client: 'C1', country: 'ZA', month: '2026-01', amount: 300 },
  { client: 'C1', country: 'ZA', month: '2026-02', amount: 50 },
  { client: 'C2', country: 'NA', month: '2026-01', amount: 700 },
  { client: 'C2', country: 'NA', month: '2026-02', amount: 20 },
  { client: 'C3', country: 'ZA', month: '2026-02', amount: 90 },
]);
const reader = open('sales.jzm');
const shapes = {
  'distinct values': { countries: { $rows: 'country', $groupBy: 'country', $sort: ['country'] } },
  'top 2 per client': { $rows: { client: 'client', largest: { $rows: 'amount', $sort: ['-amount'], $limit: 2 } }, $groupBy: 'client' },
  'totals per client and month': {
    $rows: { client: 'client', months: { $rows: { month: 'month', total: { $sum: 'amount' } }, $groupBy: 'month' } },
    $groupBy: 'client', $filter: { country: 'ZA' },
  },
  'a constant and a count': { report: { $value: 'Sales' }, rows: { $count: true }, biggest: { $max: 'amount' } },
};
for (const [name, shape] of Object.entries(shapes)) console.log(`${name}:`, toJSON(reader, { shape }));
distinct values: {"countries":["NA","ZA"]}
top 2 per client: [{"client":"C1","largest":[300,100]},{"client":"C2","largest":[700,20]},{"client":"C3","largest":[90]}]
totals per client and month: [{"client":"C1","months":[{"month":"2026-01","total":400},{"month":"2026-02","total":50}]},{"client":"C3","months":[{"month":"2026-02","total":90}]}]
a constant and a count: {"report":"Sales","rows":6,"biggest":700}

A statement: header plus lines. Export one client with a filter; the header values come from the first line:

{ "client": "client", "name": "clientName", "address": "address",
  "lines": { "$rows": { "date": "date", "description": "description", "amount": "amount" } },
  "balance": { "$sum": "amount" } }

Nest rows from other tables (1.3.0)

Since 1.3.0. Version 1.2 refuses $from as an unknown list option.

In a file with several tables, a list can take its rows from another table, linked to the row it is written for. One shape then nests each customer's orders, each order's lines, and each line's product.

import { open, toJSON, write } from '@smithsoft-studios/jazmin';

write('shop.jzm', {
  customers: [{ id: 1, name: 'Ann' }, { id: 2, name: 'Bob' }],
  products: [{ sku: 'P1', name: 'Pen', price: '2.50' }, { sku: 'P2', name: 'Pad', price: '4.00' }],
  orders: [{ id: 10, customer_id: 1, total: '9.00' }, { id: 11, customer_id: 1, total: '4.00' }, { id: 12, customer_id: 2, total: '2.50' }],
  order_lines: [
    { customer_id: 1, order_id: 10, line: 1, sku: 'P1', quantity: 2 },
    { customer_id: 1, order_id: 10, line: 2, sku: 'P2', quantity: 1 },
    { customer_id: 1, order_id: 11, line: 1, sku: 'P2', quantity: 1 },
    { customer_id: 2, order_id: 12, line: 1, sku: 'P1', quantity: 1 },
  ],
}, {
  tables: [ // each linked table sorted by its link: every table is read once
    { name: 'customers', sortedBy: ['id'], columns: [{ name: 'id', type: 'int' }, { name: 'name', type: 'string' }] },
    { name: 'products', sortedBy: ['sku'], columns: [{ name: 'sku', type: 'string' }, { name: 'name', type: 'string' }, { name: 'price', type: 'decimal' }] },
    { name: 'orders', sortedBy: ['customer_id', 'id'], columns: [{ name: 'id', type: 'int' }, { name: 'customer_id', type: 'int' }, { name: 'total', type: 'decimal' }] },
    { name: 'order_lines', sortedBy: ['customer_id', 'order_id', 'line'], columns: [
      { name: 'customer_id', type: 'int' }, { name: 'order_id', type: 'int' }, { name: 'line', type: 'int' },
      { name: 'sku', type: 'string' }, { name: 'quantity', type: 'int' }] },
  ],
});

const shape = { $rows: {
  id: 'id', name: 'name',
  orders: { $from: 'orders', $on: { customer_id: 'id' }, $rows: {
    id: 'id', total: 'total',
    lines: { $from: 'order_lines', $on: { customer_id: 'customer_id', order_id: 'id' }, $rows: {
      quantity: 'quantity',
      product: { $from: 'products', $on: { sku: 'sku' }, $one: { name: 'name', price: 'price' } } } },
    lineCount: { $from: 'order_lines', $on: { customer_id: 'customer_id', order_id: 'id' }, $one: { $count: true } } } } } };

const customers = open('shop.jzm');                                  // the first table: customers
console.log(toJSON(customers, { shape, filter: { id: 1 } }));        // one customer, with all their details
const summary = { name: 'name', spent: { $from: 'orders', $on: { customer_id: 'id' }, $one: { $sum: 'total' } } };
console.log(toJSON(customers, { shape: summary, filter: { id: 1 } })); // a link in the root follows the root set
[{"id":1,"name":"Ann","orders":[{"id":10,"total":9.00,"lines":[{"quantity":2,"product":{"name":"Pen","price":2.50}},{"quantity":1,"product":{"name":"Pad","price":4.00}}],"lineCount":2},{"id":11,"total":4.00,"lines":[{"quantity":1,"product":{"name":"Pad","price":4.00}}],"lineCount":1}]}]
{"name":"Ann","spent":13.00}

using Jazmin;
using Jazmin.Formats;
using Jazmin.Query;

JazminColumn Int(string name) => new(name, JazminType.Int);
JazminColumn Text(string name) => new(name, JazminType.String);
using (var writer = JazminWriter.Create("shop.jzm", new JazminWriteOptions
{
    Tables =
    [
        new JazminTable("customers", [Int("id"), Text("name")]) { SortedBy = ["id"] },
        new JazminTable("orders", [Int("id"), Int("customer_id"), new JazminColumn("total", JazminType.Decimal)]) { SortedBy = ["customer_id", "id"] },
        new JazminTable("order_lines", [Int("customer_id"), Int("order_id"), Int("line"), Text("sku")]) { SortedBy = ["customer_id", "order_id", "line"] },
    ],
}))
{
    writer.WriteValues(1L, "Ann");
    writer.WriteValues(2L, "Bob");
    writer.StartTable("orders");
    writer.WriteValues(10L, 1L, "9.00");
    writer.WriteValues(11L, 1L, "4.00");
    writer.WriteValues(12L, 2L, "2.50");
    writer.StartTable("order_lines");
    writer.WriteValues(1L, 10L, 1L, "P1");
    writer.WriteValues(1L, 10L, 2L, "P2");
    writer.WriteValues(1L, 11L, 1L, "P2");
    writer.WriteValues(2L, 12L, 1L, "P1");
}

var shape = JazminShape.Parse("""
    { "$rows": { "id": "id", "name": "name",
        "orders": { "$from": "orders", "$on": { "customer_id": "id" },
          "$rows": { "id": "id", "skus": { "$from": "order_lines", "$on": { "customer_id": "customer_id", "order_id": "id" }, "$rows": "sku" } } },
        "spent": { "$from": "orders", "$on": { "customer_id": "id" }, "$one": { "$sum": "total" } } } }
    """);
using var customers = JazminReader.Open("shop.jzm");
Console.WriteLine(shape.ToJson(customers));
Console.WriteLine(shape.ToJson(customers, JazminFilter.Eq("id", 2L)));
using (var output = File.Create("shop.json")) shape.WriteJson(customers, output); // every customer, streamed to disk
[{"id":1,"name":"Ann","orders":[{"id":10,"skus":["P1","P2"]},{"id":11,"skus":["P2"]}],"spent":13.00},{"id":2,"name":"Bob","orders":[{"id":12,"skus":["P1"]}],"spent":2.50}]
[{"id":2,"name":"Bob","orders":[{"id":12,"skus":["P1"]}],"spent":2.50}]
In the shape Meaning
$from The table the rows come from. Inside the list, column names are that table's
$on Pairs { linked column: column of the row it is written for }. Rows are linked when every pair is equal, as in filters: decimals by value, and a null links nothing. Paired columns have the same type
$rows A list of the linked rows, in the linked table's order unless $sort says otherwise. $filter (the linked table's columns), $groupBy, $sort, $limit and $xmlItem work as on other lists. Unlike other lists, it is allowed in a one-row item: that is how a row's details nest
$one The linked rows as one set: a column gives the first linked row's value, and totals cover all of them. null when nothing is linked (in XML, omitted)
  • The export's filter applies to the reader's own table: it picks the customers, and the links follow.
  • Only the columns the shape uses are read from each table.
  • Shared files: each linked table is read with the same key, so a key sees only its partitions and column groups, in every table.
  • Checked up front, as any shape: an unknown table, an unknown or hidden column, or paired columns of different types are named before any data is read.

Sort each linked table by its link

When a linked table is sorted by the columns of its link, and its parents arrive in that order, it is read once, front to back, in step with its parents, holding only the current parent's rows. Parents arrive in that order when their table is sorted by the columns they pair with, and their list has no $sort. That is why the file above sorts orders by customer_id, id, and lines by customer_id, order_id, line, linked on both columns.

  • Small linked tables, such as products, are read once and kept by key: up to 10,000 rows with priority memory, 100,000 by default, and 1,000,000 with speed.
  • Otherwise, parents are written in batches (2,500 with memory, 10,000 by default, 20,000 with speed), with one query per link and batch. When the linked rows lie scattered, each batch reads about the whole linked table.

100,000 customers, 1,000,000 orders and 2,498,976 lines; customers with their orders, lines and each line's product (360 MB of JSON). Exporting the three tables one by one, with the same columns, takes 2.6 s (70 MB) in .NET and 4.8 s (200 MB) in Node.

File .NET Node
Sorted by the links, as above 4.5 s, 84 MB 7.2 s, 220 MB
The same, one customer / 1,000 customers 0.17 s / 0.44 s 0.08 s / 0.13 s
Orders and lines in time order (batches), by default 19 s, 238 MB 27 s, 578 MB
The same with priority memory 37 s, 100 MB 64 s, 364 MB

Describe the output with JSON Schema

shapeSchema(reader, shape) (.NET shape.ToJsonSchema(reader)) gives a JSON Schema (draft 2020-12) of the export, with each member's type and whether it can be null, from the column definitions. Publish it for the consumers of your export.

Keep large exports fast and small

  • Sort the file by the column you group by (sortedBy: ['client']). Groups then arrive one after another, and one pass writes each group as soon as it is complete, holding only that group's rows. On 1M transactions for 10,000 clients, with nested lines: 2.3 s in Node, 1.8 s with a 66 MB peak in .NET.
  • Without that sort, groups whose items use only first values and totals still take one pass, and memory grows with the number of groups. Groups with nested lists are collected in batches of about 100,000 rows. The same export took 8.7 s in Node, and 5.6 s with a 153 MB peak in .NET.
  • From 1.3.0, Node writes shapes 2–2.6× faster: the sorted export above takes 1.9 s with a 147 MB peak. A reader opened with priority: 'speed' collects unsorted groups in batches of 1,000,000 rows: 4.0 s (559 MB) in Node and 5.3 s (381 MB) in .NET.
  • $sort holds its list in memory before writing. For very large lists, rely on the file's own order.
  • Stream to disk with exportFile(reader, 'json' | 'xml', path, { shape }) (.NET WriteJson / WriteXml) instead of building a string.

Save shapes in the file (1.5.0)

A file can carry its own export shapes, each under a name. Whoever opens it can export "Monthly totals" without writing the shape again: the viewer lists them under Export ▾, the command-line tool takes --shape "Monthly totals", and code passes the name where it would pass a shape.

write('statement.jzm', rows, {
  columns, key,
  shapes: [
    { name: 'Monthly totals', description: 'Spending per category', default: true, shape: totals },
    { name: 'Fees', table: 'fees', shape: { $rows: { date: 'date', fee: 'amount' } } },
  ],
});
const reader = open('statement.jzm', { key });
reader.shapes;                                 // the shapes this key can use
toJSON(reader, { shape: 'Monthly totals' });  // a saved shape, run on its own table
update('statement.jzm', { key, addShapes: [fees], removeShapes: ['Old'] }); // append() too
var totals = new JazminSavedShape("Monthly totals", JsonNode.Parse(totalsJson)!.AsObject()) { IsDefault = true };
// JazminWriteOptions.Shapes = [totals]; JazminUpdate / JazminAppend: AddShapes, RemoveShapes
using var reader = JazminReader.Open("statement.jzm", new JazminReadOptions { Key = key });
var json = JazminShape.FromFile(reader, "Monthly totals").ToJson(reader);
  • A saved shape has a name (unique in the file), the shape, and optionally a description, default (offered first) and table (the table it reads; the first when left out).
  • It stays with the file: append, update, compaction and key rotation keep it, encrypted like embedded files.
  • Shared files: each key sees only the shapes it can use. groups says which keys see a shape, as for embedded files: '*' (everyone, the default), partition names, or named file groups. Every key that would see a shape must see every column it uses; otherwise the file isn't written, and a later grant that would show the shape to such a key is refused, naming the key and the column. So different people can see different shapes in the same file.
  • Older versions (1.0 to 1.4) read such files as before, without the shapes; when they append to, update or compact one, the shapes are left out.

JAZMIN 1.2 · file format 1.0 · Repository · npm · NuGet · MIT license · Examples in this wiki are run and checked by tools/check-examples.mjs

⚠️ **GitHub.com Fallback** ⚠️