New in 1.4. Files with nested columns need JAZMIN 1.4 or later to read: versions 1.0 to 1.3 refuse them with an "unknown type" error, and never misread them. Files without nested columns are unchanged.

Lists and objects are normally stored as JSON text in a json column. Nested columns store them as columns of their own fields instead: a list column with an item, and an object column with fields, to any depth. Each field is then stored as a column is (numbers as numbers, repeated text once per chunk), and read back straight from the stored values, with no JSON in between. They're opt-in: JSON stays the default.

Measured on 5,000 companies, each with 3 departments of 6 employees with 3 projects each:

Give a list column an item, and an object column its fields. Each is defined as a column is, so they nest:

import { open, write } from '@smithsoft-studios/jazmin';

const columns = [
  { name: 'id', type: 'int', nullable: false },
  { name: 'name', type: 'string' },
  {
    name: 'departments', type: 'list', item: {
      type: 'object', fields: [
        { name: 'name', type: 'string' },
        { name: 'budget', type: 'decimal' },
        { name: 'staff', type: 'list', item: { type: 'string', nullable: false } },
      ],
    },
  },
  { name: 'head', type: 'object', fields: [{ name: 'city', type: 'string' }] },
];
write('companies.jzm', [
  { id: 1, name: 'Acme', departments: [{ name: 'Sales', budget: '1200.50', staff: ['Ann', 'Ben'] }, { name: 'Build', budget: '800', staff: [] }], head: { city: 'Durban' } },
  { id: 2, name: 'Bolt', departments: [{ name: 'Sales', budget: '300', staff: ['Cleo'] }], head: { city: 'Gqeberha' } },
  { id: 3, name: 'Cobalt', departments: null, head: null },
], { columns });

const reader = open('companies.jzm');
console.log([...reader.find({ id: 1 })][0].departments);
reader.close();

[
  { name: 'Sales', budget: '1200.50', staff: [ 'Ann', 'Ben' ] },
  { name: 'Build', budget: '800', staff: [] }
]

Values come back in the forms of their types (Date, Buffer, decimal text, bigint), with every field. The browser reader and writer, and the viewer, read and write nested columns too.

NestedColumns = true stores lists, arrays and classes as nested columns. [JazminNested] turns it on for one property, and [JazminNested(false)] keeps a property as JSON when the setting is on. Reading needs no settings: the file says how each column is stored.

using Jazmin;
using Jazmin.Serialization;

var companies = new List<Company>
{
    new() { Id = 1, Name = "Acme", Departments = [new() { Name = "Sales", Budget = 1200.50m, Staff = ["Ann", "Ben"] }, new() { Name = "Build", Budget = 800m, Staff = [] }] },
    new() { Id = 2, Name = "Bolt", Departments = [new() { Name = "Sales", Budget = 300m, Staff = ["Cleo"] }] },
};
File.WriteAllBytes("companies.jzm", JazminConvert.SerializeObject(companies, new JazminSerializerSettings { NestedColumns = true }));

using var reader = JazminReader.Open("companies.jzm");
Console.WriteLine(reader.Columns.Single(c => c.Name == "Departments"));
var back = JazminConvert.DeserializeObject<List<Company>>(File.ReadAllBytes("companies.jzm"))!;
Console.WriteLine(string.Join(", ", back[0].Departments.Select(d => $"{d.Name} {d.Budget} [{string.Join(" ", d.Staff)}]")));

public class Company { public int Id { get; set; } public string Name { get; set; } = ""; public List<Department> Departments { get; set; } = []; }
public class Department { public string Name { get; set; } = ""; public decimal Budget { get; set; } public List<string> Staff { get; set; } = []; }

Departments: list<object{Name: string?, Budget: decimal, Staff: list<string?>?}?>?
Sales 1200.50 [Ann Ben], Build 800 []

What can be nested: arrays, List<T> and the interfaces a List<T> fits (IList<T>, IReadOnlyList<T>, IEnumerable<T> and so on), and classes with public properties, up to 64 levels deep. What stays JSON: other collections such as HashSet<T>, dictionaries, object, JsonNode and JsonElement, polymorphic types ([JsonDerivedType]), and a class inside itself. [JazminNested] on such a property is an error.

Three operators check nested columns, each with a filter of its own (Filter Language):

• any: at least one item of the list matches. The conditions of one any apply to the same item.
• all: every item matches. An empty list does.
• match: the object's fields match.
• A null list or object matches nothing. For items that are not objects, the condition is on the item itself: { "tags": { "any": "vip" } }.

import { open, write } from '@smithsoft-studios/jazmin';

const text = (name) => ({ name, type: 'string' });
write('companies.jzm', [
  { name: 'Acme', departments: [{ name: 'Sales', budget: '1200.50', staff: ['Ann', 'Ben'] }, { name: 'Build', budget: '800', staff: [] }], head: { city: 'Durban' } },
  { name: 'Bolt', departments: [{ name: 'Sales', budget: '300', staff: ['Cleo'] }], head: { city: 'Gqeberha' } },
  { name: 'Cobalt', departments: null, head: null },
], {
  columns: [
    text('name'),
    { name: 'departments', type: 'list', item: { type: 'object', fields: [text('name'), { name: 'budget', type: 'decimal' }, { name: 'staff', type: 'list', item: { type: 'string' } }] } },
    { name: 'head', type: 'object', fields: [text('city')] },
  ],
});

const reader = open('companies.jzm');
const names = (filter) => [...reader.find(filter, { select: ['name'] })].map((r) => r.name);
console.log(names({ departments: { any: { staff: { any: 'Ben' } } } }));
console.log(names({ departments: { any: { name: 'Sales', budget: { gt: 1000 } } } })); // the same department
console.log(names({ departments: { all: { staff: { any: 'Cleo' } } } }));
console.log(names({ head: { match: { city: 'Durban' } } }));
console.log(reader.count({ departments: { any: { name: 'Build' } } }));
reader.close();

[ 'Acme' ]
[ 'Acme' ]
[ 'Bolt' ]
[ 'Acme' ]
1

• They nest: { "departments": { "any": { "employees": { "any": { "role": "Lead" } } } } }.
• Only the fields a filter reads are decoded for a nested column the rows don't return (a select without it, or count).
• Chunks no item can match are skipped. Files keep min/max statistics of the fields inside nested columns, so any and match skip chunks the way conditions on other columns do. They help when a field's values are grouped by chunk, as ids or dates that grow with the rows are; all and not read every chunk.
• On the 5,000 companies in Node.js, the names of those with an employee of a given id take 4.4 ms and 61 MB instead of 186 ms and 197 MB: 35–40 ms from reading only the filter's fields, the rest from skipped chunks.
• In .NET, JazminFilter.Any, All and Match build the same filters, and JazminFilter.Itself names the item of a list of plain values: JazminFilter.Any("Tags", JazminFilter.Eq(JazminFilter.Itself, "vip")).

AsQueryable<T>() reads only the nested fields a query uses, at any depth: the report in the table above reads 6 of 23 fields. Conditions on nested columns become the filters above, checked as the file is decoded, so only matching rows' objects are built:

using Jazmin;
using Jazmin.Serialization;

File.WriteAllBytes("companies.jzm", JazminConvert.SerializeObject(new List<Company>
{
    new() { Id = 1, Name = "Acme", Departments = [new() { Name = "Sales", Budget = 1200.50m, Staff = ["Ann", "Ben"] }, new() { Name = "Build", Budget = 800m, Staff = [] }] },
    new() { Id = 2, Name = "Bolt", Departments = [new() { Name = "Sales", Budget = 300m, Staff = ["Cleo"] }] },
}, new JazminSerializerSettings { NestedColumns = true }));

using var reader = JazminReader.Open("companies.jzm");
var companies = reader.AsQueryable<Company>(); // no settings needed to read
Console.WriteLine(string.Join(", ", companies.Where(c => c.Departments.Any(d => d.Staff.Contains("Ben"))).Select(c => c.Name)));
Console.WriteLine(companies.Count(c => c.Departments.Any(d => d.Name == "Sales" && d.Budget > 1000m)));

public class Company { public int Id { get; set; } public string Name { get; set; } = ""; public List<Department> Departments { get; set; } = []; }
public class Department { public string Name { get; set; } = ""; public decimal Budget { get; set; } public List<string> Staff { get; set; } = []; }

Acme
1

• Parts a filter can't hold (a decimal comparison, a method call) are checked on each object, so results are always those of LINQ to Objects. Where LINQ to Objects would throw on a null list or object, the row simply doesn't match.
• l != null && before an item's conditions is fine: a null item matches no item's filter.
• On the 5,000 companies, the names of those with an employee of a given id take 3 ms and 1 MB instead of 25 ms and 23 MB, and the companies with a lead among the first employees of their Build department, returned whole, 4 ms and 2 MB instead of 167 ms and 143 MB. Both find their values in a few chunks; with values spread over every chunk, reading only the condition's fields still takes them to 13 ms and 91 ms.
• Objects a query returns whole, passes to a method, compares or groups by are read whole.

Appending can add fields at the end of an object: rows written before read them as null (or the member's default). .NET's JazminFile.Append and Update add the new members of your classes; for rows given as JSON, and in JavaScript, pass the grown definitions in columns:

// not run: a continuation of the JavaScript example above
append('companies.jzm', { insert: newRows, columns: [departmentsWithHeadcount] });

Removing, renaming or retyping a field needs the file written again. A field added later must allow null.

• Nested columns and their fields can't be indexed or sorted by.
• A bad value refuses its row whole, wherever it is: the error names its place, as in Column 'departments[].budget'.
• Dates inside nested columns are stored as datetime columns are: UTC, to the millisecond.
• JSON, CSV and XML output show nested values as JSON.
• Very small files can be slightly larger, because each field has a few bytes of its own.
• The file format: a list or object column is stored with encoding 5: one stream of list lengths per level and one stream per field, inside the row's chunk, so a lookup still reads one chunk. The reader feature is nested-columns. The statistics of the fields inside are an optional addition (0.4% of the file).