DotNet Serializer and LINQ - SmithSoft-Studios/JAZMIN GitHub Wiki

.NET Serializer and LINQ

The Jazmin package gives .NET three levels of control:

Level API Use when
Newtonsoft-style one-liners JazminConvert.SerializeObject(list), DeserializeObject<T>(bytes) Replacing JSON serialization with minimal changes
Serializer with settings new JazminSerializer(settings).Serialize(path, items), DeserializeEnumerable<T>(stream) Files, encryption, indexes, streaming
Reader with LINQ reader.Query<T>(predicate), reader.AsQueryable<T>() Querying a file without loading it 
// fragment: the namespaces used on this page
using Jazmin;               // JazminReader, JazminWriter, JazminKey, JazminFile
using Jazmin.Query;         // JazminFilter
using Jazmin.Serialization; // JazminConvert, JazminSerializer, attributes, settings

Shaping the schema with attributes

The column list comes from your class's public properties. Attributes refine it.

using System.ComponentModel;
using Jazmin;
using Jazmin.Serialization;

var customers = new List<Customer>
{
    new() { Id = 1, Name = "Ann Johnson", Country = "ZA", Tier = Tier.Gold, Balance = 1520.75m,
            Joined = new DateTime(2024, 1, 2, 0, 0, 0, DateTimeKind.Utc), Tags = ["vip"], PasswordHash = "secret" },
};
byte[] bytes = JazminConvert.SerializeObject(customers);

using var reader = JazminReader.Open(bytes);
foreach (var c in reader.Columns) Console.WriteLine($"{c,-22} {c.Description}");
Console.WriteLine(string.Join(", ", reader.Indexes.Select(i => $"{i.Column}: {i.Kind}")));

var back = JazminConvert.DeserializeObject<List<Customer>>(bytes)![0];
Console.WriteLine($"{back.Name}, {back.Tier}, {back.Balance}, tags [{string.Join(", ", back.Tags)}], hash {back.PasswordHash ?? "(not stored)"}");

public enum Tier { Standard, Gold }

public class Customer
{
    [JazminIndex] // a sorted index
    public int Id { get; set; }

    [JazminIndex(JazminIndexKind.Sorted, JazminIndexKind.Trigram)]
    [Description("Full name")] // System.ComponentModel
    public string Name { get; set; } = "";

    [JazminProperty("country_code", Description = "ISO 3166 alpha-2")]
    public string? Country { get; set; }

    public Tier Tier { get; set; }        // enums are stored as their names
    public decimal Balance { get; set; }  // exact
    public DateTime Joined { get; set; }
    public List<string> Tags { get; set; } = []; // stored in a json column

    [JazminIgnore] // or System.Text.Json's [JsonIgnore]
    public string? PasswordHash { get; set; }
}
Id: int
Name: string?          Full name
country_code: string?  ISO 3166 alpha-2
Tier: string
Balance: decimal
Joined: datetime
Tags: json?
Id: Sorted, Name: Sorted, Name: Trigram
Ann Johnson, Gold, 1520.75, tags [vip], hash (not stored)
.NET property type Column type
bool bool
integer types int
float, double float
decimal decimal
string, char, Guid, TimeSpan, TimeOnly, enums string
DateTime, DateTimeOffset, DateOnly datetime
byte[] binary
anything else (lists, dictionaries, nested classes) json
  • Nullability: value types (int, decimal, DateTime, enums) make non-nullable columns unless declared nullable (int?). Reference types (string, lists, classes) make nullable columns.
  • Names: [JazminProperty("name")] or System.Text.Json's [JsonPropertyName]. Column names are matched case-insensitively when reading, as Newtonsoft does.
  • Records and immutable classes are created through their constructor.

Settings

var settings = new JazminSerializerSettings
{
    Key = JazminKey.Parse(Environment.GetEnvironmentVariable("JAZMIN_KEY")!), // or Password = "..."
    Codec = JazminCodec.Brotli,
    ChunkRows = 4096,
    Metadata = new JsonObject { ["source"] = "crm" },
    Indexes = new() { ["Country"] = [JazminIndexKind.Sorted] },  // indexes without attributes
    Formatting = Formatting.Indented,                    // JSON output (ToJson)
    NullValueHandling = NullValueHandling.Ignore,        // JSON output (ToJson)
    NamingStrategy = JazminNamingStrategy.CamelCase,     // FirstName -> firstName (also SnakeCase, KebabCase)
    Converters = [new MoneyConverter()],                 // your own types
    DefaultValueHandling = DefaultValueHandling.IgnoreAndPopulate, // defaults stored as null, filled in on reading
    PreserveReferencesHandling = PreserveReferencesHandling.Objects, // a repeated object is stored once ($id / $ref)
};

Settings that change how types map to columns (naming, converters, default values, references) are cached per settings instance. Create the settings once and reuse them, as with Newtonsoft.

Naming strategies and custom converters

A converter stores a type of your own in a column, like Newtonsoft's JsonConverter. Register it in Converters, or put [JazminConverter(typeof(...))] on a property or on the type.

using Jazmin;
using Jazmin.Serialization;

var settings = new JazminSerializerSettings
{
    NamingStrategy = JazminNamingStrategy.SnakeCase,
    Converters = [new MoneyConverter()],
};
var invoices = new[] { new Invoice { InvoiceNumber = "INV-1", TotalDue = new Money(152075, "ZAR") } };
byte[] bytes = JazminConvert.SerializeObject(invoices, settings);

using var reader = JazminReader.Open(bytes);
Console.WriteLine(string.Join(", ", reader.Columns));
Console.WriteLine(reader.Get(0)["total_due"]);
var back = JazminConvert.DeserializeObject<List<Invoice>>(bytes, settings)![0];
Console.WriteLine($"{back.InvoiceNumber}: {back.TotalDue}");

public readonly record struct Money(long Cents, string Currency);

public sealed class MoneyConverter : JazminConverter<Money>
{
    public override JazminType ColumnType => JazminType.String; // how the column stores it
    public override object? Write(Money value) => $"{value.Cents} {value.Currency}";
    public override Money Read(object stored)
    {
        var parts = ((string)stored).Split(' ');
        return new Money(long.Parse(parts[0]), parts[1]);
    }
}

public class Invoice
{
    public string InvoiceNumber { get; set; } = "";
    public Money TotalDue { get; set; }
}
invoice_number: string?, total_due: string
152075 ZAR
INV-1: Money { Cents = 152075, Currency = ZAR }

Filters and LINQ conditions on a converted property run in memory after reading, because the file stores the converted form.

Polymorphic types

Opt-in, with a closed list: name the derived types on the base type with System.Text.Json's [JsonDerivedType]. Each row records its type in a $type column.

using System.Text.Json.Serialization;
using Jazmin;
using Jazmin.Serialization;

List<Animal> animals = [new Dog { Name = "Rex", Barks = true }, new Cat { Name = "Tom", Lives = 9 }];
byte[] bytes = JazminConvert.SerializeObject(animals);
foreach (var a in JazminConvert.DeserializeObject<List<Animal>>(bytes)!)
    Console.WriteLine(a switch { Dog d => $"Dog {d.Name}, barks: {d.Barks}", Cat c => $"Cat {c.Name}, lives: {c.Lives}", _ => "?" });

[JsonDerivedType(typeof(Dog), "dog")]
[JsonDerivedType(typeof(Cat), "cat")]
public abstract class Animal { public string Name { get; set; } = ""; }
public class Dog : Animal { public bool Barks { get; set; } }
public class Cat : Animal { public int Lives { get; set; } }
Dog Rex, barks: True
Cat Tom, lives: 9

Newtonsoft's TypeNameHandling loads whatever type the data names, which has caused serious security bugs. JAZMIN creates only the types listed on the base type, and refuses any other name.

Querying with LINQ

Query<T>: a predicate, translated to indexes

reader.Query<T>(predicate) always returns exactly what the predicate selects. The library translates as much of the predicate as it can into an index-aware filter, then runs your compiled predicate on each candidate. Translation affects only speed, never results.

Expression Uses indexes and statistics
==, !=, <, <=, >, >= against constants or captured variables (bool, numbers, strings, enums, dates) โœ”
&&, ||, ! on comparisons โœ”
x.Name.Contains("a"), x.Name.StartsWith("a") โœ”
list.Contains(x.Country) โœ”
x.IsActive (a bool property) โœ”
decimal comparisons, method calls, arithmetic (x.Id % 7 == 0) โœ˜ scanned, still correct

AsQueryable<T>(): the whole query in the reader (New in 1.2)

Write the query with ordinary LINQ operators. The reader runs as much of it as it can, so less of the file is read and fewer objects are built. Results are always those LINQ gives over a list of the same objects.

using Jazmin;
using Jazmin.Serialization;

var regions = new[] { "ZA", "NA", "BW", "KE" };
var all = Enumerable.Range(1, 100_000).Select(i => new Order
{
    Id = i, Region = regions[i % 4], Customer = $"C-{i * 7919 % 5000:D4}", Amount = i % 1000 / 4m, Paid = i % 5 != 0,
});
new JazminSerializer(new JazminSerializerSettings { Indexes = new() { ["Customer"] = [JazminIndexKind.Sorted] } })
    .Serialize("orders.jzm", all);

using var reader = JazminReader.Open("orders.jzm");
var orders = reader.AsQueryable<Order>();

var page = orders.Where(o => o.Region == "ZA").Skip(40).Take(3).Select(o => o.Id).ToList();  // filter, offset, limit in the reader
int za = orders.Count(o => o.Region == "ZA");                                                 // no objects built
bool any = orders.Any(o => o.Customer == "C-1042");                                           // an index lookup
var lines = orders.Where(o => !o.Paid && o.Amount > 248m).Select(o => new { o.Id, o.Amount }).Take(2).ToList(); // reads 3 columns
decimal total = orders.Where(o => o.Region == "NA").Sum(o => o.Amount);                       // reads 2 columns
var top = orders.Where(o => o.Region == "KE").OrderByDescending(o => o.Amount).ThenBy(o => o.Id).Select(o => o.Id).Take(3).ToList(); // sorting runs in memory

Console.WriteLine($"page: {string.Join(", ", page)}");
Console.WriteLine($"ZA orders: {za}, any for C-1042: {any}");
Console.WriteLine($"unpaid over 248: {string.Join("; ", lines)}");
Console.WriteLine($"NA total: {total}");
Console.WriteLine($"top KE: {string.Join(", ", top)}");

public class Order
{
    public int Id { get; set; }
    public string Region { get; set; } = "";
    public string Customer { get; set; } = "";
    public decimal Amount { get; set; }
    public bool Paid { get; set; }
}
page: 164, 168, 172
ZA orders: 25000, any for C-1042: True
unpaid over 248: { Id = 995, Amount = 248.75 }; { Id = 1995, Amount = 248.75 }
NA total: 3118750.00
top KE: 999, 1999, 2999
Operator What the reader does
Where, and the condition of Count, Any, First, Single, Last Becomes a filter, as in Query<T>: indexes and chunk statistics, with parts it can't translate checked on each object
Skip, Take The filter's offset and limit: whole chunks before the offset are skipped unread when nothing is checked per object
Count, LongCount, Any Answered without building objects, from indexes or chunk statistics where the filter allows
OrderBy / ThenBy along the file's sortedBy columns Nothing to do: rows are stored in that order (ascending, non-nullable columns; strings ordinal)
Select; Sum, Average, Min, Max with a selector Only the columns they use are read
Everything from the first operator it can't do (OrderByDescending, GroupBy, Join, a Where after Take...) Runs in memory on the rows read, as LINQ to Objects. From 1.3.0 the rows are read with only the columns the whole query uses (below)

Across tables: lambda and query syntax

Each table of a file with several tables has its own AsQueryable<T>(). Lambda (method) syntax and query syntax both work, and the reader runs the filter, the columns and the order it can.

using Jazmin;
using Jazmin.Query;
using Jazmin.Serialization;

using (var writer = JazminWriter.Create("clients.jzm", new JazminWriteOptions
{
    Tables =
    [
        new JazminTable("clients", [new JazminColumn("clientId", JazminType.String), new JazminColumn("name", JazminType.String)]) { SortedBy = ["clientId"] },
        new JazminTable("transactions", [new JazminColumn("clientId", JazminType.String), new JazminColumn("amount", JazminType.Float)]) { SortedBy = ["clientId"] },
    ],
}))
{
    writer.WriteValues("C1", "Acme");
    writer.WriteValues("C2", "Bolt");
    writer.StartTable("transactions");
    writer.WriteValues("C1", 10.0);
    writer.WriteValues("C1", 5.0);
    writer.WriteValues("C2", 7.0);
}

using var clients = JazminReader.Open("clients.jzm");
using var transactions = clients.OpenTable("transactions");             // the same open file
var camel = new JazminSerializerSettings { NamingStrategy = JazminNamingStrategy.CamelCase }; // ClientId -> clientId
var clientRows = clients.AsQueryable<Client>(camel);
var transactionRows = transactions.AsQueryable<Transaction>(camel);

string name = clientRows.Where(c => c.ClientId == "C1").Select(c => c.Name).Single();       // lambda syntax
var amounts = (from t in transactionRows where t.ClientId == "C1" select t.Amount).ToList(); // query syntax
var ids = clientRows.Where(c => c.Name.StartsWith("B")).Select(c => c.ClientId).ToList();
var theirs = transactionRows.Where(t => ids.Contains(t.ClientId)).ToList();                  // an `in` filter
var totals = from c in clientRows.ToList()                                                   // a join runs in memory
             join t in transactionRows.ToList() on c.ClientId equals t.ClientId into ts
             select $"{c.Name} {ts.Sum(t => t.Amount)}";

Console.WriteLine($"{name}: {string.Join(" + ", amounts)}");
Console.WriteLine($"clients starting with B: {theirs.Count} transaction(s)");
Console.WriteLine(string.Join(", ", totals));

public class Client
{
    public string ClientId { get; set; } = "";
    public string Name { get; set; } = "";
}

public class Transaction
{
    public string ClientId { get; set; } = "";
    public double Amount { get; set; }
}
Acme: 10 + 5
clients starting with B: 1 transaction(s)
Acme 15, Bolt 7

Sub-queries per row, read once (1.3.0)

LINQ runs a sub-query inside a lambda once per outer row. From 1.3.0, JAZMIN reads such a sub-query's table once instead:

  • Paired with the outer row: when the condition pairs a member with a value of the outer row (o.CustomerId == c.Id), the table is kept by that member, and each outer row takes its group from the lookup (a hash join).
  • Anything else runs as LINQ to Objects over the table's rows, read once.
  • Adaptive: the first rows still run their own queries, which use indexes, so a query for one customer never reads a whole table.
  • Memory cap: tables over 1,000,000 rows (100,000 with priority Memory, 10,000,000 with Speed) keep a query per row.
using Jazmin;
using Jazmin.Serialization;

using (var writer = JazminWriter.Create("shop.jzm", new JazminWriteOptions
{
    Tables =
    [
        new JazminTable("customers", [new JazminColumn("Id", JazminType.Int), new JazminColumn("Name", JazminType.String)]) { SortedBy = ["Id"] },
        new JazminTable("orders", [new JazminColumn("Id", JazminType.Int), new JazminColumn("CustomerId", JazminType.Int), new JazminColumn("Total", JazminType.Decimal)]),
    ],
}))
{
    for (var c = 1; c <= 3; c++) writer.WriteValues((long)c, $"Customer {c}");
    writer.StartTable("orders");
    for (var o = 1; o <= 8; o++) writer.WriteValues((long)o, (long)(1 + o % 3), $"{o * 10}.50");
}

using var customers = JazminReader.Open("shop.jzm");
using var ordersTable = customers.OpenTable("orders");
var orders = ordersTable.AsQueryable<Order>();
var report = customers.AsQueryable<Customer>()
    .Select(c => new
    {
        c.Name,
        Latest = orders.Where(o => o.CustomerId == c.Id).OrderByDescending(o => o.Id).Select(o => o.Id).FirstOrDefault(),
        Spent = orders.Where(o => o.CustomerId == c.Id).Sum(o => o.Total),
    })
    .ToList();
foreach (var line in report) Console.WriteLine(line);

public class Customer { public int Id { get; set; } public string Name { get; set; } = ""; }
public class Order { public int Id { get; set; } public int CustomerId { get; set; } public decimal Total { get; set; } }
{ Name = Customer 1, Latest = 6, Spent = 91.00 }
{ Name = Customer 2, Latest = 7, Spent = 121.50 }
{ Name = Customer 3, Latest = 8, Spent = 151.50 }

The results are those of LINQ to Objects. On 5,000 customers and 50,000 orders, the latest 5 orders of each customer, with their product and payment, take 0.43 s instead of 64 s. The same code over in-memory lists takes 2.1 s, because each customer scans every order.

A Join or GroupBy across tables runs in memory, as LINQ to Objects. For nested output, such as each client with their transactions, use an export shape with $from (1.3.0): it reads each table once when the tables are sorted by the link.

Measured on .NET 10, 200,000 rows ร— 300 columns, a type mapping 9 of them (1.1.0's Query<T> followed by LINQ to Objects, against 1.2's AsQueryable):

Query 1.1.0 AsQueryable
Every row as objects 3.2โ€“3.7 s 0.30 s
A condition and two columns (Where + Select) 3.6 s 0.35โ€“0.37 s
Count(condition) 3.3โ€“3.9 s, 63 MB 0.25 s, 56 MB
Skip(150_000).Take(10) 2.5โ€“2.7 s, 63 MB 0.05 s, 35 MB
OrderBy(o => o.Id).Take(10) (file sorted by id) 3.5 s, 100 MB 0.05 s, 35 MB
Sum(o => o.Qty) 3.2โ€“3.4 s 0.25 s
First(o => o.Id == 123_456) 0.08โ€“0.09 s 0.07 s
  • Typed rows read only the columns their type maps: a class of 9 properties over a 300-column file decodes 9.
  • Only the columns a query uses (1.3.0), wherever it uses them: see below.
  • Files written with PreserveReferencesHandling.Objects run the whole query in memory, because their rows refer to earlier rows.
  • Not thread-safe, like the reader. For async code, use QueryAsync<T> and RowsAsync<T>.

Only the columns a query uses (1.3.0)

From 1.3.0, AsQueryable<T>() reads only the columns of the members a query uses, wherever it uses them:

  • in Where, Select, SelectMany, GroupBy, Join and nested queries;
  • through anonymous objects and groups (g.OrderByDescending(x => x.Revenue).First().Project);
  • in a Join with another table, or the same one: each side reads only the columns used of it.

Before 1.3.0, only queries ending in Select or an aggregate read fewer columns.

A collection member (List<Department> below) is one JSON column. A query that uses it reads that column whole: every department, employee and project in it. To read nested details piece by piece, keep them in their own tables.

JSON columns, read straight into your types (1.3.0). A JSON column is deserialized from its stored UTF-8 directly into the property's type, with no text or JsonNode in between. An untyped row still gives a JsonNode, made when the value is first read. On 200,000 orders whose customer and items are JSON columns, the report in Performance went from 6.8 s and 1.6 GB allocated (1.2.0) to 3.1 s and 965 MB.

using Jazmin;
using Jazmin.Serialization;

var companies = new[] { "Acme", "Bolt" }.Select((name, c) => new Company
{
    Name = name,
    Founded = 1990 + c,
    Departments = [.. new[] { "Sales", "Build" }.Select((d, i) => new Department
    {
        Name = d,
        Employees = [.. Enumerable.Range(0, 2).Select(e => new Employee
        {
            Name = $"{name} {d} {e}",
            Role = e == 0 ? "Lead" : "Member",
            Salary = 1000m * (e + 1),
            Projects = [new() { Name = $"{d} {c}{e}", Category = i == 0 ? "Product" : "Service", Revenue = 6_000m * (c + i + e + 1) }],
        })],
    })],
}).ToList();
File.WriteAllBytes("companies.jzm", JazminConvert.SerializeObject(companies));

using var reader = JazminReader.Open("companies.jzm");
var report = reader.AsQueryable<Company>()                      // reads one column: Departments
    .SelectMany(c => c.Departments)
    .SelectMany(d => d.Employees, (d, e) => new { Department = d.Name, Employee = e })
    .SelectMany(x => x.Employee.Projects, (x, p) => new
    {
        Employee = x.Employee.Name,
        x.Department,
        x.Employee.Role,
        x.Employee.Salary,
        Project = p.Name,
        p.Category,
        p.Revenue,
    })
    .Where(x => x.Revenue > 10_000)
    .GroupBy(x => new { x.Department, x.Category })
    .Select(g => new
    {
        g.Key.Department,
        g.Key.Category,
        Revenue = g.Sum(x => x.Revenue),
        Employees = g.Select(x => x.Employee).Distinct().Count(),
        TopProject = g.OrderByDescending(x => x.Revenue).First().Project,
    })
    .OrderByDescending(x => x.Revenue);
foreach (var line in report) Console.WriteLine(line);

public class Company
{
    public string Name { get; set; } = "";
    public int Founded { get; set; }
    public List<Department> Departments { get; set; } = [];
}

public class Department
{
    public string Name { get; set; } = "";
    public List<Employee> Employees { get; set; } = [];
}

public class Employee
{
    public string Name { get; set; } = "";
    public string Role { get; set; } = "";
    public decimal Salary { get; set; }
    public List<Project> Projects { get; set; } = [];
}

public class Project
{
    public string Name { get; set; } = "";
    public string Category { get; set; } = "";
    public decimal Revenue { get; set; }
}
{ Department = Build, Category = Service, Revenue = 72000, Employees = 4, TopProject = Build 11 }
{ Department = Sales, Category = Product, Revenue = 42000, Employees = 3, TopProject = Sales 11 }

Every column is read when the rows themselves may be used in ways the query doesn't show:

  • you receive them, or call ToList() before the rest of the query;
  • they're passed to a method or formatted, or a method of the row is called;
  • they're compared or sorted whole (Distinct(), Union, a GroupBy key of whole rows), or cast;
  • a computed property is used.

Results are the same either way. On 200,000 orders of 10 members:

Query 1.2.0 1.3.0
GroupBy(o => o.Region).Select(g => new { g.Key, Total = g.Sum(o => o.Amount) }) 1.1 s, 261 MB 0.25 s, 57 MB
Where + GroupBy on two keys, top product per group 130 ms, 54 MB 92 ms, 23 MB
SelectMany over a list member (one JSON column, parsed whole) 514 ms, 274 MB 502 ms, 183 MB

Lists and classes as nested columns (1.4)

With NestedColumns = true (or [JazminNested] on a property), lists and classes are stored as columns of their own fields instead of JSON text: smaller files, and reads several times faster. AsQueryable<T>() then reads only the nested fields a query uses, and conditions such as c.Departments.Any(d => d.Name == "Sales"), c.Tags.Contains("vip") or c.Head.City == "Durban" are checked as the file is decoded. Reading needs no settings. See Nested Columns.

Queries compiled once per shape (1.4)

The parts of a query that run in memory are compiled the first time a query of that shape runs, then reused with each call's values: a lookup in a loop, q.Where(c => c.Id == id).Select(c => c.Name), takes 0.34 ms instead of 1.05 ms. The same holds for reader.Query<T>(predicate). Nothing to set up.

LINQ in the browser: Blazor WebAssembly (1.5.0)

The library runs in the browser with Blazor WebAssembly, LINQ included: the sample dotnet/samples/Jazmin.Blazor makes a file of two tables in the browser and queries both, in lambda and query syntax. Unencrypted files only: .NET in the browser has no AES-GCM or ECDSA, which files locked with a key or password, and shared files, need. For those, use the viewer or the browser reader (JavaScript, with JazminBrowser.from for LINQ-style queries), or the library on a server.

Streaming

// Writing: Serialize takes any IEnumerable<T> and never collects it
new JazminSerializer(settings).Serialize("big.jzm", database.ReadCustomers());

// Reading without a reader: one object at a time
using var stream = File.OpenRead("big.jzm");
foreach (var c in new JazminSerializer(settings).DeserializeEnumerable<Customer>(stream)) Process(c);

// Async sources and consumers
await foreach (var c in reader.QueryAsync<Customer>(c => c.Country == "ZA")) await SendAsync(c);

JSON at the edges

string json = JazminConvert.ToJson(bytes, settings, JazminFilter.Eq("Country", "ZA")); // for an API response
byte[] jzm = JazminConvert.FromJson(json, settings);                                     // from an API request

For large results, stream them with JazminJsonStream instead of building a string: see Converting JSON, CSV and XML.

JAZMIN 1.2 ยท file format 1.0 ยท Repository ยท npm ยท NuGet ยท MIT license ยท Examples in this wiki are run and checked by tools/check-examples.mjs

โš ๏ธ **GitHub.com Fallback** โš ๏ธ