l2h - aegoroff/hc GitHub Wiki

l2h (linq2hash) is a LINQ-style query language for hashing. It uses the same algorithms as hc. A query binds sources, filters and shapes rows, then either prints the result or passes it on with into.

Hash and file I/O run only when a property needs them. Put cheap checks (size, path) before hash properties (md5, sha1, ...) so you do not hash files you are going to discard.

The language semantics are in the repo: docs/l2h-semantics.md.

Running queries

l2h -q "from string s in 'abc' select s.md5;"
l2h -f query.l2h
l2h -n -f query.l2h

If you omit -q and -f, the query is read from standard input. -n / --syntax-check parses and type-checks without executing. If both -q and -f are given, -q is used.

Several queries can sit in one file, separated by ;. A line that is only #... between queries is a comment. You cannot put a comment inside a query body or after code on the same line.

Query shape

from_clause
query_body_clauses*     -- where | let | join | orderby | additional from
select_or_group_clause
query_continuation?     -- into identifier query_body

Example:

from file f in '/home/user/file'
where f.size > 0
select f.md5;

Sources

Declaration	What it binds
`from string x in E`	one string (the value of `E`, which must be a string)
`from file x in E`	one file when `E` is a path string, or a directory walk when `E` is a `Dir`
`from dir x in E`	the directory itself, not its files
`from hash x in E`	a restore source: `E` is a digest string, not text to hash

Directory contents are not flattened into from dir. Walk files with a second from:

from dir d in '/tmp'
from file f in d
where f.size > 0
select f.md5;

from file f in d visits only immediate regular files. Symlinks are skipped. Recursion is a method on Dir:

from dir d in '/tmp'
from file f in d.tree()
select f.path;

d.tree() walks the whole tree. d.tree(n) enters at most n directory levels (tree(0) is the same as flat). d.skipErrors() continues if a subdirectory cannot be entered. tree and skipErrors can be combined in either order. They return a new Dir and do not change d.

File order is the order of the directory walk. Use orderby if you need a fixed order, for example orderby f.path.

Properties

Properties are computed on first read. Hash property names are the same as hc algorithm names (md5, sha1, sha-3-256, blake2b-256, and so on). Digests from File / String hash properties are lowercase hex. Comparisons against digests are case-insensitive.

Receiver	Property	Result
`File`	`path`	path (no I/O)
`File`	`name`	basename only
`File`	`size`	size in bytes (full file; ignores `limit`/`offset`)
`File`	`offset` / `limit`	current hash window (read); set with methods below
`File`	`readable`	`true` if the path opens as a regular file; never raises I/O
`File`	`<hash>`	hex digest of the file (uses the window)
`String`	`size`	length in bytes
`String`	`<hash>`	hex digest of the string
`Hash`	`<hash>`	restore using that algorithm, not hashing the digest characters
`Hash`	`dict` / `min` / `max` / `noProbe`	restore settings (read); set with methods below
`Dir`	`path`	directory path

NTLM hashes UTF-16LE. A non-UTF-8 string payload is a runtime error.

Methods

A method call returns a new value. The original binding is not changed.

Call	On	Effect
`f.offset(n)` / `f.limit(n)`	`File`	hash window (`n >= 0`). Same meaning as `hc --offset` / `--limit`
`d.tree()` / `d.tree(n)`	`Dir`	recursive walk
`d.skipErrors()`	`Dir`	skip unreadable subdirectories
`h.dict(s)` / `h.min(n)` / `h.max(n)` / `h.noProbe()`	`Hash`	restore alphabet, length bounds (`n >= 1`), skip the `"123"` timing probe
`recv.<hash>(expected)`	`File` / `String`	`true` if the digest matches (case-insensitive)
`seq.count()`	`Seq`	number of already collected elements

from hash uses the same defaults as hc hash: alphabet 0-9a-zA-Z, lengths 1 through 10, timing probe on. min > max is an error. n = 0 is an error. hc uses 0 to mean the option was omitted; l2h does not store that.

An offset past the end of the file is an error. offset(0) on an empty file is valid.

Expressions

String literals '...' / "..." have no escapes, so 'c:\Windows' keeps the backslash.
Byte-string literals b'...' and b"..." support \xNN, \\, \', \", \n, \r, \t.
Integers are 64-bit. true / false are literals and valid where predicates.
== / !=. > / >= / < / <= work on Int only. ~ / !~ work on String only (PCRE2, unanchored).
&&, ||, !, parentheses.
Anonymous objects { e1, e2 } or { name = e, ... }. Unnamed fields must be id.prop (field name prop) or bare id. Duplicate field names are a compile error.
Nested queries as values, as where exists-predicates, or as from/join sources.

Clauses

let id = expr: bind a name per row
where pred: keep rows where pred is true
join T y in src on e1 equals e2: inner equijoin. join ... into g is a group join (g is the sequence of matches, possibly empty)
orderby e1 [ascending|descending], e2 ...: stable sort (collects first)
group expr by key: each group is a record { key, items } (items is a Seq of the projected values)
select expr: project. If this is the last operation, print. select expr into id continues without printing

into id; with no following body stores id for later queries in the same file (one row becomes a scalar, several become a Seq). If a body follows select/group into, it runs in a fresh environment that contains only id.

Output

When select or group is the last operation, the result is printed:

one line per scalar (String / Int / Bool)
path for File / Dir, bound digest for Hash
a Seq is expanded item by item
a record prints one line per field, in field order

Hash restore (select x.md5 on a Hash) writes the restore runner output to stdout. The sink does not print the digest again.

Record formatters return a string:

Method	Required fields	Output
`sfv()`	exactly 2 fields including `name`	`name digest` (like `hc --sfv`)
`checksum()`	exactly 2 fields including `path`	`digest path` (like `hc -c`)
`json()` / `jsonPretty()`	scalars, nested records, sequences	compact or 2-space JSON; terminal sink is one object per row (NDJSON)
`csv()` / `spaced()` / `tabbed()`	scalar fields	joined in field order (no CSV escaping)

from file f in '/tmp/a'
select { f.crc32, f.name }.sfv();

from file f in '/tmp/a'
select { f.path, f.crc32 }.checksum();

Errors

Syntax and type problems fail at compile time when the types are known. Missing files, I/O errors, a bad regex, and offset past EOF fail at runtime. The query stops at the first error. Lines already written stay on stdout.