Architecture

lz6 has two compression engines inside one frame format. The frame lets every block pick its own engine.

             ┌──────────────────────────── .lz6 frame ────────────────────────────┐
 input ──►   │ block 1: seq  │ block 2: light-LZ │ block 3: raw │ block 4: seq ... │
             └────────────────────────────────────────────────────────────────────┘
                 LZ6S2: codec chosen per block      LZ6S3: blocks cut at content shifts

Engine 1: HC frame codec (--hc)

This is the LZ5-lineage byte-wise LZ77:

• 22-bit dictionary (LZ4 has 16 bits).
• MINMATCH = 3.
• Four offset codewords: 1-byte 10-bit, 2-byte 16-bit, 3-byte 24-bit, and a 1-byte "repeat last offset".

It has no entropy stage. Decode runs at ~1 GB/s. Frame magic 0x184D2206 (LZ4 = ...04, LZ5 = ...05). Specified in lz6_Block_format.md and lz6_Frame_format.md.

Engine 2: seq (levels 1-15)

A match finder and parser produce a list of sequences (literal length, match length, offset). Each group of levels uses a different one:

levels	match finder / parser
1-2	single-pass fast parser (zstd "fast"-style): one small hash of 6-byte keys, a repeat-offset check, an accelerating step through literal runs; sequences go straight into the block
3-5	row hash (zstd 1.5-style): the hash picks a row of 16 recent positions with 8-bit tags; one SSE2 compare finds the tag hits, so one cache miss yields up to 16 candidates. Greedy (3) or lazy parser (4-5), rep0 checked first, candidates ranked by 4*length - log2(offset)
6-7	hash chains, lazy "lowest price" parser with 4 or 8 candidates
8-15	chains / binary tree with the optimal (DP) parser, priced with the costs the seq coder will charge, taken from a cheap pre-parse's statistics

The sequences are entropy-coded:

field	coding
literal length, match length	log-bucket code (36 symbols) as table-ANS states + raw extra bits
offset	exact code for 1..31; above, the log2 bucket plus the low 3 bits (alignment of structured data); or a repcode (rep0/1/2). All folded into one alphabet; the bits between are raw
literals	best of raw / Huffman / FSE order-0 / FSE order-1 (16 or 256 contexts), chosen per block

Since seq v2 the three sequence symbols are table ANS (zstd FSE-style) and share one backward bitstream with all their extra bits. The literal coders ("FSE" order-0/order-1 in this codebase) are range ANS: 32-bit state, lower bound 2^16, byte-wise renormalisation, L_bits ≤ 12.

Before LZ parsing, a block may take one of these special paths:

• PRNG regeneration: output of glibc random() is detected and stored as its 2-byte seed (AIT file D: 2MB → 7 bytes).
• x86 branch filter (BCJ): code-like ranges have their CALL/JMP displacements turned into absolute addresses before matching (and back after decoding), where a trial compression says it pays. ooffice -14%, AIT H -4.6%; data that is not x86 is untouched.
• Byte-plane transform: interleaved 16/32-bit data (floats, images) is split into 2 or 4 lanes. Each lane is compressed on its own (seq, pure Huffman or raw) when that beats the normal pass.

See Seq Stream Format for the exact byte layout and Decoder Internals for how it is decoded fast.

LZ6S2: per-block codec dispatch

Every block records its codec (seq / light-LZ / raw escape) in the block header, so a frame can mix them. The encoder falls back to the fast LZ codec when seq gains less than 25%, and to raw for incompressible blocks.

LZ6S3: content segmentation

The CLI reads 256KB windows and cuts a block when the window's order-0 entropy moves ≥ 1.0 bit/byte away from the open segment's profile (minimum segment 4MB). Each segment gets its own literal mode and transform. This recovers part of the per-file advantage inside a single mixed stream, such as a tar: Silesia.tar L15 29.46% → 28.34%.

Multithreading (v1.6.10-pre)

Seq blocks are self-contained: no block refers to bytes of another, and a block's decoder needs nothing but its own payload. The CLI exploits that without touching the format (-T#, programs/lz6mt.h):

• Compression (levels 1-15 only). The classic HC frame codec keeps table contents from the previous block, so which context compresses a block would change its bytes: --hc and -0 stay on one thread. The main thread reads, runs the LZ6S3 segmentation, updates the content checksum in input order, and hands each block to a small ordered worker pool. Each worker owns a frame context and compresses whole blocks (LZ6F_compressBlockIndependent); the main thread writes the finished blocks back in input order. The output is therefore byte-for-byte what one thread writes, whatever -T is.
• Decompression. The main thread splits the frame into blocks (4-byte header word + payload), workers decode them (LZ6F_decompressBlockIndependent, stateless), and the blocks are checksummed and written in order. The frame header is still validated by the ordinary frame decoder; frames with linked blocks (which this library no longer writes) fall back to the single-thread path.
• Pool. 2 x threads + 2 slots: blocks finish out of order but leave in order, so a slow early block holds its successors' slots and extra slots keep the workers fed. Workers are started on demand, so a small input does not spin up the whole pool. Slot buffers are sized to the block actually in flight.
• Thread safety. The lazily built code tables of the seq engine are filled by LZ6_seq_init(), which the frame context constructors call; after that the library has no mutable global state. Verified with ThreadSanitizer.

What bounds the speed-up: the number of blocks (one block is never split), the biggest block, and the main thread (reading, segmentation scan, XXH32, copy: about 270 MB/s). The default block size of seq levels on large inputs (B7, up to 256MB) makes few, large blocks; -B5/-B4 make more of them at a small ratio cost. Measurements in Benchmarks.

Source map

file	role
lib/lz6.c	fast LZ block encoder + LZ block decoder
lib/lz6hc.c	match finders (hash, row hash, chains, binary tree) and parsers (fast, lazy, price, optimal DP)
lib/lz6frame.c	frame layer, per-block dispatch, checksums
lib/entropy/lz6seq.c	seq engine: sequence collection, literal modes, transforms, encoder + decoder
lib/entropy/lz6fse.c	rANS tables, order-0 and order-1 (context) coders
lib/entropy/lz6huf.c	canonical Huffman for literals
lib/mem.h	unaligned loads, match counting, wild copies
programs/	CLI (lz6cli.c, lz6io.c), test tools
bakeoff/	benchmark and safety harness (see Development)