Portable Profile

The HC block format (lz6_Block_format.md) is lz6's portable format: byte-aligned LZ77, no entropy stage, and frozen. It is byte-identical to LZ5 v1.5's block format, except that offsets go up to 2^24 - 1 (16 MB) instead of 2^22. The seq engine (default levels 1-15) is a separate, changing lab format and is not part of this profile.

Why it exists

A zpaq archive can carry its own decoder as a ZPAQL post-processor, which any zpaq (including Matt Mahoney's 7.15) runs without knowing the algorithm. zpaq-std wrote a 431-byte ZPAQL decoder for this format. It decodes all the test vectors identically, at 50-150 MB/s with JIT and ~11 MB/s without. ZPAQL rules out entropy coding, loose bits and knowing how many input bytes remain; this format needs none of them.

Reference version

Vendor tag v1.6.4-pre or later. Earlier commits have a crash in the fast encoder on 32-bit Windows: empty hash entries were NULL pointers that the 16 MB distance check let through when the heap sits low. It is fixed in 9614981, and the block format and 64-bit output are unchanged by the fix.

Encoders

encoder	API	CPU	ratio vs lz4 (Silesia file)
fast (-0)	LZ6_compress_default, LZ6_compress_fast	160-335 MB/s	dickens 52.2% vs 63.1%, samba 31.2% vs 35.7%, mozilla 45.0% vs 51.6%
HC 1-15 (--hc)	LZ6_compress_HC	2-50 MB/s	better, at more CPU

Window per block

LZ6_compress_fast_window() and LZ6_compress_HC_window() cap every offset to 2^windowLog - 1 (windowLog 10..24). The format does not change: a decoder, or a zpaq block header, only has to provide a 2^windowLog window. Cost on dickens with HC -15: 2^22 +1.2%, 2^20 +5.5%.

Details the spec now states

• A block may start with the repeat-offset codeword 011; last_off starts at 1.
• Tokens 1_, 010_ and 011_ carry a 2-bit literal length (cap 3); 00_ carries 3 bits (cap 7). A capped field continues with 255-bytes.

Test vectors (raw blocks, originals, manifest, SHA-256) are kept in /mnt/IA_LAB/compartido/lz6/ on the lab machine.

Not in the profile (yet)

rep0-2 offsets and denser length packing would improve ratio, but they change the format. If they happen, they will be a separate frozen format with its own decoder, and only if the extra ratio is worth a second ZPAQL program.