Output formats

The same LZMA2 data can leave the library in three wrappings. The encoder runs identically in all three; only the framing around its output differs.

format	how	read by	integrity
native (default)	nothing to set	this library	xxhash32, flagged in the property byte
raw LZMA2	UF2_p_omitProperties = 1	any LZMA2 decoder, given the property byte	the container's
.xz	UF2_p_format = UF2_format_xz	xz, 7-Zip, any .xz decoder	CRC64 (default), CRC32 or none

Native

fast-lzma2's framing, unchanged: a property byte, the LZMA2 chunks, and by default a 32-bit xxhash of the input. The hash is flagged in the top bit of the property byte. A standard LZMA2 property is at most 40, so that bit is never set in a standard stream, and a standard decoder rejects this output at its first byte unless it is written with UF2_p_doXXHash = 0.

Raw LZMA2, for containers

For a container that stores the property byte itself, as 7-Zip does in its archive header:

UF2_CCtx_setParameter(cctx, UF2_p_omitProperties, 1);
size_t n = UF2_compressCCtx(cctx, dst, cap, src, srcSize, level);
unsigned char prop = UF2_getCCtxDictProp(cctx);   /* store this in the container */

Omitting the property byte also drops the hash, because the flag has nowhere to go. This is how 7-Zip-zstd uses the library. It is why the default format has to stay native: an integrator that sets only omitProperties must keep receiving raw LZMA2.

.xz

UF2_CCtx_setParameter(cctx, UF2_p_format, UF2_format_xz);
UF2_CCtx_setParameter(cctx, UF2_p_xzCheck, 4);        /* 4 = CRC64, 1 = CRC32, 0 = none */
UF2_CCtx_setParameter(cctx, UF2_p_xzBlockSize, 0);    /* 0 = one block per dictionary reset */
size_t n = UF2_compressCCtx(cctx, dst, cap, src, srcSize, level);   /* or UF2_compressMt() */

A standard .xz file ("The .xz File Format" 1.2.1) with the LZMA2 data inside, in one or more blocks. The dictionary each Block Header declares is capped at the block size, the same way the native format caps its own, so a small input does not make its decoder reserve the whole dictionary of a high level. UF2_compressBound() covers both formats.

Blocks, and decoding on several threads

Each block is an ordinary compression of its slice of the input, starting with a fresh dictionary, so blocks are independent and decode in parallel. UF2_p_xzBlockSize sets their uncompressed size; the default, 0, ends a block wherever the encoder resets its dictionary anyway, every dictionary size times UF2_p_resetInterval (4 by default) bytes. That split costs only the block framing: an input no larger than one reset interval gives exactly the bytes of 1.2.0, and level 6 already writes Silesia as 4 blocks at the same size. An explicit size must be at least UF2_XZ_BLOCKSIZE_MIN (1 MiB); smaller blocks decode on more threads and compress worse.

A multi-threaded context (UF2_decompressMt(), UF2_createDCtxMt()) decodes the blocks of a stream side by side. It needs both sizes in every Block Header to place each block before decoding the one before it; this library always writes them, and so does xz when it compresses on several threads. Files without them, such as those of xz -T1, are decoded in order. xz 5.4 and later also decode this library's multi-block files on several threads.

Silesia (211 MB, one file), x86_64, assembler decoder, CRC64, best of 3 in round robin:

level	block	size	1 thread	2	4	8	16
10	default (1 block)	22.966%	107 MB/s	105	104	109	109
10	64 MiB (4 blocks)	22.998%	104	157	276	281	278
10	32 MiB (7)	23.171%	107	198	295	505	542
10	16 MiB (13)	23.363%	102	187	325	542	859
10	8 MiB (26)	23.673%	102	187	337	557	803
6	default (64 MiB, 4)	23.670%	101	151	274	272	271
6	32 MiB (7)	23.723%	104	203	285	510	547
6	16 MiB (13)	23.758%	107	184	316	553	893
6	8 MiB (26)	24.022%	101	191	332	548	810

Speed stops rising once every block has a thread: 4 blocks top out near 280 MB/s on any thread count. Past 16 MiB, halving the block costs more ratio than the extra threads give back (8 MiB is slower than 16 MiB on 16 threads). xz 5.8.3 decodes the 16 MiB level-10 file at 557 MB/s on 16 threads and 94 MB/s on one, through a pipe.

Streaming

UF2_CStream writes .xz when UF2_p_format is set before UF2_initCStream(), through UF2_compressStream(), the zero-copy UF2_getDictionaryBuffer() / UF2_getNextCompressedBuffer() pair and with a timeout alike. The LZMA2 data is exactly the native stream's; the framing is queued around it. A block starts at each dictionary reset, as in the one-shot default, but its header has to be written before its data, when neither size is known yet, so by default it leaves them out, as xz does on one thread, and such a file decompresses on one thread only.

UF2_p_xzSizedHeaders set to 1 changes that: the encoder holds each block's compressed data in memory until the block ends, then writes its header with both sizes into room kept before the data. The file then decompresses on several threads like one-shot output, here and in xz 5.4 and later. It costs memory up to one block's compressed size, and a block is dictionary size times UF2_p_resetInterval of input (4 dictionaries by default, up to 512 MiB of input at level 10; a reset interval of 1 brings it to one dictionary). UF2_flushStream() writes nothing of the open block in this mode, since its sizes are not known yet. It works with every streaming API, timeouts and dual buffering included.

UF2_DStream recognises .xz by its first byte and decodes it on one thread, checking every CRC of the framing, each block check, and the Index against the blocks decoded. The Index is compared through a summary (count, the sums of both sizes and a CRC32 of the size pairs), so no per-block record is kept however long the file. UF2_decompressStream() returns 0 at the end of each Stream and accepts Stream Padding or another Stream after it, so a concatenated file decodes the same however its input is split; anything else after a Stream is an error. Input that ends inside Stream Padding that is not a multiple of four is left unfinished (1), which is how a streaming caller sees a truncated file.

The framing costs nothing measurable. Silesia at level 6, one thread, 1 MiB input and output chunks, best of 3: native stream 102.0 MB/s, streamed .xz with CRC64 103.4 MB/s; one-shot decoding of the same files 104.9 and 103.4 MB/s.

Reading

Nothing to set: UF2_decompress(), UF2_decompressMt() and UF2_decompressDCtx() recognise .xz by its signature. Its first byte, 0xFD, is never a valid native property byte, so there is no ambiguity. They also read .xz written by xz itself: any number of blocks, with or without sizes in the Block Headers, concatenated streams, Stream Padding, and CRC32, CRC64 or no check. Every CRC of the framing and every block check is verified, and the Index is compared against the blocks actually decoded. UF2_findDecompressedSize() reads the size from the Index.

What the decoder cannot read, it rejects with parameter_unsupported rather than decoding it wrongly: filter chains such as BCJ or delta, and SHA-256.

What the check costs

The check is the one part of .xz that costs decompression speed. Silesia at level 6, one thread, assembler decoder, best of five in round robin, each check against .xz with no check:

check	one-shot, tables	one-shot, hardware	streamed, tables	streamed, hardware
CRC32	-4.1%	-1.6%, then about -0.8%	-3.2%	-0.5%
CRC64	-6.4%	-1.6%, then about -0.8%	-4.9%	-0.4%

Since the version after 1.4.0 both CRCs fold the data 64 bytes at a time with a carry-less multiply where the processor has one: PCLMULQDQ on x86 (32 and 64 bit), PMULL on ARM64 Linux and Apple, detected at run time. Elsewhere, and below 64 bytes, they use slicing-by-16 over constant tables. On the E5-2697A v4 both run at 7.2 GB/s against 3.1 (CRC32) and 2.5 GB/s (CRC64) with the tables. tools/gen_xz_tables.py generates the tables and the fold constants, and simulates the fold against the tables before writing anything. The first, byte-at-a-time version cost 22%.

One-shot decompression first kept more of the cost because it checked a block only once the whole block was decoded, when most of it had left the cache, so the CRC ran at memory speed: 7.1 GB/s for CRC64 against 16 GB/s from cache. It now runs the check 256 KiB at a time as the block is decoded, which brought the cost of CRC64 from 1.7% to 0.8% on average (best of 15, three alternated runs each; the runs overlap, so the gain is about a point, not a precise one). The streaming decoder already checked each output buffer while it was in cache. A single block decoded by the multi-threaded LZMA2 decoder, split at its dictionary resets, is still checked at the end. The ARM64 speed is untested: qemu shows only that the results are right.