Changelog

Releases of uf-lzma2 (v1.1.0 and later) and, below them, of fast-lzma2, the upstream library whose history this fork continues and which is now maintained in the repository's fast-lzma2/ directory (starting at fast-lzma2 1.1.0). Each release is also on the releases page.

v1.6.4

• A compression level above the maximum (UF2_maxCLevel(), 11) now clamps to the maximum instead of silently falling back to the default level. The one-shot and streaming entry points passed the out-of-range level to UF2_CCtx_setParameter without checking its error, so e.g. level 12 left the context at the default. In-range levels (1-11) are unchanged. The same fix is in fast-lzma2 1.1.2.

v1.6.3

Build and platform release: uf-lzma2 now builds on macOS (Apple Silicon). The compressed bytes do not change; valid input compresses and decompresses exactly as in v1.6.2.

• On Darwin the shared library is linked as a .dylib with -install_name, detected from uname -s; ldconfig, which macOS lacks, is guarded in install/uninstall. The arch detection already selected the ARM64 assembler decoder on Apple Silicon, whose source already carried the Mach-O symbol name, so the assembler decode path runs unchanged. CI runs make check on an Apple Silicon runner, the first end-to-end test of the ARM64 assembler decoder under Mach-O. Intel Macs are not yet supported: the x86_64 assembler decoder uses a MASM-style syntax the Mach-O assembler rejects.
• The repository's .gitignore was replaced with a focused list; it had been the 317-line GitHub Visual Studio/.NET template.

v1.6.2

Decoder-only release, from the differential decoder fuzzer of the stability campaign. The compressed bytes do not change; valid input compresses and decompresses exactly as in v1.6.1.

• A corrupt native stream could hold an LZMA chunk whose declared packed size is larger than its real LZMA data, so the range decoder reaches a clean end (code zero at the output limit) with packed bytes still unconsumed. The one-shot and single-threaded streaming decoders then read the next control byte from inside the chunk's leftover bytes, hit a zero there, and stopped reporting success; the multi-threaded streaming decoder, which locates chunks by their packed size, read on into the next chunk. So the four decode paths disagreed on the same corrupt input, each returning success with different output. A completed LZMA chunk must now consume its whole packed size, as the LZMA SDK requires, so the chunk is rejected on every path. Valid multi-chunk streams are unaffected, as their chunks consume exactly their packed size. The same defect is in fast-lzma2.

v1.6.1

Stability release from a fuzzing and sanitizer campaign (libFuzzer with ASan and UBSan, ThreadSanitizer, injected allocation failures, a real-file corpus). The compressed bytes do not change; valid input compresses and decompresses exactly as in v1.6.0. Eleven defects were fixed, most of them inherited from fast-lzma2.

Decoding corrupt data:

• The LZMA decoder could read up to 20 bytes (LZMA_REQUIRED_INPUT_MAX) past the end of a chunk's input. It checks its limit only between symbols, and one symbol can read that far; the limit now leaves that much room and the last bytes are decoded from a zero-padded copy, as the LZMA SDK does. Using more than the input holds is a corruption error.
• An LZMA chunk with no properties after a dictionary reset was decoded with properties no chunk had set. need_init_prop was raised at the reset but never checked, and a fast-lzma2 typo (p->prop.lc = 2 where pb was meant) left pb holding whatever the context contained. With pb out of range the C decoder shifted by 190 bits (UBSan) and the assembler decoder indexed past its probability tables and crashed. The chunk is now rejected and pb starts at 2.

Threading:

• UF2POOL_waitAll() could return while a job was still running, by reading the pool's state outside the lock; it now reads everything under the lock.
• A streaming compression with a timeout could publish and read a compression's results before it had finished.
• The match finder's shared list index is now read atomically in both the increment and the cancel check, so ThreadSanitizer is clean.

Robustness:

• Compression returns an error, instead of crashing, when the match finder's per-thread builders cannot be allocated; RMF_createMatchTable now frees and reports the failure.
• Every .xz allocation goes through UF2_setAllocator() rather than bare malloc/free.
• Raw LZMA2 of an empty input no longer writes a stray property byte, nor one byte past the output buffer.
• UF2_compressBuffer() no longer advances a pointer past the end of a small input (pointer overflow is undefined behaviour; found by UBSan).
• The match finder's 4-byte repeat probes are read with MEM_read32() instead of an unaligned pointer cast.

v1.6.0

• UF2_p_xzSizedHeaders: streamed .xz can state both sizes in every Block Header, by holding each block in memory until it ends, so that it decompresses on several threads like one-shot output. Off by default; default streaming output is unchanged. See Output formats.
• One-shot .xz: a block split at its dictionary resets by the multi-threaded decoder is now also checked while it is decoded, each thread over its part, with the parts' CRCs combined in order.
• Two compiler warnings inherited from fast-lzma2 are gone.
• bench/lzbench/add_uflzma2.py renames the bundle's internal symbols instead of making them local, which also works on Windows, where the PE linker merged a COMDAT section of the two libraries (found by zpaq-std).

v1.5.1

• Fixed: on 32-bit x86 Windows, builds made by a GCC that assumes a 16-byte aligned stack (Debian's mingw-w64 does) crashed with an access violation when the hardware CRC of 1.5.0 ran in a thread that Windows started: multi-threaded or asynchronous .xz decompression, or any caller's own thread. Windows starts threads with a 4-byte aligned stack on x86. The fold functions now realign it. 64-bit builds, MSVC builds, Linux and MSYS2's default i686 GCC were not affected. Checked on Windows 7 (x86 and x64) and Windows 10 (x86); CI now runs make check on Windows x86 too, set up so that it crashes without the fix.

v1.5.0

• Hardware CRCs for .xz: CRC32 and CRC64 fold 64 bytes at a time with a carry-less multiply, PCLMULQDQ on x86 (32 and 64 bit) and PMULL on ARM64 Linux and Apple, detected at run time; tables elsewhere. One-shot decoding also runs the check 256 KiB at a time as each block is decoded, while the data is in cache. The cost of CRC64 to decompressing Silesia fell from 6.4% to about 0.8% one-shot and from 4.9% to 0.4% streamed. See Output formats.
• make check: .xz and streaming tests against files written by xz and 7-Zip and against the library's own output, with a mutation stage, plus a hardware-against-tables CRC test. CI runs it on every push for x86_64, x86_64 under ASan and UBSan, 32-bit x86, a single-threaded build without xxhash, ARM64 under qemu and Windows x64. See Building.
• Fixed: streamed .xz output passed a NULL buffer to memcpy when no framing was queued (found by UBSan; zero bytes were copied, so no output changed).
• Build: test/Makefile keeps its include path when CFLAGS is given on the command line and links the static library by name; the musl job of the binaries workflow now compiles uf2_xz.c.

v1.4.0

• Streaming .xz: UF2_CStream writes .xz with UF2_p_format, through UF2_compressStream(), the zero-copy dictionary functions and with a timeout alike; UF2_DStream reads .xz, including concatenated Streams. See Output formats.
• Fixed: UF2_decompressStream() crashed on a native stream whose property byte had the hash flag set and an invalid dictionary size (FF 00 is enough): the failed init left the hash flag set with no hash state. Inherited from fast-lzma2.

v1.3.0

• Multi-block .xz: UF2_p_xzBlockSize, one block per dictionary reset by default, and parallel decoding of multi-block .xz, this library's or xz's, on a multi-threaded context. See Output formats. An input no larger than dictionary size times reset interval gives the same bytes as before (checked at levels 1, 6 and 10); a larger one now splits exactly there, where the encoder's own resets used to land a little earlier (Silesia's files at level 1: -0.17% to +0.09%).
• UF2_compressBound() grows by 66 bytes per MiB of input, the worst case of 1 MiB blocks.
• Fixed: builds with NO_XXHASH wrote .xz with no check instead of CRC64.

v1.2.0

• Standard .xz output and input for the one-shot functions: UF2_p_format, UF2_p_xzCheck (CRC64 by default, CRC32 or none). Decompression recognises .xz by itself and also reads .xz written by xz, including multi-block and concatenated files. See Output formats.
• Level 11, level 10 plus UF2_p_propertySearch, a per-input lc/lp/pb search that keeps the smallest output. See Level 11.
• UF2_compressBound() grows by up to 87 bytes so that one bound covers both formats.
• Fixed: UF2_estimateCStreamSize() read past the level table for any level it should reject.
• Fixed: uf-lzma2.h shared its include guard with fast-lzma2.h, so including both lost the second.
• Fixed: the fuzzer divided by zero when run with its default thread count.
• bench/curve measures the speed/ratio curve; bench/lzbench/add_uflzma2.py adds the library to lzbench with its assembler decoder.

v1.1.0

This release is Conor McCarthy's unreleased dev branch, which sat unpublished after v1.0.1. Every commit on it other than the two security fixes already in master was carried over.

• Added support for custom allocators.
• Replaced alloc_struct with allocation_size and used it for the compatibility check.
• Greatly reduced the number of UF2_SINGLETHREAD ifdefs.
• New lit_pos_mask formula, folding the position and the previous symbol into a single masked operation for literal probability indexing. Compressed output is unchanged.
• Changed cache_size to size_t and removed some unused functions.
• Renamed DICT_destruct to DICT_free, RMF_calBufSize, the CCtx 'factory' to 'pool', and the custom allocator functions.
• Added a test for the async behaviour of the dual buffer cstream, and DEBUGLOG on deallocation.

fast-lzma2 1.1.2

The releases above are uf-lzma2's; this one and those below are fast-lzma2's, the upstream library this fork started from, now kept current in the fast-lzma2/ directory of this repository. 1.1.x backports uf-lzma2's correctness and robustness fixes and adds two compatible build features; the compressed output for in-range levels is unchanged from 1.0.1.

• 1.1.2: a compression level above the maximum (FL2_maxCLevel(), 10) now clamps to the maximum instead of silently falling back to the default level 6 (the one-shot and streaming entry points did not check the error from passing an out-of-range level to FL2_CCtx_setParameter). Found via zpaq-std's -ma:flzma2:11.

1.1.0 shipped without the decoder fixes in its lzma2_dec.c (the change was lost before the commit; zpaq-std found a heap read overflow on damaged blocks with AddressSanitizer). 1.1.1 restored them, verified against the reporter's corpus under ASan. Use 1.1.1 or later.

The 1.1.0 decoder and build fixes below:

• Decoding corrupt data: the LZMA decoder no longer reads past the end of a chunk; a chunk with no properties after a dictionary reset, or one that ends before consuming its declared packed size, is now rejected.
• Match finder: unaligned-safe repeat-probe reads, an atomically read shared list index, and a reported failure (instead of a NULL builder dereference) when the per-thread builders cannot be allocated.
• Threading: FL2POOL_waitAll() reads the pool's state under the lock; and streaming compression with a timeout reads a block's results only once a lock-synchronised idle check has seen it done, verified race-free with ThreadSanitizer.
• Robustness: empty and small raw-LZMA2 inputs no longer write a stray property byte or step a pointer past the input.
• An ARM64 assembler LZMA decoder (used on Linux and macOS), and a macOS (Apple Silicon) .dylib build.

v1.0.1

• The root makefile for GNU make now builds and installs a shared library and headers.
• Fixed a potential crash on memory allocation failure during structure allocations for multi-threaded decoding.
• Added a file compression test program.
• Renamed the VS DLL project and some structures / types.
• Removed some duplicated typedefs.

v1.0.0

• Breaking changes have been made to the API functions.
• Some of the options have been renamed.
• Optimized the encoder, incorporating some of Igor Pavlov's improvements to 7-Zip 18.05, and some novel ones. The speed increase is about 5% - 8%.
• Moved detection of repeats from the single-threaded initialization stage to a later, multi-threaded stage to increase speed.
• Removed two compression levels, reducing the total to 10, and tweaked the parameters.
• Improved calculation of the match buffer size. It can still be changed in the options, but the meaning of the value is different.
• Replaced the callbacks for writing and progress with timeouts and new functions to gain direct access to the dictionary buffer and the compressed data buffers.
• Added Igor Pavlov's assembler-optimized decoder.
• Multi-threaded decompression.

v0.9.2

• Fixed excess memory allocation when the dictionary size is > 64Mb

v0.9.1

• Fixed a bug in compression of very small files when using a high search depth.
• Added an incompressibility checker which processes high-entropy (e.g. encrypted or already compressed) data about twice as fast as before.