Building - YadeWira/ultra-fast-lzma2 GitHub Wiki

Building

Library

make                 # libuf-lzma2.so.1.0 and libuf-lzma2.a
make test            # builds test/file_test and runs it on a file
make check           # make test, plus test/xz_check: the .xz and streaming tests
make install         # PREFIX=/usr/local by default

On x86_64 and aarch64 the makefiles substitute Igor Pavlov's assembler LZMA decoder for the C one, chosen by uname -m. To override the detection:

make CC=aarch64-linux-gnu-gcc arm64=1   # cross-compile for ARM64, assembler decoder included
make x86_64=0 arm64=0                   # force the C decoder

make check builds test/xz_check and runs it on the fixtures in test/xz, written by xz 5.8.3 and 7-Zip 26.03: every check type, presets, multi-block files with and without sizes in the headers, an empty and a one-byte file, concatenated Streams, and files it must reject (SHA-256, BCJ, delta, bad Stream Padding, trailing bytes). Each is decoded one-shot on one and four threads and streamed in chunks of 1 byte to 64 KiB. Then the library's own .xz, one-shot and through every streaming API, is decoded every way, and the framing of two files is mutated bit by bit and truncated: no mutation may decode to completion with wrong data. If xz is installed, it tests the files written, too. RUN= runs the test programs through a wrapper, which is how the ARM64 build is checked under qemu:

make check CC=aarch64-linux-gnu-gcc x86_64=0 arm64=1 RUN="qemu-aarch64 -L /usr/aarch64-linux-gnu"

The GitHub workflow check.yml runs it on every push: x86_64, x86_64 with ASan and UBSan, 32-bit x86, single-threaded without xxhash, ARM64 under qemu, macOS on an Apple Silicon runner, and Windows x64 with MSYS2's mingw64, which exercises the assembler decoder with the Windows calling convention and the Win32 threading code. The tests also pass on Windows 10 built with mingw for x86, and against the DLLs that MSVC builds from dll/uf-lzma2.vcxproj for x64 and x86.

The ARM64 decoder is GNU assembler syntax, so it is built on Linux and on macOS (Apple Silicon), where its source already carries the Mach-O symbol name. On macOS make links a .dylib with -install_name instead of a .so with -soname, detected from uname -s, and the Apple Silicon CI job runs the whole make check against the assembler decoder. Intel Macs are not yet supported: the x86_64 decoder uses a MASM-style syntax the Mach-O assembler rejects. The Visual Studio solution in build/VS does not include the assembler decoder.

On Windows, build/VS holds a Visual Studio 2015 solution with projects for the DLL, a benchmark program, the fuzz tester and the file compression tester. The bench, fuzzer and test directories have makefiles for the same programs. The CMake file of earlier fast-lzma2 releases had no install script and is not included.

FXZ Utils wraps fast-lzma2 in a derivative of liblzma from XZ Utils, built with GNU autotools.

Measuring: bench/curve

make -C bench curve
./bench/curve -n3 /path/to/silesia/*        # per file, best of 3
./bench/curve -n3 -s -H /path/to/silesia/*  # solid stream, plus the high-compression table

One TSV row per compression level: ratio, compression and decompression MB/s. Every decompressed file is compared against its source, so a run that reports numbers has also checked correctness. Single threaded by default so that runs stay comparable.

Two things change the answer rather than the presentation. Use large files: on a corpus averaging 81 KB per file, the per-call overhead dominates (65 MB/s decompression, against 92 MB/s on one 29 MB file). And use -s for anything involving dictionary size: file by file, the useful dictionary is capped by the largest file.

Compare like with like. This library substitutes an assembler LZMA decoder on x86_64 and ARM64, and 7-Zip ships the same one, so a build of either without it looks slower for reasons that have nothing to do with the design. Build with x86_64=0 arm64=0 for a C-to-C comparison.

lzbench

lzbench builds fast-lzma2 from its C files only. bench/lzbench/add_uflzma2.py adds uf-lzma2 to an lzbench checkout as a separate codec, uflzma2, next to fastlzma2, with the same calls and the same compile flags plus the assembler decoder, and registers levels 1-11:

git -C lzbench checkout 3d22c41
python3 bench/lzbench/add_uflzma2.py lzbench        # wiring: Makefile, codec table, wrappers
mkdir -p lzbench/lz/uf-lzma2 && cp *.c *.h lzma_dec_*.S lzbench/lz/uf-lzma2/
make -C lzbench -j
cd lzbench && ./lzbench -t1,1 -efastlzma2,1,5,10/uflzma2,1,5,10 <files>

Run lzbench from its build directory: it resolves libdensity_rs.so relative to it, and from anywhere else it exits with code 127 without measuring anything.

The two libraries' internal symbols share names (LZMA2_*, RMF_*, xxhash), so uf-lzma2 is linked into a single relocatable object in which every global symbol but the UF2_* API is renamed with a ufl2_ prefix. Renaming, not making them local, is what works on Windows too: there GCC reaches extern data through COMDAT .refptr.<name> sections, and the PE linker keeps one .refptr.price_table for both libraries even when the symbol is local, which crashed from level 2 on. zpaq-std found that linking both libraries with mingw x64. The same recipe serves any program that links fast-lzma2 and uf-lzma2 together. The compressed sizes of the two codecs must match exactly on every file; if they do not, something was assembled wrong.

⚠️ **GitHub.com Fallback** ⚠️