Building - YadeWira/ultra-fast-lzma2 GitHub Wiki
make # libuf-lzma2.so.1.0 and libuf-lzma2.a
make test # builds test/file_test and runs it on a file
make check # make test, plus test/xz_check: the .xz and streaming tests
make install # PREFIX=/usr/local by defaultOn x86_64 and aarch64 the makefiles substitute Igor Pavlov's assembler LZMA decoder for the C one,
chosen by uname -m. To override the detection:
make CC=aarch64-linux-gnu-gcc arm64=1 # cross-compile for ARM64, assembler decoder included
make x86_64=0 arm64=0 # force the C decodermake check builds test/xz_check and runs it on the fixtures in test/xz, written by xz 5.8.3 and
7-Zip 26.03: every check type, presets, multi-block files with and without sizes in the headers, an
empty and a one-byte file, concatenated Streams, and files it must reject (SHA-256, BCJ, delta, bad
Stream Padding, trailing bytes). Each is decoded one-shot on one and four threads and streamed in chunks
of 1 byte to 64 KiB. Then the library's own .xz, one-shot and through every streaming API, is decoded
every way, and the framing of two files is mutated bit by bit and truncated: no mutation may decode to
completion with wrong data. If xz is installed, it tests the files written, too. RUN= runs the test
programs through a wrapper, which is how the ARM64 build is checked under qemu:
make check CC=aarch64-linux-gnu-gcc x86_64=0 arm64=1 RUN="qemu-aarch64 -L /usr/aarch64-linux-gnu"The GitHub workflow check.yml runs it on every push: x86_64, x86_64 with ASan and UBSan, 32-bit x86,
single-threaded without xxhash, ARM64 under qemu, macOS on an Apple Silicon runner, and Windows x64 with
MSYS2's mingw64, which exercises the assembler decoder with the Windows calling convention and the Win32
threading code. The tests also pass on Windows 10 built with mingw for x86, and against the DLLs that
MSVC builds from dll/uf-lzma2.vcxproj for x64 and x86.
The ARM64 decoder is GNU assembler syntax, so it is built on Linux and on macOS (Apple Silicon), where
its source already carries the Mach-O symbol name. On macOS make links a .dylib with
-install_name instead of a .so with -soname, detected from uname -s, and the Apple Silicon CI
job runs the whole make check against the assembler decoder. Intel Macs are not yet supported: the
x86_64 decoder uses a MASM-style syntax the Mach-O assembler rejects. The Visual Studio solution in
build/VS does not include the assembler decoder.
On Windows, build/VS holds a Visual Studio 2015 solution with projects for the DLL, a benchmark
program, the fuzz tester and the file compression tester. The bench, fuzzer and test directories
have makefiles for the same programs. The CMake file of earlier fast-lzma2 releases had no install
script and is not included.
FXZ Utils wraps fast-lzma2 in a derivative of liblzma from XZ Utils, built with GNU autotools.
make -C bench curve
./bench/curve -n3 /path/to/silesia/* # per file, best of 3
./bench/curve -n3 -s -H /path/to/silesia/* # solid stream, plus the high-compression tableOne TSV row per compression level: ratio, compression and decompression MB/s. Every decompressed file is compared against its source, so a run that reports numbers has also checked correctness. Single threaded by default so that runs stay comparable.
Two things change the answer rather than the presentation. Use large files: on a corpus averaging 81 KB
per file, the per-call overhead dominates (65 MB/s decompression, against 92 MB/s on one 29 MB file).
And use -s for anything involving dictionary size: file by file, the useful dictionary is capped by
the largest file.
Compare like with like. This library substitutes an assembler LZMA decoder on x86_64 and ARM64, and
7-Zip ships the same one, so a build of either without it looks slower for reasons that have nothing to
do with the design. Build with x86_64=0 arm64=0 for a C-to-C comparison.
lzbench builds fast-lzma2 from its C files only. bench/lzbench/add_uflzma2.py adds uf-lzma2 to an
lzbench checkout as a separate codec, uflzma2, next to fastlzma2, with the same calls and the same
compile flags plus the assembler decoder, and registers levels 1-11:
git -C lzbench checkout 3d22c41
python3 bench/lzbench/add_uflzma2.py lzbench # wiring: Makefile, codec table, wrappers
mkdir -p lzbench/lz/uf-lzma2 && cp *.c *.h lzma_dec_*.S lzbench/lz/uf-lzma2/
make -C lzbench -j
cd lzbench && ./lzbench -t1,1 -efastlzma2,1,5,10/uflzma2,1,5,10 <files>Run lzbench from its build directory: it resolves libdensity_rs.so relative to it, and from anywhere
else it exits with code 127 without measuring anything.
The two libraries' internal symbols share names (LZMA2_*, RMF_*, xxhash), so uf-lzma2 is linked into
a single relocatable object in which every global symbol but the UF2_* API is renamed with a ufl2_
prefix. Renaming, not making them local, is what works on Windows too: there GCC reaches extern data
through COMDAT .refptr.<name> sections, and the PE linker keeps one .refptr.price_table for both
libraries even when the symbol is local, which crashed from level 2 on. zpaq-std found that linking
both libraries with mingw x64. The same recipe serves any program that links fast-lzma2 and uf-lzma2
together. The compressed sizes of
the two codecs must match exactly on every file; if they do not, something was assembled wrong.