fio testing - vincentkfu/fio-blog GitHub Wiki

Fio has a large, complex code base that supports myriad options for IO generation. Like virtually all software projects, fio benefits from automated testing. This blog post is an overview of testing fio (as of fio 3.30) and covers automated testing of fio features and fio's continuous integration testing infrastructure. At the end is an assessment of fio's test infrastructure and discussion of areas where it can be improved. To be clear, this blog post discusses tests for fio's features (not how to use fio to test storage devices).

Let us begin with a tour of components fio has for automated testing. These components include test jobs, compiled test programs, unit tests, and test scripts.

The first general class of fio's test components are test jobs. Fio has fourteen sets of test jobs in the subdirectory t/jobs.

root@ubuntu:~/fio-dev/fio-canonical# ls -l t/jobs
total 64
-rw-r--r-- 1 root root 106 Jan 28 20:23 t0001-52c58027.fio
-rw-r--r-- 1 root root 272 Jan 28 20:23 t0002-13af05ae-post.fio
-rw-r--r-- 1 root root 258 Jan 28 20:23 t0002-13af05ae-pre.fio
-rw-r--r-- 1 root root 323 Jan 28 20:23 t0003-0ae2c6e1-post.fio
-rw-r--r-- 1 root root 243 Jan 28 20:23 t0003-0ae2c6e1-pre.fio
-rw-r--r-- 1 root root 364 Jan 28 20:23 t0004-8a99fdf6.fio
-rw-r--r-- 1 root root 198 Jan 28 20:23 t0005-f7078f7b.fio
-rw-r--r-- 1 root root 307 Jan 28 20:23 t0006-82af2a7c.fio
-rw-r--r-- 1 root root 167 Jan 28 20:23 t0007-37cf9e3c.fio
-rw-r--r-- 1 root root 191 Jan 28 20:23 t0008-ae2fafc8.fio
-rw-r--r-- 1 root root 608 Jan 28 20:23 t0009-f8b0bd10.fio
-rw-r--r-- 1 root root 146 Jan 28 20:23 t0010-b7aae4ba.fio
-rw-r--r-- 1 root root 293 Jan 28 20:23 t0011-5d2788d5.fio
-rw-r--r-- 1 root root 369 Jan 28 20:23 t0012.fio
-rw-r--r-- 1 root root 262 Jan 28 20:23 t0013.fio
-rw-r--r-- 1 root root 589 Jan 28 20:23 t0014.fio

Most of the test jobs include a commit hash in the name. For jobs referencing a commit, the hash in question refers to the patch that fixed a bug exposed by the job file. Thus, commits before the hash in question will exhibit buggy behavior when the job in question is run. Comments are included in the job files describing the correct and buggy behavior. For instance, t0004-8a99fdf6.fio contains the following:

# Expected result: fio runs to completion
# Buggy result: fio segfaults
[global]
ioengine=libaio
direct=1
filename=foo
iodepth=128
size=10M
loops=1
group_reporting=1
readwrite=write
do_verify=0
verify=md5
numjobs=1
thread
verify_dump=1

[small_writes]
offset=0G
blocksize=512
verify_interval=1M

[large_writes]
stonewall
offset=1G
blocksize=1M
verify_interval=512

Prior to commit 8a99fdf6, the job in question would cause a segmentation fault. This bug was fixed in 8a99fdf6 and from that commit onwards fio should complete the job without error. Many of these test jobs are related to verification workloads. Some of these test jobs include a preconditioning step. The final few jobs in this set do not reference a specific commit.

Fio has a set of test programs exercising various portions of the code base. These are located in the t/ subdirectory and are built by the Makefile along with the main fio executable. These test programs include:

Fio's codebase is integrated with the CUnit unit testing framework. Relevant code is located in the unittests subdirectory. If CUnit support is detected by the configure script, unit tests will be automatically built. Unit tests exist for only seven fio components and are mostly for fio's string manipulation functions.

Also in fio's repository is fuzzing for fio's option parsing functions. By default a test program is built that can take a fio command line and offer it to fio's option parsing routines to check for problems. By defining appropriate environment variables alternative fuzzing strategies can be used. To build this test program the symbol FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION must be defined. One way to accomplish this is to run fio's configure script with the --extra-cflags=-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION option. When the executable is built it can be used with fio command lines such as the following:

root@ubuntu:~/fio-dev/fio-canonical# t/fuzz/fuzz_parseini examples/aio-read.fio
root@ubuntu:~/fio-dev/fio-canonical# echo $?
0
root@ubuntu:~/fio-dev/fio-canonical# t/fuzz/fuzz_parseini examples/aio-read.png
fio: option <▒PNG> outside of [] job section
root@ubuntu:~/fio-dev/fio-canonical# t/fuzz/fuzz_parseini kflfhd
root@ubuntu:~/fio-dev/fio-canonical# echo $?
2

Fuzzing is also incorporated into fio's continuous integration tests which will be discussed later.

Fio's source repository also contains a set of bash and Python test scripts.

Notable are the bash scripts in the t/zbd subdirectory. As the directory name suggests, these scripts test fio's zoned block device support. test-zbd-support runs up to 58 tests against a device, testing various aspects of fio's support for zoned storage. Tests ensure that error conditions are correctly handled, exercise many of the options related to fio's support for zoned storage devices, and carry out a variety of read and write operations using the libaio and sg ioengines. run-tests-against-nullb is a wrapper for test-zbd-support. It creates a zoned null block device and uses this device as a test target for test-zbd-support. This script has 13 sections covering various configurations of zoned null block devices. These sections have different configurations for the number of conventional and sequential zones, various combinations of zone capacity and zone size, and devices with and without limitations on the maximum number of open zones.

Fio's repository also contains a set of Python test scripts. These include:

As the file extension implies, this is a Python script, but it deserves special mention as it does not contain any actual test cases. Instead it is a wrapper script for the other test tools cataloged here. It runs the test jobs in t/jobs, fio's compiled test programs, the first two sections of run-tests-against-nullb, and fio's other Python test scripts. This script supports Linux, Windows, and macOS and is able to automatically skip test cases on platforms where they cannot be run. This script forms the basis of fio's continuous integration testing which will be described next.

How are all of these test components actually put into regular use? Fio relies on GitHub Actions for Linux and macOS continuous integration testing and AppVeyor for Windows continuous integration testing. A set of tests is run with every new push to the main branch of the fio repository on GitHub and for updated pull requests. Most of the setup for continuous integration testing is available in the repository's ci subdirectory.

Within the .github/workflows subdirectory of the fio repository are configuration files for GitHub Actions. ci.yml specifies four fio builds: 64-bit gcc, 64-bit clang, 32-bit gcc, and macOS. Inside the repository's ci subdirectory are a set of shell scripts that set up the environments and carry out the tests. actions-install.sh installs the required packages for building fio, actions-build.sh actually carries out the build, actions-smoke-test.sh carries out a quick, basic test of fio functionality using the Makefile's test target, and actions-fulltest.sh executes the run-fio-tests.py script and also does a test build of the HTML documentation.

On AppVeyor, fio is built on three Windows platforms: cygwin 64-bit, cygwin 32-bit, and msys2 64-bit. At ci/appveyor-install.sh is a shell script for creating the Windows build environments. This is a useful reference for developers building fio on Windows. Also useful is that installers are created as part of the build process and these can be downloaded as artifacts from AppVeyor. So the latest fio Windows builds are easily available. However, artifacts are available only for one month. So installer builds for tagged versions will no longer be available one month after a version is tagged. After the executable and installer are built, AppVeyor runs run-fio-tests.py and results are available in the test log. Test artifacts are also uploaded and can be examined.

At .github/workflows/cifuzz.yml are configuration settings for a Github Actions workflow that carries out fuzz testing for fio's option parsing routines. It builds the fuzzer and then feeds it the files in the fio repository's examples subdirectory. Fio then attempts to parse the job files as well as the image files contained therein. Any resulting crashes or memory leaks can be detected.

Fio actually has a substantial set of test components. However, the test components have not been systematically developed and cover only a subset of fio's features. There is no systematic test coverage of fio's myriad features. For instance, some basic options have no explicit test coverage. Does fio actually issue sequential IO when directed to? Are generated offsets actually random when fio is directed to issue random IO? Some of this functionality may be indirectly assessed by existing test jobs but the fact remains that no explicit testing exists for these and other basic features. Other areas that merit test coverage include client/server mode and bandwidth/IOPS logging. Adding tests for basic and more advanced fio features would guard against regressions and improve reliability.

As for existing test components, one area crying out for attention is the Python test scripts. Many of them share a common pattern where test objects are created for fio jobs. These test objects have methods to actually run the tests and evaluate the output for success or failure. Such scripts would benefit from having their duplicated elements abstracted into a testing library that could be imported into each of the separate test scripts. Finally it may be worthwhile to consider whether these test cases could be integrated into a formal test framework such as Gauge or whether the current homegrown framework is sufficient.

Most fio code has not been written in a way conducive to unit testing because functions in the fio source code typically depend on many different data structures that would have to be laboriously mocked up for unit testing. However, augmenting the existing unit tests where possible would also be a worthwhile contribution.

As for continuous integration testing, one improvement that could be made is to save the AppVeyor installer builds for tagged versions. AppVeyor artifacts expire after one month. So installers for tagged releases are available from AppVeyor for only a short period. If installers built on AppVeyor were, for instance, deposited on GitHub as releases, Windows users would have easier access to tagged versions.

The (free) continuous integration testing infrastructure also is unable to run some tests because of platform limitations. Thus, the t/zbd zoned block device scripts are not publicly run regularly. Not running these tests regularly is a lost opportunity.

Overall, fio has test coverage for many use cases. However, like almost all projects, its test coverage can be improved. Given how widely used fio is in the storage industry, improving test coverage would be of great benefit.