fio test coverage - vincentkfu/fio-blog GitHub Wiki
Fio's test coverage is far from complete. In fact seventy-three percent of fio's options are not exercised by its automated tests. This blog post is the first in a series that aims to construct a test development roadmap for fio. The first step is this post which describes the current state of test coverage for fio.
The scope of this blog post will be a list of fio's options accompanied by identification of the extent of automated testing for each option. The emphasis will be on end-to-end testing (as opposed to unit testing, for example). This assessment is based on my experience maintaining fio's continuous integration test suite and on the results of searching in fio's test scripts for each option. The focus of the assessment is on basic use cases for each option with most other options at their default values. Fio's options are so numerous that it would be impractical to exhaustively test all possible combinations of options. I classified option test coverage into four categories: basic, limited, no explicit testing, and none. Descriptions for each category and an overall summary are in the table below.
Test Coverage Summary
| Label | Count | Percentage | Description |
|---|---|---|---|
| basic | 23 | 6% | Option is (near) fully exercised with other options typically at their default values |
| limited | 32 | 9% | Option settings are partially exercised with other options typically at their default values |
| no explicit testing | 44 | 12% | Option is used in the course of testing other options |
| none | 263 | 73% | This option is essentially not used in any automated tests in the fio repository |
| Total | 362 | 100% |
Fio version: fio-3.35
We definitely have our work cut out for us if we are seeking any reasonable
level of test coverage. Nearly three quarters of fio's options are listed in
the category none as they are not exercised at all by automated testing. Six
percent of fio's options have basic testing where most of the option's possible
values are tested with remaining options mostly at their defaults. An example
of this are the options related to setting random seeds where combinations of
the different options are tested in t/random_seed.py. Nine percent of options
have limited testing where test cases include only a subset of its possible
values. The option cmdprio_percentage, for example, is used in some test
cases in t/latency_percentiles.py but the value is always set to 50 percent
(and there are no checks to see that the outcomes actually meet this
threshold). Finally, nine percent of options receive no explicit testing. This
category is for options such as filename which are widely used in fio's test
cases but no where does any test explicitly check that fio actually used the
specified file or device.
Categories of Options
Does examining test coverage for different categories of options provide any additional insight? The documentation divides fio's options into 23 categories. The largest ones are I/O engine specific parameters (94 options), Command-line options (36 options), Measurements and reporting (33 options), and Target file/device (30 options).
The greatest contributor to the global absence of test coverage is I/O engine specific parameters. The table below shows test coverage for options falling within this category.
I/O Engine Specific Parameters Test Coverage
| Coverage | Count | Percentage |
|---|---|---|
| basic | 0 | 0% |
| limited | 2 | 2% |
| no explicit testing | 0 | 0% |
| none | 92 | 98% |
| Total | 94 | 100% |
Fio has 94 I/O engine-specific parameters and has test coverage for only two of
them (limited testing for cmdprio_percentage and cmdprio_bssplit). The
remaining 92 options (e.g., fixedbufs, hipri, nfs_url, etc). do not
appear in any of fio's automated test scripts. These I/O engine options
comprise approximately one-third of all of fio's options that are not exercised
by automated tests. I/O engine options tend to be narrow in scope since they
typically apply only to no more than a handful of I/O engines. This partially
amerliorates the lack of test coverage for all of fio's options, but even if we
remove I/O engine specific options from the equation, options with no test
coverage at all still significantly outnumber those that do appear in automated
tests.
Command-line options and options related to Buffers and memory also contribute
greatly to the global lack of test coverage. As the table below shows,
Command-line options have 75% of the 36 options absent from any automated test.
These include options such as status-interval and trigger-file. The
command-line options that do have testing include readonly (basic features
tested in t/readonly.py) and output which receives no explicit testing but
is used widely across tests that inspect fio output.
Of the sixteen options related to buffers and memory, fifteen or 94 percent are
untested. These include options such as zero_buffers and iomem. The only
option from this set that is tested is buffer_pattern which appears in
t/jobs/t0027.fio and t/jobs/t0028-c6cade16.fio.
Test Coverage for Low Test Coverage Categories
| Coverage | Count | Percentage |
|---|---|---|
| Command-line options | ||
| basic | 2 | 6% |
| limited | 0 | 0% |
| no explicit testing | 7 | 19% |
| none | 27 | 75% |
| Total | 36 | 100% |
| Buffers and memory | ||
| basic | 0 | 0% |
| limited | 1 | 6% |
| no explicit testing | 0 | 0% |
| none | 15 | 94% |
| Total | 16 | 100% |
The other side of the coin is categories with relatively high test coverage.
These are categories where more than half of the options do have test coverage.
The categories include I/O type and Verification. As the table below shows, for
I/O type options, 10 percent have basic testing, 14 percent have limited
testing, 34 percent have no explicit testing, and the remaining 41 percent are
not covered by any testing. I/O type options that appear in automated tests
include options like offset and random_generator which both receive limited
testing in strided.py and the zbd test suite. Untested I/O type options
include random_distribution and percentage_random. random_distribution
allows users to specify non-uniform random distributions for random offsets
whereas percentage_random specifies the mix of random and sequential offsets
for random workloads.
For Verification options, 32 percent have limited coverage, 21 percent have no
explicit testing, and 47 percent have no test coverage at all. Verification
options covered by automated testing include do_verify and verify both of
which receive limited by jobs in t/jobs and the zbd test suite.
Untested Verification options include trim_percentage and trim_backlog both
of which are related to a fio feature that provides a means to trim blocks
after they are written.
Test Coverage for High Test Coverage Categories
| Coverage | Count | Percentage |
|---|---|---|
| I/O type | ||
| basic | 3 | 10% |
| limited | 4 | 14% |
| no explicit testing | 10 | 34% |
| none | 12 | 41% |
| Total | 30 | 100% |
| Verification | ||
| basic | 0 | 0% |
| limited | 6 | 32% |
| no explicit testing | 4 | 21% |
| none | 9 | 47% |
| Total | 19 | 100% |
Summary
The headline here is that fio's test coverage is poor and would benefit from significant improvement. Since fio is widely used by storage vendors and the Linux kernel community for validation, it is important for fio to function correctly. Part 2 of this blog post series will follow with a discussion of priorities for test development. Improving test coverage to any reasonable level will require substantial effort. How should we direct these efforts to obtain the greatest payoff?
The Appendix below contains a detailed breakdown of fio's option categories and how I classified each of fio's options with respect to test coverage. Raw data and scripts used for data analysis are available here.
Appendix
Frequency distribution of options by category
| Category | Option count | Percentage |
|---|---|---|
| I/O engine specific parameters | 94 | 26% |
| Command-line options | 36 | 10% |
| Measurements and reporting | 33 | 9% |
| Target file/device | 30 | 8% |
| I/O type | 29 | 8% |
| Threads, processes, and job synchronization | 23 | 6% |
| Verification | 19 | 5% |
| Buffers and memory | 16 | 4% |
| I/O rate | 12 | 3% |
| I/O replay | 12 | 3% |
| Act profile options | 7 | 2% |
| I/O depth | 7 | 2% |
| Time-related parameters | 7 | 2% |
| Block size | 6 | 2% |
| Tiobench profile options | 5 | 1% |
| I/O latency | 5 | 1% |
| I/O size | 5 | 1% |
| Steadystate | 4 | 1% |
| Error handling | 4 | 1% |
| Job description | 4 | 1% |
| Units | 2 | 1% |
| Predefined workloads | 1 | 0.3% |
| I/O engine | 1 | 0.3% |
Coverage for each of fio's options
| Category | Option | Coverage | Notes |
|---|---|---|---|
| Units | kb_base | none | |
| unit_base | none | ||
| Job description | name | no explicit testing | low priority |
| description | none | low priority | |
| loops | limited | a few tests in t/zbd/test-zbd-support | |
| numjobs | no explicit testing | ||
| Time-related parameters | runtime | no explicit testing | |
| time_based | no explicit testing | ||
| startdelay | none | ||
| ramp_time | none | ||
| clocksource | none | ||
| gtod_reduce | none | ||
| gtod_cpu | no explicit testing | ||
| Target file/device | directory | none | |
| filename | no explicit testing | ||
| filename_format | none | ||
| unique_filename | none | ||
| opendir | none | ||
| lockfile | none | ||
| nrfiles | none | ||
| openfiles | none | ||
| file_service_type | none | ||
| ioscheduler | none | ||
| create_serialize | none | ||
| create_fsync | none | ||
| create_on_open | none | ||
| create_only | none | ||
| allow_file_create | limited | two appearances in t/zbd/test-zbd-support | |
| allow_mounted_write | none | ||
| pre_read | none | ||
| unlink | none | ||
| unlink_each_loop | none | ||
| zonemode | basic | strided.py, t/zbd | |
| zonerange | basic | ||
| zonesize | basic | ||
| zonecapacity | basic | ||
| zoneskip | basic | ||
| read_beyod_wp | basic | ||
| max_open_zones | basic | ||
| job_max_open_zones | none | ||
| ignore_zone_limits | basic | ||
| zone_reset_threshold | basic | ||
| zone_reset_frequency | basic | ||
| I/O type | direct | no explicit testing | |
| buffered | no explicit testing | ||
| readwrite | no explicit testing | test should include rw=randread:8 | |
| rw_sequencer | no explicit testing | ||
| unified_rw_reporting | limited | t/latency_percentiles | |
| randrepeat | basic | t/random_seed.py | |
| allrandrepeat | basic | t/random_seed.py | |
| randseed | basic | t/random_seed.py | |
| fallocate | none | ||
| fadvise_hint | none | ||
| write_hint | none | ||
| offset | limited | t/strided.py, t/zbd, t/jobs/0002, t/jobs/0003 | |
| offset_align | none | ||
| offset_increment | no explicit testing | t/zbd | |
| number_ios | no explicit testing | t/jobs/t0009 | |
| fsync | limited | t/latency_percentiles | |
| fdatasync | none | ||
| write_barrier | none | ||
| sync_file_range | none | ||
| overwrite | none | ||
| end_fsync | none | ||
| fsync_on_close | none | ||
| rwmixread | no explicit testing | t/latency_percentiles, t/steadystate_tests | |
| rwmixwrite | no explicit testing | t/zbd | |
| random_distribution | none | ||
| percentage_random | none | ||
| norandommap | no explicit testing | t/latency_percentiles, t/strided, t/zbd, t/jobs/0007,0009,0021,0022,0023 | |
| softrandommap | no explicit testing | t/zbd | |
| random_generator | limited | t/sgunmap-perf, t/strided, t/zbd, t/jobs/t0021 | |
| Block size | blocksize | no explicit testing | |
| blocksize_range | no explicit testing | t/zbd, t/jobs/0001,0002,0023,0024 | |
| bssplit | no explicit testing | ||
| blocksize_unaligned | none | ||
| bs_is_seq_rand | none | ||
| blockalign | none | ||
| Buffers and memory | zero_buffers | none | |
| refill_buffers | none | ||
| scramble_buffers | none | ||
| buffer_compress_percentage | none | ||
| buffer_compress_chunk | none | ||
| buffer_pattern | limited | t0027, t0028 | |
| dedupe_percentage | none | ||
| dedupe_mode | none | ||
| dedupe_working_set_percentage | none | ||
| dedupe_global | none | ||
| invalidate | none | ||
| sync | none | ||
| iomem | none | ||
| iomem_align | none | ||
| hugepage-size | none | ||
| lockmem | none | ||
| I/O size | size | no explicit testing | t/latency_percentiles, t/strided, t/zbd, t/jobs |
| io_size | no explicit testing | t/strided.py, t/zbd | |
| filesize | no explicit testing | t/log_compression, t/random_seed, t-strided, t/jobs | |
| file_append | none | ||
| fill_device | none | ||
| I/O engine | ioengine | no explicit testing | |
| I/O engine specific parameters | cmdprio_percentage | limited | t/latency_percentiles |
| cmdprio_class | none | ||
| cmdprio | none | ||
| cmdprio_bssplit | limited | ||
| fixedbufs | none | ||
| nonvectored | none | ||
| force_async | none | ||
| registerfiles | none | ||
| sqthread_poll | none | ||
| sqthread_poll_cpu | none | ||
| cmd_type | none | ||
| hipri | none | ||
| userspace_reap | none | ||
| hipri_percentage | none | ||
| nowait | none | ||
| fdp | none | ||
| fdp_pli | none | ||
| cpuload | none | ||
| cpuchunks | none | ||
| cpumode | none | ||
| exit_on_io_done | none | ||
| namenode | none | ||
| port | none | ||
| hostname | none | ||
| serverip | none | ||
| direct_write_to_pmem | none | ||
| busy_wait_polling | none | ||
| interface | none | ||
| ttl | none | ||
| nodelay | none | ||
| protocol | none | ||
| listen | none | ||
| pingpong | none | ||
| window_size | none | ||
| mss | none | ||
| donorname | none | ||
| inplace | none | ||
| clustername | none | ||
| rbdname | none | ||
| clientname | none | ||
| conf | none | ||
| busy_poll | none | ||
| touch_objects | none | ||
| pool | none | ||
| cont | none | ||
| chunk_size | none | ||
| object_class | none | ||
| skip_bad | none | ||
| hdfsdirectory | none | ||
| verb | none | ||
| bindname | none | ||
| stat_type | none | ||
| readfua | none | ||
| writefua | none | ||
| sg_write_mode | none | ||
| stream_id | none | ||
| http_host | none | ||
| http_user | none | ||
| http_pass | none | ||
| https | none | ||
| http_mode | none | ||
| http_s3_region | none | ||
| http_s3_key | none | ||
| http_s3_keyid | none | ||
| http_s3_sse_customer_key | none | ||
| http_s3_sse_customer_algorithm | none | ||
| http_s3_storage_class | none | ||
| http_swith_auth_token | none | ||
| http_verbose | none | ||
| uri | none | ||
| gpu_dev_ids | none | ||
| cuda_io | none | ||
| nfs_url | none | ||
| program | none | ||
| arguments | none | ||
| grace_time | none | ||
| std_redirect | none | ||
| xnvme_async | none | ||
| xnvme_sync | none | ||
| xnvme_admin | none | ||
| xnvme_dev_nsid | none | ||
| xnvme_dev_subnqn | none | ||
| xnvme_mem | none | ||
| xnvme_iovec | none | ||
| libblkio_driver | none | ||
| libblkio_path | none | ||
| libblkio_pre_connect_props | none | ||
| libblkio_num_entries | none | ||
| libblkio_queue_size | none | ||
| libblkio_pre_start_props | none | ||
| libblkio_vectored | none | ||
| libblkio_write_zeroes_on_trim | none | ||
| libblkio_wait_mode | none | ||
| libblkio_force_enable_completion_eventfd | none | ||
| I/O depth | iodepth | no explicit testing | |
| iodepth_batch_submit | limited | t/sgunmap-test.py | |
| iodepth_batch_complete_min | limited | t/jobs/t0009 | |
| iodepth_batch_complete_max | none | ||
| iodepth_low | none | ||
| serialize_overlap | limited | t/jobs/0013 | |
| io_submit_mode | limited | t/jobs/0010. t/jobs/0013 | |
| I/O rate | thinktime | none | |
| thinktime_spin | none | ||
| thinktime_blocks | none | ||
| thinktime_blocks_types | none | ||
| thinktime_iotime | none | ||
| rate | none | ||
| rate_min | none | ||
| rate_iops | limited | t/jobs/0011 | |
| rate_iops_min | none | ||
| rate_process | none | ||
| rate_ignore_thinktime | none | ||
| rate_cycle | none | ||
| I/O latency | latency_target | none | |
| latency_window | none | ||
| latency_percentile | none | ||
| latency_run | none | ||
| max_latency | none | ||
| I/O replay | write_iolog | limited | t0007 |
| read_iolog | none | ||
| read_iolog_chunked | none | ||
| merge_blktrace_file | none | ||
| merge_blktrace_scalars | none | ||
| merge_blktrace_iters | none | ||
| replay_no_stall | none | ||
| replay_time_scale | none | ||
| replace_redirect | none | ||
| replay_align | none | ||
| replay_scale | none | ||
| replay_skip | none | ||
| Threads, processes, and job synchronization | thread | no explicit testing | t/jsonplus2csv_test, steadystate_tests, t/zbd, t/jobs |
| wait_for | no explicit testing | t/zbd | |
| nice | none | ||
| prio | none | ||
| prioclass | none | ||
| cpus_allowed | no explicit testing | t/jobs/0009 | |
| cpus_allowed_policy | no explicit testing | t/jobs/0009 | |
| cpumask | none | ||
| numa_cpu_nodes | none | ||
| numa_mem_policy | none | ||
| cgroup | none | ||
| cgroup_weight | none | ||
| cgroup_nodelete | none | ||
| flow_id | limited | t0011, t0012, t0014 | |
| flow | limited | t0011, t0012, t0014 | |
| flow_sleep | no explicit testing | t0012 | |
| stonewall/wait_for_previous | no explicit testing | ||
| exitall | none | ||
| exit_what | none | ||
| exec_prerun | none | ||
| exec_postrun | none | ||
| uid | none | ||
| gid | none | ||
| Verification | verify_only | none | |
| do_verify | limited | t/zbd, t0002-t0006, t0008, t0009, t0024-0027 | |
| verify | limited | t/zbd, t0002-t0006, t0008, t0009, t0024-0027 | |
| verify_offset | none | ||
| verify_interval | no explicit testing | t0004 | |
| verify_pattern | limited | t0006, t0027 | |
| verify_fatal | limited | t0002, t0003 | |
| verify_dump | no explicit testing | t0003, t0004 | |
| verify_async | no explicit testing | t0009 | |
| verify_async_cpus | no explicit testing | t0009 | |
| verify_backlog | limited | t0005, t0006, t0008, t0009, t/zbd | |
| verify_backlog_batch | none | ||
| verify_state_save | none | ||
| verify_state_load | none | ||
| trim_percentage | none | ||
| trim_verify_zero | none | ||
| trim_backlog | none | ||
| trim_backlog_batch | none | ||
| experimental_verify | limited | t0025, t0026 | |
| Steadystate | steadystate | basic | t/steadystate_tests.py |
| steadystate_duration | basic | t/steadystate_tests.py | |
| steadystate_ramp_time | basic | t/steadystate_tests.py | |
| steadystate_check_interval | limited | t/steadystate_tests.py | |
| Measurements and reporting | per_job_logs | limited | t0019-t0024, t/log_compression.py |
| group_reporting | limited | t0002-t0004, t0009, t/latency_percentiles, t/zbd/test-zbd-support | |
| new_group | none | ||
| stats | none | ||
| write_bw_log | limited | t0019-0024, t/log_compression.py | |
| write_lat_log | limited | latency_percentiles.py | |
| write_hist_log | none | ||
| write_iops_log | limited | t0012, t0014, strided.py | |
| log_entries | none | ||
| log_avg_msec | limited | t0012, t0014 | |
| log_hist_msec | none | ||
| log_hist_coarseness | none | ||
| log_max_value | none | ||
| log_offset | limited | t0019-t0024, log_compression.py, strided.py | |
| log_compression | basic | log_compression.py | |
| log_compression_cpus | none | ||
| log_store_complressed | basic | log_compression.py | |
| log_unix_epoch | none | ||
| log_alternate_epoch | none | ||
| log_alternate_epoch_clock_id | none | ||
| block_error_percentiles | none | ||
| bwavgtime | none | ||
| iopsavgtime | none | ||
| disk_util | none | ||
| disable_lat | no explicit testing | t/zbd | |
| disable_clat | none | ||
| disable_slat | none | ||
| disable_bw_measurement | none | ||
| slat_percentiles | basic | t/latency_percentiles | |
| clat_percentiles | basic | t/latency_percentiles | |
| lat_percentiles | basic | t/latency_percentiles | |
| percentile_list | none | ||
| significant_figures | no explicit testing | t/zbd/test-zbd-support | |
| Error handling | exitall_on_error | no explicit testing | t/zbd/test-zbd-support |
| continue_on_error | limited | t/zbd | |
| ignore_error | none | ||
| error_dump | none | ||
| Predefined workloads | profile | none | |
| Act profile options | device-names | none | |
| load | none | ||
| test-duration | none | ||
| threads-per-queue | none | ||
| read-req-num-512-blocks | none | ||
| large-block-op-kbytes | none | ||
| prep | none | ||
| Tiobench profile options | size | none | |
| block | none | ||
| numruns | none | ||
| dir | none | ||
| threads | none | ||
| Command-line options | debug | no explicit testing | t/random_seed, t/steadystate_tests, t/zbd/test-zbd-support |
| parse-only | no explicit testing | t/steadystate_tests | |
| merge-blktrace-only | none | ||
| output | no explicit testing | various | |
| output-format | no explicit testing | various | |
| bandwidth-log | none | ||
| minimal | none | ||
| append-terse | none | ||
| terse-version | none | ||
| version | none | ||
| help | none | ||
| cpuclock-test | none | ||
| crctest | none | ||
| cmdhelp | none | ||
| enghelp | none | ||
| showcmd | none | ||
| readonly | basic | t/readonly.py | |
| eta | none | ||
| eta-interval | none | ||
| eta-newline | none | ||
| status-interval | none | ||
| section | none | ||
| alloc-size | no explicit testing | t/zbd/test-zbd-support | |
| warnings-fatal | none | ||
| max-jobs | no explicit testing | various | |
| server | none | ||
| daemonize | none | ||
| client | none | ||
| remote-config | none | ||
| idle-prof | none | ||
| inflate-log | basic | t/log_compression.py | |
| trigger-file | none | ||
| trigger-timeout | none | ||
| trigger | none | ||
| trigger-remote | none | ||
| aux-path | no explicit testing | t/zbd/test-zbd-support |