Feature: run only selected tests - default-to-open/rpmgrill GitHub Wiki
RpmGrill: allow filtering by test
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1202633
Changes to user interface
Listing all plugins and its tests
rpmgrill --list-plugin-tests
plugin a : description of plugin
- test : foobar
- test 2 : does something else
plugin b
- test
list tests of some plugins
rpmgrill --plugin x,y,z --list-tests
will list all tests of plugin x, y, and z
plugin x : description of plugin
- test a : foobar
- test b : does something else
plugin y
- test
- ...
Run: only some plugins
rpmgrill --run x,y,z --exclude a,b,z
will run all tests of x,y plugin
NOTE: it will not run z
since it is the exclude list
Run: only some tests of some plugins
rpmgrill --run x:t1,t2,t3,y:,z:t2
# -- is same as --
rpmgrill --run x:t1,t2,t3 --run y --run z:t2
will run - t1, t2, t3 tests of plugin x - all tests of plugin y - t2 tests of plugin z
NOTE: when user filters by test, (s)he needs to use y:
to indicate y is plugin
and not a test in plugin x
What needs to be changed
How it works currently
RpmGrill uses Module::Pluggable
to load all plugins under lib/Rpm/Grill/Plugin/*.pm
rpmgrill script calls Rpm::Grill::plugins
which lists all available plugins.
it then invokes $grill->invoke_plugin($plugin)
which in turn calls $plugin->analyze()
Inside a plugin, analyze
is the main method, which calls sub-methods and deep within
the call tree is where the actual analysis is done.
Changes needed to support sub-test filtering
To support sub-test filtering, we need to be able to run individual sub-tests, so the plugins should start returning a list of TestCodes it supports and be able to run it.
approach 1: call any test
@tests_in_plugin = $plugin->tests();
$plugin->test('FooBar')->run()
instead of $plugin->analyze which runs all tests - currently
--- or ---
approach 2: pass sub-tests to run and let plugin do the work
$plugin->analyze(List, of, TestCodes, To, Run);
and it is up to plugin to decide how to use the list of TestCodes passed and run only those. This may result in faster runs since the plugin gets to optimise common method like: setup, cleanup and just have one outer loop.
Question: wouldn't every plugin then have repeated
Preferred approach
Both approaches involve changing every plugin but a nice thing about the first one is that, Grill.pm which runs each plugin can use the Module::Pluggable::can
and test if tests
sub exist and use that to run individual tests.
### pseudo-code
Grill::invoke_plugin($plugin, @disabled_tests) {
if ( $plugin->can('tests') ) {
# filter out disabled tests
foreach $test (filter_out_disabled(@disabled_tests)) {
$plugin->test($test)->setup()
$plugin->test($test)->run()
$plugin->test($test)->cleanup()
}
} else {
$plugin->analyse;
}
}
Questions/Doubts/Conclusion
Does this all make sense?
Isn't all this equal to separating out each test in a plugin into another plugin?