What's wrong with tarantool tests? - AnaNek/tarantool GitHub Wiki

Diff-based approach (pro et contra)

contra: write easy, maintainance is hard
contra: explicit test oracle, test is useless without verified .result file
contra: fragile because depend on formatting
contra: one cannot create a test when tested functionality is not ready, because output is a mandatory part of a test
contra: during review you should inspect both artifacts: source code of a test and it's output. It's double efforts for reviewers.
pro: you can easily reproduce a bug with copy-paste test source code in interactive mode

test-run.py and test harness limitations

Can't use remote instances
No support of additional test statuses like XFail, XPass. Instead we comment blocked test out and uncomment it after a fix.
Low code reuse among different tests. Only basic and often used operations are in a test_run.lua and it's number is small.
Pure tests isolation, test should be hermetic. Many tests doesn't restore environment state at the end of testing and new test starts in a "dirty" environment.
Pure parametrization of tests. One cannot easily rerun async replication tests for synchronous replication.
test-run.py lack features existed in a popular testing framework for years: just take a look to a pytest for example.
Pure testcase management: cannot to run single testcase in a file
One cannot use common fixtures in a number of tests, instead he should implement new functionality in a test-run.py