002_setup_tests.md - rapidsai/cucim GitHub Wiki
002 - Setup Tests
Created: 2021-06-14
Last-updated: 2021-06-14
Contents:
Summary
Issue
We need to set up tests for all modules including C++ and Python codes.
Currently, only scikit-image API (skimage
module) has unit test cases and automated testing setup and no automated tests are available for clara
module.
Test folder layout, test data, types of tests, libraries to use for testing need to be determined and properly documented.
Decision
Test folder layout
Use Tests outside application code
except skimage
Python module.
Test data
Generates test images at runtime
as possible.
For system tests, find representative test images and upload them to a publicly available site. Download/use those images locally.
Types of tests
- Unit tests
- Integration tests (integration with MONAI/Clara Train SDK)
- System tests
- Performance tests (load testing, checking overall memory usage)
Libraries to use
For C++:
Testing frameworks
- Use Catch2 over GoogleTest.
Benchmark
- Use GoogleBenchmark.
Dynamic Analysis
- Setup tests with Valgrind and Google Sanitizers.
For Python:
Testing frameworks
- Use PyTest.
Benchmark
- Use pytest-benchmark.
Profiler
- Provide a way to run a system test with Scalene.
Status
Decided. We are open to new alternatives as they arise.
Details
Assumptions
We want to set up automated testing on RAPIDS CI/CD infra.
We want to provide a way to execute small, medium, large/long-running tests.
We want to verify the integration with MONAI or Clara Train SDK.
We want to track the performance of loading/processing images.
We want to keep the current scikit-image API's test folder layout (tests
folder beside each source module folder) and focus on folder layouts of Clara C++/Python modules.
Constraints
Downloading large test data is not available on CI/CD infra.
Positions
Test folder layout
We considered:
- Tests outside application code
- Tests as part of application code
Test data
We can choose one of the following approaches depending on the use case:
- Generates test images at runtime
- Uses representative test images downloaded
Types of tests
We can include the following types of tests:
- Unit tests
- Integration tests (integration with MONAI/Clara Train SDK)
- System tests
- Performance tests (load testing, checking overall memory usage)
Libraries to use
For C++:
Testing frameworks
Benchmark
Dynamic Analysis
For Python:
Testing frameworks
Benchmark
Profiler
Argument
Test folder layout
-
Tests outside application code: Fits well for most of our cases (except
skimage
package) because we also want to include other types of tests including integration/system tests. -
Tests as part of application code: Good for unit tests and developers can easily correlate source code files with test files.
Test data
-
Generates test images at runtime: Good for small test cases (unit/integration tests) running at CI.
-
Uses representative test images downloaded: Good for long-running/system tests that require testing with realistic data.
Types of tests
- Unit tests: Can be run at CI.
- Integration tests (integration with MONAI/Clara Train SDK): Can be run at CI.
- System tests: We may want to provide a command to execute this test locally.
- Performance tests (load testing, checking overall memory usage): We may want to set up a different CI (NVIDIA internal infra) for this type of test.
Libraries to use
For C++:
Testing frameworks
- GoogleTest: Popular framework. Supports multi-threads.
- Catch2: No heavy dependency. Easy to implement test cases. See Why Catch.
Benchmark
- GoogleBenchmark: Popular framework. Provides good utility functions for the benchmark.
Dynamic Analysis
- Valgrind: We may want to run unit tests with valgrind.
- Related to #38.
- Google Sanitizers
- Related to #38.
For Python:
Testing frameworks (see https://www.softwaretestinghelp.com/python-testing-frameworks/)
- Built-in unittest package: No dependency with 3rdparty package but test code is verbose compared with PyTest.
- PyTest: Many functionalities and plugins. Easy to use.
skimage
module is already using this. - nose2: Extension of
unittest
.
Benchmark
- pytest-benchmark: This looks a good choice with PyTest (https://subscription.packtpub.com/book/application_development/9781787282896/1/ch01lvl1sec2/better-tests-and-benchmarks-with-pytest-benchmark).
Profiler
- Scalene: Looks better than existing profilers. See the PyCon 2021 video. We may want to use this locally during the development, instead of installing it at CI.
Implications
We may need to come up with a way to generate artificial test images and to find representative images for each image format.
Related
Related decisions
We will need to decide which system to use for measuring and tracking the performance.
Related requirements
The test cases implemented need to comply with Clara Train team's requirements (MONAI integration tests).
Related artifacts
We expect we may collect a set of test images and push it somewhere in public for local testing. We may want to create a test data repo in Github to store those images.
Related principles
Easy to implement test cases.