UCX Testing Procedure - openucx/ucx GitHub Wiki

The following page describes the testing and verification procedures that are currently being done within the UCX project.
In order to make UCX stable and resilient to potential bugs, please read the following mandatory actions that need to be taken when contributing to the project.

  1. Before submitting a Pull Request to the UCX project, please make sure to:
  • Add a unit-test in gtest for it.
    gtest was integrated into the UCX project and the tests there are organized according to the project’s structure. Please add your test to the relevant part – UCP/UCT/UCS. Tests are compiled with the C++11 standard flag, meaning most of the language features can be used in test code.
  • Test the new code you added as much as possible:

    • If you fixed a bug, please make sure that corner cases are covered and that running the test in a loop also passes.

    • If you added a new feature, besides adding a gtest for it, please test it on the largest scale you can. Testing a new feature on a low number of processes is usually not enough to conclude that it works properly.
      For example, if you added a new feature to the shared memory transport, please run a test that would use all the cores on your host and make sure that the transport remains intact. Check all the available devices of the transport, the environment parameters that may be relevant and running a data-check test is always a good idea. When testing, please use MPI or another library that uses UCX.
      Here is an example of testing UCX on a single host with the IMB benchmark:
      mpirun –H host1 -np 16 -mca pml ucx -x UCX_TLS=mm -x UCX_SHM_DEVICES=sysv --bind-to core --map-by core --display-map IMB-MPI1

    • If your new feature was added to the UCP layer, please be sure to test all the transports/devices/link layers that were affected.

    • If your code has any potential performance improvements or if you think that it may cause a performance degradation, please test it with ucx_perftest.
      Note: when testing UCX’s performance, make sure that UCX was compiled in the ‘release’ mode. Use the ‘contrib/configure-release’ file for this.
      Please mention in your PR that the UCX performance may be affected by it.

  1. Once you submit your Pull Request, Jenkins will be triggered. Jenkins will test your new code on top of the latest master branch of UCX. It will test compilation on various operation systems, on several CPU architectures and run all the gtest test suite (on one host), which includes all the gtest tests along with testing potential memory leaks with the help of Valgrind. Coverity checking will also run to analyze the code.
    If you wish to run Jenkins on your host, simply run this file: ‘contrib/test_jenkins.sh’
    Please note: PRs that didn’t pass Jenkins, will not be accepted.

  2. UCX is also tested with the MPI-Testing-Tool (MTT). It is run daily and serves as the regression tool for UCX. MTT consists of many benchmarks and applications and tests many different command line permutations that invoke UCX – transports, devices, environment parameters, different scales and tests arguments are all tested.
    If you added a new environment parameter and/or a new feature, it will probably be added to MTT.