Non deterministic Test Tracker - CDCgov/prime-simplereport GitHub Wiki

Background

SimpleReport engineers recently identified a pain point pertaining to CI/CD: non-deterministic or "flaky" tests.

Non-deterministic tests cannot be guaranteed to behave predictably, and may cause CI/CD failures even if the underlying code is healthy.

Below, please document any such automated tests. Unit tests, integration tests, and end-to-end tests are all relevant to this effort and may all be tracked on this page. Engineers did a flaky-test bug bash session in the 9/7 eng sync. Agenda and notes can be found here.

Tests

A log of flaky tests are kept here

Reproducing

• Java, use Intellij's repeat until failure can help make it troubleshoot
• Javascript, use jest's test.each() to repeat the test many times.