This is a living document of testing standards to use across the org. As issues are found and addressed, learnings will be added to this doc so that everyone can benefit.

Unit testing

Language/Framework-specific Guidelines

JavaScript/React

When writing unit tests in JavaScript, the test files should be located in the same directory, and share the same name, as the files they are testing, rather than putting all tests in a separate directory structure. For example, if a React component is defined in src/common/components/my-component.jsx, its tests should be in src/common/components/my-components.test.js. Note that the only difference in the paths is the file extension (.jsx vs. .test.js). Organizing tests this way allows the tests to be found much more easily and makes them more resilient to refactoring if files are moved around.

Snapshot Testing

For React, unit tests should utilize Snapshot tests as much as possible for validating the structure of HTML components. In general, these tests should capture all of the non-interactive features of a component, such as text, className, etc. Most changes to these types of features are stylistic and subjective in nature, rather than being directly tied to a functional requirement, so snapshot tests allow developers to quickly identify any unwanted changes or update the snapshots for acceptable changes. Non-static or interactive features, such as login, onClick scripts, menu navigation, etc. should be tested with more traditional unit tests. For more information about snapshot tests, see the link above, or see this blog post for more details about when it is appropriate to use snapshot tests.

Updating snapshots

When a snapshot test fails, Jest prints out a diff between the generated HTML and the expected HTML. After reviewing all snapshots and addressing any unwanted changes, you can accept all remaining changes and update the snapshots by running npx jest -u. After doing so, make sure to double-check the changes to the snapshot files in git before pushing your changes.

Code coverage

Code coverage metrics should be collected for all projects. We use codecov to generate our code coverage reports, because it supports a wide variety of programming languages and test frameworks, and is free to use for open-source projects.

Coverage threshold

All projects should enforce a 90% code coverage requirement. This means that all new PRs must have at least 90% coverage for changed code and not decrease the overall code coverage for the project. This can be configured in your project's codecov.yml config. The patch target should be set to 90% or higher, and the project target should be set to "auto" (meaning the overall code coverage must not decrease).

Onboarding codecov

To onboard codecov to your project, you will need a codecov.yml config file and a GitHub Action to generate and upload your code coverage metrics. You can copy these from an existing project that already has codecov onboarded, such as WebApp. TODO: link to the actual files in WebApp once merged. There are a few considerations when onboarding:

1. Codecov does not generate code coverage metrics itself. It only generates the reports and runs GitHub status checks to ensure that the configured thresholds are met. Before uploading to codecov, you must generate the metrics using your project's chosen framework. Please see https://docs.codecov.com/docs/supported-languages for language-specific examples.

2. Make sure untested files are reported! Some test frameworks (such as Jest), by default, only report test coverage for files that already have at least 1 test. Make sure to configure your framework to include all testable files, so that untested files will be correctly reported as 0% covered instead of being completely ignored. For example, with Jest, you can use the collectCoverageFrom config to include all *.js and *.jsx files in your src directory.

3. Configure your GitHub Action to trigger on both PRs and merges into the test/dev/main branches.

4. Test your reports! After configuring codecov and setting up your GitHub Action, be sure to test the reports to make sure they correctly report coverage of changed lines. To do this, create a new temporary branch based on your codecov branch (call it something like codecov-test) and make a few temporary changes to the source code anywhere. At minimum you'll need to make a change that is covered by a test and a change that is not covered by a test (don't add a test for it!), and to make sure that less than 90% of your changes are covered by tests. Then you'll need to create a test PR to see how the codecov comment and report look on your PR:

   1. In your codecov branch, temporarily configure the GitHub Action to trigger on pushes into your codecov branch, then push the change. This should trigger the action to run on GitHub, which will generate a baseline coverage report. Once the action completes, revert the config change.
   2. In your codecov-test branch, change your GitHub Action to trigger on PRs into your codecov branch.
   3. Push your changes in your codecov-test branch and create a temporary PR from codecov-test into your codecov branch.
   4. Wait for the build to complete, then check the comment that codecov generates. It should report that coverage for your changes are under the 90% coverage threshold and present a list of changes files with their coverage scores. It should also give a link to the full codecov report, which includes coverage stats for all files and directories (this was generated in the baseline report from step i above).
   5. If the report looks correct, delete your temporary test branch and close the test PR (DO NOT MERGE). You are now ready to create the real PR to add codcov to your project!

5. Once your codecov PR is merged, all new PRs will start running codecov. If a PR has insufficient coverage, the coverage check in the PR will show as failed, but it will not yet block merging of the PR. In order to enforce the code coverage threshold, an admin must configure the branch protection rulesets for your GitHub repo to require the codecov check to pass before merging.