• What to test
• Java unit tests
  • Attaching a debugger to unit tests
  • Using the @VisibleForTesting annotation
  • Controller tests
  • View tests
• TypeScript unit tests
• Functional browser tests
  • Running browser tests
  • Guidelines for functional browser tests
    • Browser test reliability
    • Screenshot diffing
    • De-randomizing
    • Axe accessibility tests
  • Debugging browser tests
    • Playwright's UI mode
    • Local debug mode
    • Debugging failed GitHub actions
    • Running probers locally against staging
  • Running and Debugging in VS Code
    • Run and debug outside of Docker
    • Run inside of Docker
  • Using Playwright Trace Files
    • Downloading Trace Files
    • Viewing Traces
      • View trace files on Playwright's Trace Website
      • View traces on local playwright server
• Screen reader testing

This guide offers best practices for writing unit and browser tests for CiviForm, as well as debugging tips and practices.

In general, all execution paths in the system should be covered by java unit tests or typescript unit tests. If you submit code that is infeasible or impractical to get full test coverage for, consider refactoring. If you would like to make an exception, include a clear explanation for why in your PR description.

In contrast, browser tests should cover all major user-facing features to make sure the system generally works from a user's point of view, rather than exhaustively test all execution paths.

For Java, classes generally have their own unit tests. The unit test file should mirror the implementation file. For example, /app/package/path/MyClass.java should have a unit test /test/package/path/MyClassTest.java.

Tests that require a Play application should either use extends play.test.WithApplication, or extends repository.WithPostgresContainer if a database is required. By default, using extends play.test.WithApplication will produce an application with a binding to an in-memory postgres database that is incompatible with everything and is pretty much useless.

To run the unit tests (includes all tests under test/), run the following:

bin/run-test

If you'd like to run a specific test or set of tests, and/or save sbt startup time each time you run the test(s), use these steps:

bin/sbt-test

• Run a subset of all test suites

testOnly services.export.*

• Run specific test suite

testOnly services.export.JsonPrettifierTest

• Run specific test

testOnly services.export.JsonPrettifierTest -- -z *asPrettyJsonString_prettyPrintsJsonString

When running an individual unit test via bin/sbt-test, a debugger can be attached. In order to support this, JVM forking must be disabled since the debugger needs a well-known port to attach to. Within build.sbt, uncomment the following line:

Test / fork := false,

Then, use your debugger of choice to attach to port 8459 (VSCode workspace configuration already has a configuration for this).

Note that this causes the configuration override not to have an effect. This means that overriding feature flags in tests will not work and code gated by a feature flag will not be run.

When attaching, a deadlock can occur if trying to attach too early. Consider waiting until the log line indicating connection to the database succeeded before attaching a debugger.

The @VisibleForTesting annotation puts a restriction on the method you are annotating so that it can only be called from code contained in tests. It does not change the access level of the method, so you will still need to make the method accessible to your test (eg. public if it's not contained in the same package as your test).

Controller tests should test the integration of business logic behind each HTTP endpoint. Most controller tests should likely extend WithPostgresContainer which provides a real database. Controllers should contain very little if any conditional logic and delegate business logic and network interactions (database, auth service, file services, etc.) to service classes.

• Assertions should be on the method's Result rather than the rendered HTML.
• Assertions may also be on the database state after the controller method has completed.
• Controller tests should not rely heavily on mocking.

See AdminProgramControllerTest.java for a good example of a controller test. See the Play documentation for information on framework-provided testing tools.

BaseHtmlView provides a number of HTML tag-producing methods, for example Tag submitButton(String textContents). These methods tend to be fairly simple, with unit tests that are brittle to small, inconsequential changes. Whether or not to test these types of methods is at the discretion of the implementer and code reviewer(s).

View classes that render a complete page should not be unit tested, but instead should have corresponding browser test(s) that assert the key interactions for a user on that page.

Question type rendering and client-side logic deserves a special mention since they can have complex interaction logic. These should be unit tested in isolation, in browser test(s).

Add models to the list of models in Model.java so that the test database will be cleared between tests.

For TypeScript code in server directory each file should generally have corresponding unit test file. The unit test file should be placed in the same directory as implementation file use pattern <impl>.test.ts.

Unit tests use Jest test runner.

Unit tests are run using Node.js. By default, Node.js doesn't provide browser APIs such as DOM (document, querySelector, etc). At the same time client-side typescript code being tested assumes that it is executed in browser environment and assumes standard browser APIs are available. To workaround this issue we use jsdom environment in Jest. It adds a fake implementation of main browser APIs. Developers should keep that in mind as fake implementations will differ from actual browser API implementations in certain cases and don't support all browser APIs.

To run the unit tests (includes tests under app/assets/javascripts), run the following:

bin/run-ts-tests

If you'd like to run a specific test or set of tests, run the following:

bin/run-ts-tests file1.test.ts file2.test.ts

Functional browser tests use the Playwright browser automation TypeScript library.

The code for those tests lives in the browser-test/ subdirectory.

Browser tests run against an application stack that is very similar to the local development stack. The test stack has its own application server, postgres database, and fake IDCS server that all run in Docker, separate from the test code. The test stack is intended to stay up between test runs to reduce the iteration time for running the tests while developing.

To run the tests:

1. (Optional) If you have made any changes to the browser test container, rebuild the Docker image:

   bin/build-browser-tests
   

2. Bring up the local test environment with the AWS emulator. This step can be done in a separate terminal window while the Docker image is still building.

   Leave this running while you are working for faster browser test runs:

   bin/run-browser-test-env
   

   TIP: the server is accessible manually at http://localhost:9999.

   To run browser tests with the Azure browser test environment, using Azurite (the Azure emulator) instead of the AWS emulator, run:

   bin/run-browser-test-env -–azure
   

   This runs the tests using Azurite, the Azure emulator. Because the Azure deployment of CiviForm requires SES, the AWS email sending service, we also have to start Localstack, the AWS emulator, when running the Azure browser tests.

3. Once you see "Server started" in the terminal from the above step, in a separate terminal run the Playwright tests in a docker container:

   bin/run-browser-tests
   

   To run the tests in a specific file, pass the file path relative to the browser-test/src directory. For example:

   bin/run-browser-tests landing_page.test.ts
   

   Use the --debug flag to print debug logs as the test runs.

   To run a single test within a file, pass the line number of the test line in addition to the file name:

   bin/run-browser-tests landing_page.test.ts:###
   

   Alternatively, you can edit the test file to temporarily replace test with test.only:

   test.only("my test", async => { ... })

In contrast to unit tests, browser tests should not attempt to exhaustively test all code paths and states possible for the system under test. Browser tests should:

• be fewer and larger, covering major features of the application
• only create state in the database by interacting with the UI (e.g. when testing the applicant experience for answering of a certain type, first login as an admin, create a question and a program with that question)
• encapsulate UI interaction details into page object classes
• as much as is practical navigate using the UI and not by directly referencing URL paths
• use IDs and classes to select elements, rather than selecting based on text inside the element, so the test does not need to change when copy changes. If you only need the ID for testing purposes, consider setting data-testid (using .withData("testid", "value")) and then Playwright's page.getByTestId().

Screenshot diff tests should cover every question type, and should cover every page of the admin and applicant flow. See the screenshot diffing section for more details.

Accessibility (a11y) testing also needs to happen at the browser test level, since it checks the final generated HTML for a11y violations. All applicant facing features should be covered by an a11y test check. See the accessibility test section for more details.

Browser tests can be flaky if they trigger actions on the page and don't wait for CiviForm javascript event handlers to complete before triggering subsequent actions or validating page contents. Some examples of where this can happen:

• Page initialization: CiviForm typically attaches javascript event handlers after pages load. Tests must therefore wait for pages to be ready after initial page load or any page navigation (whether triggered by anchor link clicks, form submissions, or js event handlers). To accomplish this, main.ts and modal.ts set data attributes on the html tag when they are done running, and the browser test function waitForPageJsLoad can be used to wait for these attributes to be set. In general, stay very aware of when page navigations are happening to maintain correctness.
• DOM modification: CiviForm sometimes uses javascript to show/hide DOM elements like modal dialogs, or makes copies of hidden templates (e.g., to populate radio/checkbox option lists). Browser tests can use element selectors to block on these manipulations finishing, but selectors must be specific enough to differentiate (e.g., waiting for a specific matching element index to appear, instead of targeting the last match). For typical CiviForm modal dialogs, clickAndWaitForModal may be helpful.
• Input validation: CiviForm javascript input validators sometimes modify the DOM (e.g., making sure text has been changed before enabling a submit button). Browser tests can use specific selectors to have playwright wait for input validation to complete (e.g., specifying an enabled button to click instead of just specifying the button).

The beforeAll and afterAll functions inherently cause tests to become less isolated by creating shared resources. The preference should be to use beforeEach and afterEach. If you need to assert multiple things on a page do so in the same test function.

Use the test.step function to wrap multiple lines together into a logical group. This is preferred to using comments as the steps will show up in Playwrights reporters (i.e. such as the trace file). You'll probably see older tests using comments to group blocks of code, please update them to use steps if you happen to modify them.

Update screenshots locally with -u:

bin/run-browser-tests some_file.test.ts -u

Update screenshots from GitHub with:

bin/sync-failed-browser-test-images

Screenshot tests are implemented with playwright. To add a screenshot test, simply call: await validateScreenshot(page, 'name-of-image'). When testing a particular component or section of the page, prefer capturing a smaller screenshot by passing a selector rather than the whole page, like this: await validateScreenshot( page.locator('.some-class'), 'name-of-image', )

New screenshots are saved automatically to a subdirectory in .../image_snapshots/ with the test file name. If a screenshot diff is found, an image showing the diff will be saved to .../diff_output/. To accept the diff as expected and update the screenshot, re-run the test with the -u or --update-snapshots flag (e.g. bin/run-browser-tests -u some_file.test.ts).

Note that we've disabled screenshot tests when run locally (e.g. with bin/run-browser-tests-local) to minimize variability. They are enabled when running with the usual command via docker (bin/run-browser-tests).

When run as a GitHub action, screenshot diff images will be uploaded on test failure. These are available in the Artifacts section of the Summary tab on the GitHub action run. You can pull them from GitHub to your working copy with bin/sync-failed-browser-test-images.

Timestamp/Dates and applicant IDs will change each time a test is run, to automatically normalize these, UI elements that contain them need to have the ReferenceClasses BT_DATE, BT_APPLICATION_ID and a few others classes added respectively. Check the normalizeElements function for an up-to-date list of elements and update if necessary.

Accessibility tests are run at the browser test level on the final generated HTML page, using axe. You can run an accessibility test on a page simply by calling: await validateAccessibility(page)

If the accessibility test fails, the error message will output the AxeResults.violations array. See API docs for more info. The violation will include a helpUrl with a link of suggestions to fix the accessibility problem, and will include a snippet of the problematic html. Running the tests locally with debug mode is particularly helpful here since you can manually inspect the html.

Please see the Playwright debug docs for a lot more info on this topic.

Set the env var RECORD_VIDEO=1 to tell playwright to record a video of the test run. Videos are available in browser-tests/tmp/videos after the test run completes.

Playwright's UI Mode lets you run, pause, and debug tests from a UI, even if the browser under test is in headless mode in a docker container or Codespace. It also lets you test and debug locators.

To use UI mode:

1. Start the test server, either in a Codespace or docker container, as normal, with bin/run-browser-test-env. It doesn't need to be the local version.
2. Make sure the test server port 9999 is open to your local machine. In a Codespace, you can open that port from the Terminal window of the VS Code instance that's connected to your Codespace.
3. From your local machine, not the Codespace, run bin/run-browser-tests-local mytest.test.ts --ui to launch the test runner in UI mode. Note that this will use the code checked out to your local machine, not the version in your Codespace. You may need to push your changes from you Codespace and then pull them back to your local machine in order to test them.

You can step through a test run line-by-line with a visible browser window by running the tests locally (i.e. not in Docker) with debug mode turned on.

Note: These instructions need some work

You will need to start the browser test environment by running:

bin/run-browser-test-env --local

Because this exposes port 3390 for the oidc-provider container, this can not be run concurrently with bin/run-dev. (Note: You will also need to manually kill the civiform container in Docker.)

To run them in debug mode with the open browser add the PWDEBUG environment variable:

PWDEBUG=1 bin/run-browser-tests-local

You can find more documentation on debugging Playwright in this BrowserStack guide. A few tips:

• During debugging, a separate window will open which allows you to play and pause the browser test.
• await page.pause() is particularly useful. If you add it to the test you're debugging, then it acts as a breakpoint and always pauses the test at that line. You can then step over each call to manually move the browser test through the test. At any pause point, you can also manually inspect the DOM.

On failure, a test video of the browser test will be uploaded to the Artifacts section of the Summary tab on the GitHub action run.

Sometimes, we'll find a test that passes when run locally via bin/run-browser-tests, but fails when run via probers on staging deploy. These can be tricky to debug. You may want to modify some tests in your local Civiform repo clone and try them against staging.

You can run a locally-modified browser test against AWS staging with the following command. You should create a test auth0 account for this purpose. Do not use your admin auth0 account.

$ BASE_URL=https://staging-aws.civiform.dev TEST_USER_LOGIN=<your auth0 user> TEST_USER_PASSWORD=<your auth0 password> bin/run-browser-tests-local <test-file-name>

1. Install the Playwright Test for VSCode extension.
2. Click the manage (gear icon) button for the extension > extension settings.
3. Click Edit in settings.json.
4. Set the playwright.env key to the following settings:

"playwright.env": {
    "BASE_URL":"http://localhost:9999",
    "LOCALSTACK_URL": "http://localhost.localstack.cloud:6645",
    "DISABLE_SCREENSHOTS": true
}

5. Start bin/run-browser-test-env --local
6. In another terminal, run bin/run-browser-tests-local. This lets VSCode index the browser tests.
7. In VSCode, click the green "Play" button next to the file or test you want to run.

Note: If the "Play" button doesn't show up, you may need to open just the browser-tests folder in VSCode. Tests that validate screenshots are likely to fail due to rendering differences to the docker container.

• Install the Dev Containers extension.
• Start bin/run-browser-test-env.
• Run bin/run-browser-tests once to get the container running.
• In VSCode click on the Remote Explorer to go to the Dev Containers.
• Click civiform-browser-test-runner to attach to the container, accept prompts to install components.
• Install the Playwright Test for VSCode extension. This is installed inside the container.
• Open the tests at the path /usr/src/civiform-browser-tests.

At this point you can run Playwright tests.

Use the bin/dev-show-trace-file script to easily view a trace file. This script will let you select which run and which artifact to load into the the Playwright Trace Viewer.

When running browser tests in a codespace/VSCode, traces are found in:

browser-test/tmp/test-output/

Right-click the file and click "Download".

Video and Trace artifacts can be downloaded from a failed GitHub action run. They are both found in test video (aws) - batch-#-run-# artifacts.

Drag and drop any trace.zip file into https://trace.playwright.dev

For traces run in CI, Download and unzip the artifact, then run:

npx playwright show-report "path/to/html-output"

For traces generated on your local computer or in a codespace run:

npx playwright show-report "browser-test/tmp/html-output"

If you're using a Codespace, you may need to forward the trace server port 9323 to your local machine.

For any UI changes on the applicant side, please test manually with at least one screen reader. The easiest is the built-in screen reader (see below for shortcuts). We recognize that not all screen readers behave alike, so this is not thorough testing. The goal is to find any obvious screen reader issues.

• On MacOS: Turn on VoiceOver with the 'Command + F5' shortcut.
• On Windows: Turn on Narrator with the 'Ctrl + Windows key + Enter' shortcut.

Here are things to test with the screen reader:

• All content must be presented in text (e.g., alt text for images or other non-text objects).
• All functionality must be available using only the keyboard (Note: there are subtle differences in keyboard behaviors when the screen reader is on).
• Users must receive immediate feedback after all actions (Examples of feedback: Expanded/collapsed region, value changed on a control (e.g., on a slider, successful/unsuccessful form submission, etc.).
• Users must be able to navigate by landmarks and headings.
• When a screen reader user is on a mobile device, swipe actions are used by the screen reading software. All features (controls, widgets) on a mobile web page require a click action to work.
• Link text clearly describes the purpose or destination of a link.
• All form labels are adequately descriptive and instructive.
• All interactions have success and failure feedback available with the screen reader
• The screen reader announces the role of interactive elements (e.g. a text field, a tab panel widget, a tree view, a button, a link, a graphic, a dialog, etc.).