End-to-end Tests are black-box integration tests.

External systems are primarily simulated using Mock Services. The main reason to use Mock Services instead of the test servers of the external systems are:

• Data in test servers of the external systems do no line up. For example a claim id from MAS API development server cannot be found in the UAT server of BIP Claims API.
• VRO team does not have control over test data on external test servers.
• BIP Claims API and BIP Claim Evidence API do not have servers that are accessible outside of VA.
• Interactions are more easily recorded in Mock Services for verification.

Implementation

End-to-end Tests require almost all VRO containers specified in the VRO docker-compose file to be running. Only those marked as prototype can be ignored.

An environment setting script is provided. You must run this script before building the project.

source setenv-e2e-test.sh

Once the environment is set you can build VRO and run the containers as usual.

End-to-end Tests are configured as Integration Tests as described here. After all the containers are run using ./gradlew :app:dockerComposeUp, you can run the end-to-end test task:

./gradlew :app:end2endTest

This will run all the end-to-end tests and report success or provide an html link for errors.

Action

End-to-tests are automatically run when a PR is merged to develop or when a PR is assigned a reviewer. Automation is achieved using a Github action. This PR: End-to-end test for v2 action can also be run manually.

MAS Integration (V2 - iMVP) Tests

All iMVP End-to-end Tests are in the Java file VroV2Tests.java. The tests are implemented as JUnit 5 tests.

End-to-end Tests for iMVP aims to verify most, if not all, of Road Map for iMVP. There are two types of end-to-end tests: positive and negative. Positive tests verify the paths in the Road Map for iMVP when they are arrive at completion as opposed to negative tests that verify error cases where any of the paths are interrupted due to errors.

Positive Tests

Each positive End-to-end Test is driven by a JSON file in the resource directory test-mas. Each JSON file is the mocked payload to the VRO /automatedClaim end-point which is the primary interface to MAS for iMVP.

Critical fields in each JSON file are

• collectionId: the id of the collection for which annotations are retrieved. The collection can either exist in the MAS development server (currently 350 only) or be available from the Mock Mas API.
• veteranIdentifiers.icn: the ICN for the patient. The patient can either exist in the Lighthouse Health API Sandbox server or be available from Mock Lighthouse Health API
• claimDetail.benefitClaimId: the id for the claim. The claim has to be available from Mock Bip Claims API.
• veteranIdentifiers.veteranFileId: the file id for the veteran. This id is used to generate and identify generated PDF's.

The tests calls to /automatedClaim end-point with the payload and verify the response and the expected events. The events that happen are queried using end-points in Mock Services or VRO Claim Metrics API.

The following are the primary verifications

• Response code
• PDF Generation: Mock BIP Claim Evidence API is queried to make sure the evidence PDF is generated when necessary. The generated PDF is downloaded from Mock BIP Claim Evidence API and existence of Veteran name and last name verified using a simple parse.
• Exam Ordering: Mock MAS API is queried to make sure exam is ordered for the collection when needed.
• Slack Messaging: Mock Slack Server is queried to make sure a slack message is sent when needed.
• Claim Updates: Mock BIP Claims API is queried to make sure the claim and contention records are updated when necessary.
• Database Updates: VRO Claim Metrics end-points are used to make sure database is populated as expected.

All positive End-to-end Tests resets the records in Mock Services for the collection under test to enable multiple running of the test during local development.

Negative Tests

Negative tests are for conditions that generate 4xx and 5xx status codes. All the negative tests are in VroV2Tests.java as well. The following are some of the examples:

• Check if VRO is sanitizing the request by filtering disallowed characters such as those that are not printable.
• Check if a non-existent claim id is validated.

API (V1) Tests

All API (V1) End-to-end Tests are in the Java file VroV1Tests.java. The tests are implemented as JUnit 5 tests.

End-to-end Tests for MVP aims to verify most, if not all, of Road Map for MVP. There are two types of end-to-end tests: positive and negative. Positive tests verify the paths in the Road Map for MVP when they are arrive at completion as opposed to negative tests that verify error cases where any of the paths are interrupted due to errors.

Positive Tests

There are two positive End-to-end Tests.

Both tests use existing records from Lighthouse Health API Sandbox server.

The first test is driven by the JSON files in the resource directory test-7101-01 in particular assessment.json which is the expected response from /full-health-data-assessment end-point for a hypertension case. The input payload for /full-health-data-assessment is prepared from assessment.json, the end-point is exercised, and the response is compared to assessment.json. The second step of the test exercises /evidence-pdf. The input payload is formed using data retrieved from the /full-health-data-assessment call and the file veteranInfo.json. The generated PDF file is sanity checked by comparing pdf data to fields in assessment.json and veteranInfo.json.

The second test is similar but verifies responses for an Asthma case. This test case is driven by the JSON file in the resource directory test-6602-01. Otherwise the methodolody is identical to the first test case and in fact both share majority of the test code.

Negative Tests

Negative tests are for conditions that generate 4xx and 5xx status codes. All the negative tests are in VroV1Tests.java as well. The following are some of the examples:

• Check if authorization fails without an API key.