From LHDI kubernetes docs, clusters are:

• ldx-nonprod-1 (DEV, SANDBOX, QA)
• ldx-prod-1 (PROD)
• ldx-preview-1 (PREVIEW)

Associated gateways

• ldx-nonprod (NONPROD)
• ldx-nonprod-1 (NONPROD1)
• ldx-dev (DEV)
• ldx-prod-1 (PROD1)
• ldx-preview-1 (PREVIEW1) (no gateway)
• ldx-mapi-1 (MAPI-1)

DNS:

• dev.lighthouse.va.gov
• sandbox.lighthouse.va.gov
• qa.lighthouse.va.gov
• (PROD) api.lighthouse.va.gov

Output from lightkeeper list team va-abd-rrd:

{
  "name": "va-abd-rrd",
  "id": "...",
  "description": "va-abd-rrd description",
  "namespaces": [
    {
      "name": "va-abd-rrd-prod",
      "cluster": "ldx-prod-1",
      "cpu": 128,
      "ram": "256G",
      "provided": true,
      "imageSigningEnabled": true
    },
    {
      "name": "va-abd-rrd-qa",
      "cluster": "ldx-nonprod-1",
      "cpu": 128,
      "ram": "256G",
      "provided": true,
      "imageSigningEnabled": false
    },
    {
      "name": "va-abd-rrd-sandbox",
      "cluster": "ldx-nonprod-1",
      "cpu": 128,
      "ram": "256G",
      "provided": true,
      "imageSigningEnabled": true
    },
    {
      "name": "va-abd-rrd-dev",
      "cluster": "ldx-nonprod-1",
      "cpu": 128,
      "ram": "256G",
      "provided": true,
      "imageSigningEnabled": false
    }
  ],
  "scm": null,
  "artifact_store": null,
  "pipeline": null,
  "registry": null,
  "reporting": null,
  "observer": null
}

See:

• LHDI's AWS Infrastructure as a Service
• LHDI - Managed Customer Resources

The release process requires access to the VA system. Information on the release process can be found on this page of the internal wiki.

Review current tags and follow Semantic Versioning guidelines to choose the next version number for the release tag. For the desired branch (usually develop), use the create-release-tag GitHub Action workflow in the public repo to update version numbers in the code and create a release tag.

• When the release tag is created, the create-gh-release GitHub Action workflow will automatically create a GitHub Release (these auto-created GitHub releases with suffix -auto are candidates for deletion during cleanup).
• Additionally, the mirror GitHub Action workflow will copy the release tag to the internal repo, where an identical GitHub Release is auto-created in the internal repo, which will trigger SecRel GitHub Action workflow to run and sign the images. The packages/images published to GHCR should be tagged with the release tag version number (v*.*.* or release-*.*.*).
  • The Secure Release process pushes packages to the GitHub Container Repository. Note that the packages are associated with the abd-vro-internal repo.
  • You can follow progress of SecRel (Secure Release) workflow run on the internal repo's Actions.
  • If the SecRel workflow is not successful, the SecRel alerts need to be addressed -- see SecRel Processing Guide.

All GitHub releases that are annotated as a Pre-release (and its associated git tag) will be automatically deleted by the Delete old Pre-Releases and tags GitHub Action workflow. To prevent deletion, remove the Pre-release annotation.

Refer to diagram at Deployment-Overview#process-diagram.

create-release-tag workflow runs the following:

1. ./gradlew release creates a git tag locally (until it is pushed to the remote in the last step)
2. scripts/image-version.sh pin to "Pin unpinned image versions" to the release version number created in the previous step -- see Container Image Versions
3. git push updated image versions to develop
   • triggers mirror workflow to push changes to internal repo
     • triggers SecRel workflow (on internal repo) but any processing is skipped due to a gate-check, which exists to avoid unnecessary processing. SecRel will be triggered in the next step.
4. git push release tag
   • triggers Auto-create GitHub release on public repo
   • triggers mirror workflow to push changes to internal repo
     • triggers Auto-create GitHub release on internal repo)
     • triggers SecRel workflow (on internal repo) on release tag
       • publishes non-dev images (i.e., those without the dev_ prefix) to GHCR -- see CI-CD-Workflows#1-continuous-integration-ci
       • calls SecRel pipeline to scan images

Periodically, the Delete old Pre-Releases and tags runs to declutter pre-releases and git release tags the public repo. These deletions will be automatically propagated to the internal repo when the mirror workflow runs, at which point:

• Git release tags (associated with GitHub pre-releases) are deleted in the internal repo
• These tag deletions will result in GitHub releases disappearing in the internal repo, but the number of releases (shown on the front page) is incorrect (i.e., there are orphaned GH releases) -- submitted bug report to GitHub.
  • 8/24/2023: Bug report closed with:

    Our engineering team has flagged this issue as it appears to be a bug in displaying the releases. There is no current ETA on when the fix will be available, but I recommend you keep an eye on the GitHub Blog and our social media feeds for the latest announcements about new features and fixes. Our policies prevent us from maintaining a ticket in an open state for extended amounts of time. With this I am going to resolve this ticket. I understand that this doesn't resolve the issue reported, however it will likely be some time before any changes are made.

Quoting from this CI/CD page:

1. Continuous integration (CI) is practice that involves developers making small changes and checks to their code. ... this process is automated to ensure that teams can build, test, and package their applications in a reliable and repeatable way.
2. Continuous delivery (CD) is the automated delivery of completed code to environments like testing and development. CD provides an automated and consistent way for code to be delivered to these environments.
3. Continuous deployment is the next step of continuous delivery. Every change that passes the automated tests is automatically placed in production, resulting in many production deployments.

When the develop branch is updated (e.g., due to a PR being merged), it triggers two major GH Action workflows:

• Continuous Integration: runs various tests, including linters, unit tests, end-to-end tests, and CodeQL vulnerability check.
• SecRel: builds and publishes Docker images and has SecRel sign them if the images pass SecRel checks. These images are published to GHCR (GitHub Container Registry) so they can be used in deployments to LHDI's Kubernetes environment.

At this stage, the published Dev images have the prefix dev_ and can be deployed to the dev and qa LHDI environments. Other environments have different requirements, so non-dev images (i.e., those without the dev_ prefix) are used to clearly distinguish them to prevent accidental deletion.

Every afternoon, an automated GH Action workflow called Continuous Delivery updates the deployment in the dev LHDI environment with the latest Dev images published (by the SecRel workflow). There is a small chance that the latest published Dev images may not reflect the latest develop commit since it can take 15 minutes for the images to be built and published.

Q1: Why not deploy with every update to the develop branch? A: If someone is testing in the dev environment, an unexpected deployment would interfere and possibly produce unexpected behavior. Predictable deployments mitigate the likelihood of this scenario.

Q2: Does that mean we have to wait until VRO is automatically deployed to do testing? A: Nope -- see the "Manual deployments" section below.

Q3: Can I deploy and test a specific version of the develop branch? A: Yup! See the "Manual deployments" section below.

Q4: Won't GHCR be filled with Dev images? A: We have a GitHub Action workflow for that! Delete old published DEV images runs twice a month to delete Dev images older than 30 days but keeping at least 10 of the latest versions (regardless of age) for each image. This can also be run manually with different parameters.

To manually deploy to the dev environment, use the Update Deployment workflow to update containers that have changed (the DB, MQ, and Redis containers do not typically need redeployment). Click the "Run workflow" button, then:

• (Leave "Use workflow from" as develop)
• Ensure the dev is the target LHDI environment
• Choose a specific version to deploy or leave it as latest
• Choose which sets of containers to update in the deployment -- the default selections are usually fine. Helm (our deployment tool) will redeploy only images that have changed (usually only the VRO App and domain services) -- this makes deployments efficient. Unselecting an option will undeploy the associated containers -- this is only needed if there is some deployment problem. The following Helm Charts can be enabled:
  • Enable & update VRO Application API
  • Enable & update RRD domain services
  • Enable MQ
  • Enable DB
  • Enable Redis
  • Enable VRO Console
• For "If deploy fails, rollback to previous? (Always true for prod)", leave it unchecked so that if it fails, the failure state will be available for diagnostic investigation. Let VRO DevOps know if a deploy unexpected fails. Since this is a non-production environment, it's okay that VRO is not operational so a rollback is not needed.
• "Shut down deployment completely, then redeploy" can be useful to completely reset and restart everything; however this is not typical, so keep it unselected (false). Use this option only if deployments continue to fail and diagnostics is unfruitful.

Reminders when using this workflow:

• Make sure no one is actively testing VRO in the target environment (i.e., dev).
• Automated deployments will still run every afternoon (as part of Continuous Delivery). So if a manual deploy is done immediately before the automated deploy, the deployment may be overridden with the latest version.

We currently do not automatically deploy changes to the production environment (prod). Rigorous manual QA testing is first conducted in the qa and prod-test environments, with fake and realistic data respectively.

Before describing deployments to the sandbox, prod-test, and prod environments, let's talk about GHCR packages and signed images.

The published images are called "packages" in GHCR, as seen here:

• mirror-* images are copied from external sources. These images are unmodified mirrors of the original and provide services used by VRO. They are added when the Publish 3rd-party image workflow is manually run.
• vro-* images are created from the VRO codebase.
  • These images are typically used for deployments to the sandbox, prod-test, and prod environments.
  • New tagged versions are added as part of the SecRel workflow when run on the main branch or when a release-* is created in the abd-vro-internal repo.
  • New versions of images are tagged using the first 7 characters of the commit hash or the release version (if a release was created).
  • These non-dev images should rarely be deleted.
• dev_vro-* images are the Dev counterparts to the vro-* images.
  • These images are typically used for deployments to the dev and qa LHDI environments
  • New tagged versions are added as part of the SecRel workflow when run on the develop and qa branches.
    • This allows you to address vulnerabilities quickly; from changes to the image or newly discovered vulnerabilities
    • This does not delay you from testing images while secrel is failing, as unsigned images are pushed to dev and qa
  • New versions of images are tagged using the first 7 characters of the commit hash.
  • A dev_ prefix is used to distinguish from images used in prod, so that these Dev images can be readily deleted.

The tags for an image in GHCR can be seen by clicking on the image, then "View and manage all versions". To determine if an image is signed, compare the image "Digest" with the sha256-*.sig tag -- see these instructions.

LHDI environments (clusters) pull images from GHCR. LHDI enforces the following policies for deployment to each environment:

• dev and qa do NOT require signed images
• sandbox, prod-test, and prod require signed images

Images are signed by the SecRel (Secure Release) pipeline, which runs in the the abd-vro-internal repo. VRO's SecRel GitHub Action workflow publishes the images to GHCR and runs the SecRel pipeline on the images, which signs them if the images pass SecRel checks). If SecRel checks fail (for various reasons), the unsigned images are still available for deployment to dev and qa for testing.

Deployments to the sandbox, prod-test, and prod environments are manual and use signed non-Dev images, which are created in 2 ways:

1. Create a GitHub release in the abd-vro-internal repo (Note that a release needs to be created in the abd-vro repo first to prevent the abd-vro-internal's release from being removed due to repo mirroring). The images will be tagged with the release name (e.g., release-1.2.3).
2. Update the main branch. The images will be tagged with first several characters of the git commit hash.

These actions will trigger SecRel to publish and hopefully sign the images. If SecRel fails and cannot be remedied, identify the associated "SecRel scan failed" Slack message and notify @Tom to investigate.

If the SecRel workflow succeeds, run the Update Deployment workflow with the desired image tag to deploy to the desired environment -- see the "Manual deployment to Dev" above.

Windows users could run into some common issues when trying to build or run VRO. If you are a windows user and run into any issues that are related to using windows, please feel free to update this page with the error and your solution.

When trying to run the app on docker, I get this error message: Error creating bean with name 'bipCERestTemplate' defined in class path resource [gov/va/vro/config/BipApiConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.client.RestTemplate]: Factory method 'getHttpsRestTemplate' threw exception; nested exception is [gov.va](http://gov.va/).vro.service.provider.bip.BipException: Failed to create SSL context. Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect

This is because in your abd-vro-dev-secrets repo the file endings are CRLF, when they need to be LF to work. To fix this:

• Open the abd-vro-dev-secrets project into an IDE. I use intelliJ for abd-vro, but for abd-vro-dev-secrets I use VS code to keep them separate.
• In the bottom right corner of the screen, you will see that it says CRLF. Click on this and change it to LF. You will need to repeat this process for every file.
• Make sure you get the files under abd-vro-dev-secrets/local as these are the files causing the issue.
• Make sure to save every file that you made the change on and then open abd-vro
• Now set your env variables using source scripts/setenv.sh
• Run the app and see that the app now starts up as expected.

The code collected under abd-vro serves a several purpose:

1. To act as a parent for all subprojects relating to claim processing

2. To host the Glue code (JPA, REST, and Camel) that connects the individual sub-services

3. To host the CI and deployment scripts

It's multi-purpose nature already makes this repository pretty complex. To simplify its structure, we should aim to concentrate the source code and, especially, the configurations, in a few files as possible.

In my opinion the fact that the project structure is spread over several sub-modules is problematic as it involves several unnecessary features:

• Figure out which module depends on which and try to enforce those dependencies
• Require a build script for each module
• Require spring configuration for every module, especially for testings
• Makes it very hard to run integration tests because spring requires a full configuration for @SpringBootTest

In my opinion, the solution is to simplify the project as much as possible, and this can be accomplished by combining all the sub-modules of "app" into a single module. This module will include the following sub-modules:

• api
• app
• controller
• service (and all sub-modules of service)
• persistence (and all sub-modules of persistence)

There will be one build script and one spring boot configuration (two if you count the tests) for all these modules. That's it.

Whereas it is important that the layering of the architecture should be preserved, the layering does not need to rely on submodules. In the current structure, persistence, services, controllers, etc reside in separate gradle module. It suffices that they reside in different packages within the same module. We can leverage archunit, which we are already using, to enforce the dependencies between layers.

Overall, I don’t think this application is a good candidate for breaking up into even more microservices. When refactoring into microservices, we accept the price of additional complexity in order to gain flexibility advantages. These advantages include the ability to develop parts of the application independently as well as scale different pieces independently. If no such advantage can be gained, then we just pay a complexity price for no reason.

My reasoning for not recommending microservices can be summarized by the following points:

• This application is already an orchestration layer which coordinates the actions of several microservices into a cohesive workflow.
• The same external APIs are used at every step of the workflow: Lighthouse, MAS, BIP, and the python services. This means that there is no particular component which is responsible for communicating exclusively with some type of API.
• Everytime we interface we an external service, the probability that something will go wrong increases. If we inject additional microservices, there will be two levels of indirection before calling, say, the MAS API thus augmenting the probability of error and the difficulty of debugging
• There is actually not that much code in this application.

Nevertheless, if we really want to extract microservices, the key is to identify components that can act virtually independently of one another. I can think of the following components that satisfy this criterion:

• The database/auditing framework can become a downstream service that receives claim updates and persists them in the database and generates alerts as needed
• The v1 and v2 processing streams have their own controllers and their own camel routes. This makes them good candidates for independent services. The downside of separating them is that they share a lot of the same code, so some common libraries will need to be extracted as well.

To refactor the interaction with MAS into self contained microservice, it could include the following

1. Authorization flow support components in the "app folder" (props, config)
2. MAS API service components(auth, service, mapper) in the "service folder" (mas)
3. Abstract the call to get the MAS collection status and annotations into a single call with options to return the response from MAS asis and mapped to a common model(abdevidence)
4. Abstract the call to MAS order exam

Mock MAS API is mainly developed to validate MAS related functionality in automated end-to-end tests.

Mock MAS API is a Spring Boot based REST API.

It is implemented as a Gradle subproject mocks:mock-mas-api in abd-vro project.

The primary functionality of Mock Mas API is to provide mock collections. All mock collections are specified in a resource directory called annotations as json files. A naming convention collection-ddd.json is followed where ddd stands for the collection id.

These collections are modelled after actual collection data receieved from MAS before and during end-to-end testing.

These mock collections are stored in a Java HashMap with collection id's as keys. Spring Boot Application configuration initializes the HashMap as part of initialization of the Collection Store bean.

To facilitate end-to-end tests, this mock server keeps tracks of the collections for which exams are ordered. This information is stored in a Java HashMap with collection id's as keys. The values are Boolean objects. Spring Boot Application configuration initializes the HashMap as part of initialization of the Exam Order Store bean.

End-points are directly defined and implemented in a controller.

Three end points mock MAS functionality:

• POST /token: retrieves a jwt.
• POST /pcQueryCollectionAnnots: retrieves a collection.
• POST /pcCheckCollectionStatus: retrieves status of a collection.
• POST /pcOrderExam: orders an exam.

The remaining end points provides information about the state of the server to facilitate automated end-to-end tests:

• GET /checkExamOrdered/{collectionsId}: checks if an exam ordered for a collection.
• DELETE /checkExamOrdered/{collectionsId}: resets exam ordered status for a collection.

The end-point /token retrieves jwt. This end-point is a pass-through; it simply calls to MAS development server for token and returns the retrieved token.

Note that this mock server itself does not validate the token when it is attached as the Authorization HTTP header in calls to other end-points. This has been left to future development primarily due to other priorities.

The end-point /pcQueryCollectionAnnots retrieves a collection. This end-points accepts a simple json object that specifies the collection's id.

Currently MAS development server provides only one collection id'ed 350. For this collection id, this end-point retrieves the collection from MAS development server and returns the the caller. For all collection id's that identify one of the mock collections, the mock collection is return. If the collection id is not 350 or does not identify one of the mock collections, Not Found (404) status code is returned.

The end-point /pcCheckCollectionStatus currently simply returns VRONOTIFIED status. Since MAS is now preparing the collection before calling to VRO, we do not expect any other status.

The end-point /pcOrderExam mocks ordering an exam. This end-point currently always returns success. In addition the collection id for which the exam is ordered for is stored in the Exam Ordered Store.

The end-point GET /checkExamOrdered returns if an exam is ordered for a collection. This information is useful in end-to-end tests. A DELETE end point is also provided so that the mock server can be reset for a particular collection id. This mackes it possible to run end-to-end multiple times after this mock api is starred.

This mock API uses the same MAS related environment variables that VRO application uses. You can find these environment variables in the application.yml mock-mas properties.

VRO Database is primarily used for audit purposes. It does not store any Personal Identifiable Information (PII) or Personal Health Information (PHI). The following information is available

• Claim ID (VBMS ID), disability action type, presumptive and RFD (Ready for Decision) flags
• Claim Submission data associated with the attempt to process each Claim Submission Request through the workflow
  • any external vendor ID (Reference ID — used by external vendors to track the claim in their systems)
  • along with the associated type (representing the specific vendor associated with the Reference ID)
• Non PII Ids for the Veteran who owns the claim
• Contentions in the claim
• Exam Order state as issued by VRO and updated from MAS
• Non PHI summary information for the assessment for each of the contention
• Non PHI summary information for each evidence PDF generated for the contention
• Various event audit logs as a claim goes through VRO system
• Database version information to support future changes through database migrations

All tables and fields are available in the Entity Relationship Diagram (ERD). This version of ERD is manually generated using DBeaver on 3/12/2023.

Note that the database is designed for multi issue claims although iMVP will not support such claims.

VRO uses PostgreSQL. Since RDS is currently not available in LHDI, a PostgresSQL Docker based container serves as the database. The subproject postgres is used to build the container.

The Dockerfile in the postgres subproject is a simple wrapper around the Docker Hub PostgresSQL image. The primary reason to use the Dockerfile instead of using the Docker Hub image directly is to run the initialization script init_db.sh. This script creates the application database and a database user to run database migrations that are different from the default database postgres and the super user for security purposes.

The version of the database can be found in the subproject Dockerfile.

VRO achieves persistance through persistent Docker Volumes. The setup the volumes can be found in the application docker-compose file for local development and in the Helm's chart templates for deployment.

VRO uses Flyway to initialize and change the database. The subproject db-init containts all database migration related artifacts. In particular all versioned migrations are SQL based and in the directory here. These migrations create all the schemas and tables. In addition an additional user with limited privileges is created. This user is used to access the database within all non migration VRO functionality.

The subproject db-init also contains a Dockerfile. The container based on this Dockerfile is used to run the migrations both in the local development docker-compose file and in application deployment.

The strategy for creating migration files is simple: Each work branch and subsequent Pull Request should be its own contained version number. Furthermore, you should create one migration file per proposed table change. Smaller, incremental changes on a per-file basis allows for better maintainability, troubleshooting, and testability.

VRO uses Spring Data JPA to access and manage data in the database from the Java code. The subproject persistance/model contains the Object-relational mapping (ORM). In particular entity files that map to the database tables are in here.

To access and manage entity data , VRO uses JPA Repositories. All the JPA repositories are in here. These JPA Repositories inherit basic CRUD methods and also contains explicit definition of more complex methods needed by the VRO applications. In either case implementation of the methods are provided by Spring Data JPA automatically.

VRO defines 3 service interfaces (Java Interfaces) in the service/spi subproject to populate and access the database. These interfaces are

• Save to Db service: This service is used within the Camel routes to store information about claims, assessments, and evidence pdf generations.
• Claim Metrics service: This service serves as the service later for the REST interface that exposes information in the database.
• Audit Event service: This is also used within the Camel routes and logs various information about events that occur as claims are processed and external systems are called.

These services are implemented (Java Implementations of Interfaces) in the service/db subproject as Spring Boot services and are autowired in the rest of the projects. Implementations use JPA Repositories in the subproject persistance/model.

The Save to Db service and Audit Event service are primarily used in Camel routes which are defined in the service/provider.

The Claim Metrics service is used in the implementation of claim-metrics and claim-info REST end points. These end points are defined in Claim Metrics resource and implemented in Claim Metrics controller.

The Entity Relationship Diagram (ERD) shows all the tables and fields used. All the tables reside in the claims schema. These are the tables in the database

• claim: This table stores audit information about the incoming Claims themselves (as unique entities). The corresponding entity class is ClaimEntity.
• claim_submission: This table stores audit information concerning each attempt to submit and process a Claim Submission through VRO. The corresponding entity class is ClaimSubmissionEntity.
• veteran: This table stores audit information about the veterans in the claims. The corresponding entity class is VeteranEntity.
• contention: This table stores audit information about the contentions in the claims. The corresponding entity class is ContentionEntity.
• exam_order: This table stores audit information about the contentions in the Exam Order status. The corresponding entity class is ExamOrderEntity.
• assessment_result: This table stores audit information about the assessmen results for the claims. The corresponding entity class is AssessmentResultEntity.
• evidence_summary_document: This table stores audit information about the evidence documents created for the claims. The corresponding entity class is EvidenceSummaryDocumentEntity.
• audit_event: This table stores log information as claim processing progresses through the Camel routes. The corresponding entity class is AuditEventEntity.
• schema_history: This table stores migration version information and is used by Flyway database migrations. Database migration are described in Database Migrations

The tables claim, claim_submission, veteran, contention, exam_order, assessment_result and evidence_summary_document all have created_at and updated_at columns. These columns are inherited by the corresponding Entities from a BaseEntity. BaseEntity uses Spring Data JPA @CreatedAt and @LastModifiedDate annotations to implement the functionality. With these annotations Spring Data JPA automatically populates the fields without additional code in VRO.

For the same tables id column implementation is also shared in BaseEntity.

The following are the column descriptions for the claim table.

• vbms_id: This represents the VBMS system ID for the Claim. This is not intended as the source of truth and should be paired with submission_date to determine when the last known valid associated timestamp was.
• veteran_id: Foreign key to the veteran table and identifies the Veteran the claim belongs to.
• disability_action_type: i.e. INCREASE
• presumptive_flag: Represents the Claim's presumptive status for fast-tracking
• rfd_flag: Represents a Ready For Decision (RFD) state

The following are the column descriptions for the claim_submission table.

• claim_id: Foreign-key to the corresponding ID in the claim table.
• reference_id: Represents an external vendor's internal system ID for tracking the Claim. Used in addition to id_type to identify the source of the claim.
• id_type: Represents the external source, or vendor, of the Claim. Used in addition to reference_id to identify the source of the claim. This was the constant va.gov-Form526Submission for V1. For V2 this constant is 'mas-Form526Submission'
• incoming_status: Status of the incoming claim. This was the constant submission for V1.
• submission_source: Taken from the claimSubmission.claimDetails.claimSubmissionDateTime initially sent by MAS
• submission_date: Taken from the claimSubmission.claimDetails.claimSubmissionSource initially sent by MAS
• off_ramp_reason: Explanation for why the claim was off-ramped.
• in_scope: Boolean flag representing the claim is in scope of being processed. Set by VRO. Defined in #428 but potential duplicate of off_ramp_reason=outOfScope — we might not need this anymore

The following are the column descriptions for the veteran table.

• icn: The Internal Control Number (ICN) for the Veteran. It uniquely identifies the Veteran in VHA systems.
• participant_id: The Participant Id for the Veteran. It uniquely identifies the Veteran in VBA systems and is actually the database ID in the CorpDb.
• icn_timestamp: Since it is possible for ICNs to change, you can tell when the ICN was last updated with this timestamp. In theory, you could also use updated_at, but that column could possibly apply to other pieces of data here, so icn_timestamp provides a targeted "last known good time".

The following are the column descriptions for the contention table.

• claim_id: Foreign key that links the contention to the claim it is submitted with.
• diagnostic_code: The diagnostic code for the contention. It links the contention to the VASRD codes.
• condition_name: Name of the condition to be assessed
• classification_code: Taken from the claimSubmission.claimDetails.classificationCode initially sent by MAS

Note that this design assumes multiple contentions per claim for future developments although iMVP will support only one contention (Hypertension) in single issue claims.

The following are the column descriptions for the exam_order table.

• claim_submission_id: Foreign key that links the Exam Order to the Claim Submission that issued it.
• collection_id: Collection ID associated with the Claim Submission.
• status: The current status of the Exam Order
  • ORDER_SUBMITTED: initial status from VRO, which creates a record when it issues a new Exam Order
  • VRO_NOTIFIED: TODO: Seek clarification on if this will continue to remain the initial submitted status follow-up from IBM/MAS
  • DRAFT: This was the initially assumed initial submitted status follow-up from IBM/MAS indicating the Exam Order is in Draft State
  • FINAL: status from IBM/MAS indicating the Exam Order has been completed
• ordered_at: TODO: Change to status_at to represent the various statuses

A Claim Submission can have multiple Exam Orders.

The following are the column descriptions for the assessment_result table.

• contention_id: Foreign key that links assessment result to contention.
• evidence_count_summary: Summary of evidence counts for the assessments. This is a JSON objects that summarizes assessment and is provided by the assessment microservice.
• sufficient_evidence_flag: Originally from #447. Represents that the Assessment Result has determined there is sufficient evidence to mark the claim as RFD.

The following are the column descriptions for the evidence_summary_document table.

• contention_id: Foreign key that links evidence summary document to contention.
• evidence_count: Summary of evidence counts for the document. This is a JSON objects that summarizes the information shown in the document.
• document_name: Name of the document generated.
• folder_id: Represents the UUID of the folderId returned by BIP on PDF upload, in order to facilitate easier tracking down of the file in eFolder.

This is the table structure for audit events:

• event_id: A unique id identifying the request
• route_id: The id of the camel route from which the event is issued. Example: "mas-order-exam"
• payload_type: The type of payload being processed. Example: "Automated Claim"
• details: Other details pertinent to the event, but specific to the type of processing. For Example, collectionId, offRampReason, presumptiveFlag, et cetera.
• message: A descriptive message explaining the action. Example: "Collecting evidence"
• throwable: The stacktrace of an exception, if the even indicates an error.
• event_time: Date and time the event was issued.

Simplified ER (detailed ERD can be found here: Entity Relationship Diagram (ERD)):

VRO is a workflow system that uses microservices to assess the claims. It connects to MAS, BIP, and LH API. MAS starts the process by sending an eligible claim, BIP and LH API are used gather evidence/order exams, the process finishes with a PDF submitted to BIP or a claim off-ramped.

The VRO Console can query Redis and DB to determine the state of the claim

• When the APIs that VRO depends on (BIP, LH API, BGS, MAS) are down, or slow, the workflow will retry multiple times, then go into a fail state and the claim is 'stuck'
• When a microservice used in the work flow fails or times out the claim is 'stuck'
• When a notice is posted that an API or Microservice will not be available, during the time they are not available, claims will be 'stuck'
• VRO does not receive confirmation to exam order status API within 24 hours of processing the claim
• BIP Claim Evidence API is down and an exam order has been requested via the MAS API
• VRO cannot successfully retrieve data from Lighthouse Patient Health API

• Slack #benfits-vro-alerts contains "Exception occurred", This shows that a claim is 'stuck' and VRO cannot complete the workflow.
• Slack notification message: "PDF upload failed after exam order requested"
• Slack notification message: "Lighthouse health data not retrieved"

Post on #benefits-vro. Use this template:

VRO CLAIM STUCK:
Claim ID: 123 was not properly processed, @your-name investigating. 

Post situation on #lighthouse-delivery-platform-support Notify MAS to stop sending eligible claims. Post to #rating_automation_collaboration. Use this template:

VRO CLAIM STUCK:
VRO is not currently processing claim. Please suspend sending claims until further notice. 

In order to escalate the issue to BIP use the following contacts: email: [email protected] [email protected] is the Manger in charge for further escalation. Work phone: 650-512-2936

For BGS service failure: You can reach out to [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected] For Production we should also open a SNOW or service now ticket. (Steps TBD)

Slack #benfits-vro-alerts shows an AuditEvent for each blocked claim with no exceptions

There is a special issue flag (RDR1 flag) in the VBMS database (or CorpDB to be more specific) for each claim. As the claim is moving forward the flag gets added and after finishing each step, the flag for that step is removed indicating that the step was executed successfully. If a claim is stuck, this flag will have the indicator showing the step that was failing in.

If the BIP, LH API, BGS, or MAS APIs are not working properly, a notification should be posted on the #lighthouse-delivery-platform-support channel using the provided template. The recommended action is to verify the APIs are working properly, and if not, escalate the issue to the MAS team for further investigation.

Overall, the recovery process involves identifying the root cause of the issue, resolving it, and notifying relevant teams and channels about the status of the VRO workflow.

The following steps can help to recognize where claim was stuck and resolve it to manual process:

• Follow instructions for VRO Console, results should follow examples in wiki

  • Inspect DB
  • Inspect Redis
  • Inspect Camel routes

• Verify that API are working properly

  • BIP is working if...
  • if not post to Slack #lighthouse-delivery-platform-support
  • LH API is working if status is good
    • if not working
    • post to Slack #lighthouse-delivery-platform-support
  • BGS is working if...
  • if not working
  • post to Slack #lighthouse-delivery-platform-support
  • MAS is working if...
  • if not working
  • post to Slack #lighthouse-delivery-platform-support

• Verify that microservices are working properly

  • If microservices show exceptions in the log that they are unable to handle a claim

    • rollback to previous version of VRO

  • If data dog shows a microservice is down

    • restart the microservice

• If claims are stuck and VRO is not able to process additional claims

  • Notify MAS team to suspend the flow of claims to VRO
  • Post situation on #rating_automation_colaboration with this template

VRO CLAIM STUCK:
VRO is not currently processing claims. Please suspend sending claims until further notice. 

Further more, there is a ticket for development of offramp process that will remove the flag and the claim from the system to offramp that claim to manual process.

https://github.com/department-of-veterans-affairs/abd-vro/issues/1312

What are the known/likely causes of this state

Who/how to notify that system is in this state

What to do if the recovery process does not restore the system

What indicates that the system is back to good state

What indicates that system is in this state

Steps to recover...

TLDR:

• We’ll migrate/port specific functionalities currently implemented in va.gov’s platform to be microservices running in LHDI’s platform.
• We’ll use LHDI’s Java Starter Kit and Apache Camel (a Java library that facilitates gluing together components in various workflows) to expose REST endpoints and route requests to microservices written in any language.
• Workflows (such as those running on va.gov — see others in the “Claim submissions trigger VRO” section) will call VRO endpoints deployed on LHDI.

Features of VRO: (more may be added in the future)

• Assess Veteran’s health data from various sources
• Order an exam when additional health data is needed
• Establish a claim in VBMS/BGS
• Set special issue on the claim to fast-track it
• Generate and upload PDF for fast-tracking the claim
• Associate supplemental PDF documents to the claim to help fast-track it

Software Design for VRO: revolves around QP components and leverages EIPs

VRO implements an event-driven architecture.

• QP (Queue-Processor(s)) component acts like an internal microservice that modularizes functionalities so that it can be updated and maintained more easily
  • Consists of one Queue, which has a globally unique name and simply holds items. Items can be added to the queue by anything. Queue’s contents should be persisted to restore VRO state in case of system failure.
  • A Processor processes items from the Queue. They are stateless and preferably idempotent. A Processor can be implemented in practically any language (Java, Ruby, Python, etc.), as long as it can interface with a Message Queue (such as RabbitMQ or Amazon SQS). For scalability, a Processor can be replicated to process items from the Queue in parallel — shown as “instance 1” and “instance 2” in the diagram.

• QP components will be connected together using well-tested and stable Enterprise Integration Patterns (EIP) tools (such as Apache Camel) so that we can focus on VRO functionality and less on “glue code”.

  • As the software becomes more complex, using the same QP pattern for all VRO functionalities promotes low software coupling) and, as a result, simplifies debugging and maintenance.
  • Using EIPs facilitate future migration of parts of the architecture to cloud-based services if needed (see Implementing EIP with AWS and with Azure), where for example a Processor could be implemented as AWS Lambda. This is mentioned to emphasize the flexibility of the architecture to evolve in case we decide to implement some subset of the QP components in those cloud services.
  • Using the Apache Camel implementation of EIPs provides time-saving features and integrations with many existing tools and data formats.
  • Example workflow using QP components:

• The Router QP responds to VRO API requests and determines how the claim should be processed

  • For each new claim, it quickly validates the claim and Veteran to send a response back to the API caller.
  • Based on the claim characteristics and any other data it gathers, it determines how the claim should proceed and adds the claim to an appropriate Queue for processing.

• Each hypertension/asthma/apnea QP components is expected to:

  1. receive an item from its Queue (either pushed or pulled)
  2. gathers health data by querying data sources (e.g., Lighthouse)
  3. decides on the claim (see details in the “VRO processing” section)
  4. adds a new item to the QA Queue

• A Manual QP component can be included for discovery and research of new types of claims to fast track, or for scenarios where a claim has curious characteristics that require manual intervention.

• The QA Processor performs quality assurance; to validate the VRO processing output and prep it for downstream processing (external to VRO). (Not shown in the diagram: another Manual QP component can be added for cases where the QA Processor finds an unsupported problem with the results.)

• If requested as part of the original VRO API request, the Notifier Processor will execute any requested callbacks to indicate completion of VRO processing on the claim.

External Interactions:

• vets-api’s DB to get claimant’s info and Veteran’s ICN (needed to query Lighthouse)

• Lighthouse’s Veterans Health API to query for medical data

• AWS S3 to temporarily store documents for uploading to eFolder

• EVSS to submit claim (including setting special issue and uploading PDF to eFolder)

  • Lighthouse is replacing EVSS interactions with a new API

• Central Mail Portal’s API to extract structured data from unstructured eFolder docs

  • (future) BAM’s Evidence DB for health data extracted from unstructured eFolder docs

• Central Mail Portal’s API to order an exam

• eMIS (an API for the VADIR database) to query for military service history

  Example external interactions for the hypertension QP component:

Our VRO’s software design will be deployed to the Lighthouse DI (Delivery Infrastructure, a.k.a. LHDI) platform:

• They provide a Java-based LHDI Starter Kit - Java "to build a service [on their platform] and deploy it into production hassle free" (see out-of-the-box features).

  to quickly create new services for the VA without having to spend any extra cycles on tasks that are common to all development efforts. It provides lift to the development process, such that developers do not need to build their services from scratch.

• We expect to see Starter Kits for non-Java languages in their list of repos.

• LHDI tools: AWS EKS, AWS Parameter Store, Istio Service Mesh and Kiali, Jaeger tracing, Helm and Argo continuous deployment server, DataDog, Prometheus and Grafana monitoring, GitHub Actions, CodeClimate, OpenAPI/Swagger, Spring Boot, Flyway DB migrations, Java and Gradle

• The Lighthouse Platform provides devops and scalability support, so that the VRO team can focus on the VRO software.

• Lighthouse Infrastructure Topology, code repositories, and more info here.

Our Plan: Using LHDI’s Starter Kit, migrate our code to use the Queue-Processor design pattern and deploy to the LHDI platform.

Tech Stack:

• LHDI Platform (provides Kubernetes, monitoring, and devops support)
• LHDI Starter Kit - Java (creates Docker container and provides JVM for Apache Camel)
• Message Queue and Apache Camel (provides VRO’s routing pathways among QP components)
• QP components (where Processors implement VRO functionalities in Java, Groovy, Ruby, Python, etc.)

Incremental Migration to Lighthouse DI:

• Build VRO skeleton code in Lighthouse DI with one hello-world VRO API endpoint exposed via Lighthouse VRO API (This tests plumbing, network connections, etc.)
• Incrementally build out VRO in Lighthouse DI (extract/duplicate certain functionalities in the current vets-api VRO) and expose them as Lighthouse VRO API endpoints
• Incrementally change vets-api VRO to call Lighthouse VRO API endpoints
• Gradually have other applications (such as those in Central Mail) call Lighthouse VRO API endpoints

Single vs multiple repos: Initially, the VRO source code will reside in a single GitHub repository since each QP processor will be a small chunk of code. If the code becomes large and complex, we’ll have the option of extracting pieces of it into their own repositories with minimal effort due to the low coupling of the architecture.

Shared code: Within a single code repository, any common code (such as clients to query external APIs) among the processors can be easily shared. Common client classes or libraries can be created for reuse. As the software grows in size and complexity (due to incorporating more contentions and/or more external data sources), we can create QP components whose sole purpose is to act as shared services that other QP components use, for example to request data from external systems. For either scenario, the source code to request external data would be updated in one location to encourage reuse and reduce maintenance.

Collaboration: VRO source code will be publicly available by default. Any sensitive code can be extracted into a separate private repository. Other teams can contribute to the VRO code base or they can have separate code repositories to build their own QP components to be included in the VRO architecture.

Deployment: For the Lighthouse DI platform, the QP components would be deployed together as a set of Docker containers. If we wanted to deploy QP components separately, we’d have to work with Lighthouse DI to enable deployment of individual Docker containers. If components are implemented in AWS Lambdas (not currently planned), then they could be deployed separately.

The following 3 sections describe VRO input, processing, and output/artifacts.

Expose an VRO API (via Lighthouse DI) (maybe as one of Lighthouse APIs) so that an VRO API endpoint can be called from VA.gov or CMP (Central Mail Portal) for example. Following are examples of paths for claim submissions and how they trigger the VRO process (refer to “Example workflow using QP components” diagram in prior section):

Path 1: From VA.gov to VRO

• VA.gov (vets-api) receives claim submission and calls the VRO API with claimant info and claim submission data
• VRO creates claim and quickly responds back to VA.gov with a “Success” after claim establishment

Path 2: From CMP (Central Mail Portal) to VRO

• CMP receives claim submission, optionally establishing the claim, and calls the VRO API with claim submission data, along with other relevant data from CMP
• VRO creates a claim (if needed) and quickly responds back with a “Success”

Path 3: From a “New claim” VBMS Poller to VRO

• Poller queries VBMS for new claims and calls the VRO API with claim data
• VRO quickly responds back with a “Success” if there is enough info to initiate the VRO process

Path 4: From a “New claim” Subscriber to VRO

• When the Subscriber get notified about a new claim, it calls the VRO API with claim data
• VRO quickly responds back with a “Success” if there is enough info to initiate the VRO process

For all paths, after VRO responds with “Success”, VRO continues its process to collect medical data and if possible fast-track the claim — details in next sections.

For a claim submission, VRO collects relevant data and assesses the data for fast-tracking.

1. If the claim is not eligible for fast tracking (or not yet supported by VRO), then skip VRO processing, else continue.
2. Collect Veteran’s health and military service history data
   • via Lighthouse APIs
   • from OCR+NLP of docs (via MA’s processing of eFolder docs)
   • from BAM’s Evidence DB
   • from VADIR via eMIS
3. Assess Veteran’s health and military service history data (based on disability rating rules)
4. If the claim can be fast-tracked,
   1. Create claim in VBMS/BGS if it doesn’t already exist
   2. Generate and upload PDF for fast-tracking claim
   3. Set RRD special issue on the claim for NWQ fast-track routing
5. Else order an exam if appropriate

VRO Database - not necessary (given Queue contents are persisted by the Message Queue service) but having an VRO DB offers some benefits:

• Track claim status through VRO’s process
• Log and replay requests
• Produce statistics and dashboards; enables data analyses

The expected results of VRO processing include:

• Claim record in VBMS/BGS
• VRO-generated PDF in VMBS eFolder
• RRD special issue for NWQ to route the claim
• Exam ordered

If a notification is warranted (to inform of updates to the claim or RRD-processing status of the claim), the Notifier QP will call other APIs as needed.

PR #302 Remove use of CircleCI

Followed LHDI instructions and used lighthouse-di-circleci-java17-image. CircleCI was enabled in PR #16.

The configuration runs similar but not all operations as Github Actions.

• config.yml uses $GITHUB_USERNAME and $GITHUB_ACCESS_TOKEN (to pull a Docker image), which are set in CircleCI's project settings (This needs to be done in the internal repo as well to get CircleCI running there.)

CircleCI does not push container images ("packages") to the Github Container Registry. We don't want to pollute it with development packages. See Docker-containers#Packages.

Commit c7b786c limits CircleCI runs for only the main and develop branches to reduce the time for PR checks.

CircleCI runs

The PDF Generator contains all the different templates used to generate documents by providing the appropriate data in the generation requests. The following libraries are available for rendering:

• WKHTMLTOPDF: https://wkhtmltopdf.org/
• WeasyPrint: https://weasyprint.org/

The PDF Generator routes allow you to specify which library you wish to use and if it is not provided in the request, it will default to WKHTMLTOPDF

On pdfgenerator startup, the consumer will attempt to create a generate-pdf, fetch-pdf, and generate-fetch-pdf queue on a pdf-generator exchange.

• Request PDF Generation - POST /evidence-pdf
• Fetch Generated PDF - GET /evidence-pdf/{claimSubmissionID}

Both functions work off the same endpoint. The main difference being that the POST has a JSON request body while the GET uses the URL(claimSubmissionID) to find the corresponding PDF.

• Immediate PDF - POST /immediate-pdf

This endpoint is a combined version of the generate-pdf and fetch-pdf. Pass it the POST request body and it will respond with the PDF associated with the specified claimSubmissionID

Any messages passed to the generate-pdf queue will use the diagnosticCode, pdfTemplate and pdfLibrary(optional) to select the appropriate template and generate the PDF. diagnosticCode is used to translate this into a human readable diagnostic type based on the mapping in config/settings.py in the codes variable.

Using pdfTemplate like v1 along with a diagnostic type like hypertension for example, it will pull up the appropriate template and template variables in the templates and template_variables folder respectively. For example, "diagnosticCode"=7101 and "pdfTemplate"="v1"will fetch templates/hypertension-v1.html and template_variables/hypertension-v1.json.

The generator will first load the hypertension-v1.json file which is prefilled with default values for the available variables within the template and attempt to replace them with what is provided in the message. If it cannot replace the data based on what's provided in the request, it will keep the default that is defined in the JSON file.

After the HTML template is generated with the replaced variables, it uses WKHTMLTOPDF by default or the library selected by pdfLibrary to create a PDF based on the HTML file.

Once the PDF has been generated, the consumer will create a key value pair in Redis to store the data due to it containing PII information. The key being claimSubmissionId while the value is a base64 encoded string representation of the PDF.

The consumer will return a response similar to:

{
  "claimSubmissionId: "1",
  "status": "COMPLETE"
}

When the consumer receives a message in the fetch-pdf queue, it will use the provided claimSubmissionId to look it up on Redis.

If the PDF still hasn't been generated, you will receive a response similar to:

{
  "claimSubmissionId: "1",
  "status": "IN_PROGRESS"
}

but if the PDF is available then the response will be a downloadable file

ToCs are not part of the normal HTML template that gets generated for the PDF. They need to be created through a different process and merged with the main PD template

The PDF generator will check if there is a ToC file already created for the diagnosticCode that gets passed. If not found, it will generate the PDF without a ToC so you don't have to worry about having a empty section or page

1. Create a directory in templates where the name will be the human readable diagnosis name used in the codes variable in settings.py

2. Within this folder, create a toc.xsl file. Most ToCs will follow the same format so you can just copy one from any other diagnosis if available. If you needed to create one from scratch, in the command line run the following: wkhtmltopdf --dump-default-toc-xsl and copy the contents of the output to a new toc.xsl file as stated above

3. By default a ToC is generated by finding all <h?> related tags(<h1>, <h2>, etc) so you need to modify them if you want them ignored.

   1. To ignore a heading, it must start with ‌ like this example: <h3 class="text-center">&zwnj;Test PDF Heading</h3>. The toc.xsl file has logic in place to skip over any headings that start with this special character. This character was used since it's an invisible character so it won't render on the PDF

4. The ToC page is fully customizable just like any HTML page

1. The library was built using Webkit 2.2 (~2012) and QT4 (2015) so many newer HTML features are unavailable or need to be added through other ways for them to render properly
2. This library renders at 96DPI but the value can be altered through the meta tags. We need to verify that the Figma design or other design software matches the proper DPI settings by making sure the resolution matches the paper size. Use the following links for proper conversions: https://a-size.com/legal-size-paper/ and https://www.papersizes.org/a-sizes-in-pixels.htm as well as https://pixelsconverter.com/inches-to-pixels

Work in Progress

1. This library does not accept Javascript. At the moment, we would need to come up with a workaround by prerendering in a secondary library or just using WKHTMLTOPDF for Javascript specific portions but this solution has yet to be implemented.
2. This library renders at 96DPI and the value cannot be changed. We need to verify that the Figma design or other design software matches the proper DPI settings by making sure the resolution matches the paper size. Use the following links for proper conversions: https://a-size.com/legal-size-paper/ and https://www.papersizes.org/a-sizes-in-pixels.htm as well as https://pixelsconverter.com/inches-to-pixels

Before you can start working on the content of the PDF design, you must first match your DPI to the size dimensions of the design tool. If you skip this step, then the measurements will be all wrong and the design won't be a perfect match. This in turn will cause the developer to modify/test random values or mess with the zoom setting to get it to fit the design.

For example, if a user wants to make a new PDF, they must:

1. Get the size/dimension that the design team wants to use. Not just whether its A4, Legal, etc. but the pixel dimensions of the blank design page. We will use this number to set or match the DPI accordingly
2. Setup the new template to match based on the library:

• WKHTMLTOPDF: DPI is customizable so you can use the following for proper conversions:
  • Legal and other sizes like A4, etc.: See what DPI setting the pixel dimensions fall under and set the meta tag to that DPI once you make the template file in the next step
• WeasyPrint: The DPI value is set to 96 and cannot be changed. Due to this, the process is somewhat backwards. The developer needs to use the above links to get the dimensions based on the requested page size and 96 DPI and send the dimensions back to the design team so they can adjust their document to match.

1. Edit the codes dictionary in config/settings.py by adding a new key, value pair for your code.
   • Example:

   codes = {
     "0000": "cancer",
     "0001": "diabetes" //new code with human readable diagnosis
   }
   

2. Create a HTML version of the PDF you want to generate and save it in the templates folder along with a version number
   • Take a look at Jinja 2 Template Documentation for a better idea of what you can do within the template files
     • Every template file needs a version number. By default, the system looks for v1 if one is not specified in the request
     • The file name should match the name you used in Step 1. Following that example, it should be called diabetes-v1.html
3. Create a JSON file in template_variables that will contain default values for the HTML file in case they are not provided in the RabbitMQ message
   • The file name should match the name you used in Step 1 and 2. Following that example, it should be called diabetes-v1.json

Some diagnostic codes might need specific changes that don't need to affect other templates and instead of adding it to the assessment logic, we can use a helper function.

When generating a PDF, it will look for a helper function following this naming convention pdf_helper_0000 where 0000 would be the code we want to use. If it does not find it, it move on and then applies a pdf_helper_all that gets applied to every single template. Usually these are edits like turning date string into proper date time objects, etc that would benefit all the templates.

Currently there are 2 ways to develop/test the PDF service:

1. Run ./gradlew build check docker to build all containers and run a full test. This can be used for the testing any updates that are made to the endpoints through Swagger but it takes longer due to having to load all the containers. After the containers are built, you can take it a step further and run the containers themselves using ./gradlew app:dockerComposeDown app:dcPrune app:dockerComposeUp and then heading to the Swagger page to view and run the available endpoints.
2. Run python pdfgenerator/src/lib/local_pdf_test.py from the service-python directory. This file calls the PDF generator while bypassing all the related RabbitMQ and Redis code. You can alter the diagnosis_name and message to simulate an endpoint request and to quickly debug any template or PDF issues. The diagnosis_name should be the full name including version number like hypertension-v1

VRO v2 is using Apache Camel to coordinate several services into the workflow described by this document.

Apache Camel provides a lot of message-oriented abstractions but it is using it is own DSL (Domain Specific Language) which can be difficult to master. Familiarity with Apache Camel is a prerequisite for understanding the code. A brief overview of Apache Camel can be found here.

The MasController implements the two endpoints:

• v2/examOrderingStatus: This endpoint does not do anything except record the fact that it has been called.
• v2/automatedClaim: This endpoint kicks off the functionality for automated claim processing.

The rest of the document will be referring the diagram below (from the VRO v2 Roadmap) and is dedicated to explaining the implementation of the pictured workflow.

The endpoint v2/automatedClaim is handled by MasController which immediately hands off to MasProcessingService.

MasProcessingService is responsible for implementing the logic contained in the topmost yellow box of the diagram, where several checks are performed to verify if the claim qualifies for automated processing. If any of these checks fails the API returns a response explaining the reason. This is the only part of the code that executes synchronously. From that point on, a message is sent and the process continues asynchronously.

The following response indicates that the claim is not in scope:

{
  "id": "c9fda1ac-2422-47e1-aeff-6c0a2b08d6df",
  "message": "Claim with [collection id = 351], [diagnostic code = 7101], and [disability action type = FD] is not in scope."
}

The following response indicates that one of the anchor tests failed:

{
  "id": "b34bc26a-68b1-4c08-bca0-ce28db9c4c98",
  "message": "Claim with [collection id = 351] does not qualify for automated processing because it is missing anchors."
}

The following response indicates that the claim passed the initial checks and is undergoing processing, which can potentially take a long time.

{
  "id": "a14fd4e5-abe0-48b7-95df-3f9c85164a3a",
  "message": "Received Claim for collection Id 350."
}

The definitions of all the relevant Camel routes are in MasIntegrationRoutes. This class implements RouteBuilder which provides access to the constructs of the Camel DSL. The configure() method is the entry point and it is divided into several logical groupings of routes:

• configureAuditing: Sets up routes for audit messages. Audit messages end up in the database and in some cases appear as Slack notifications. A full description of the Auditing framework is given in a different section of this document. This method also implements exception handling.

• configureOffRamp: This handles any claims that are off-ramped (rightmost path on the diagram) and forwards them to the "Complete Processing" step.

• configureAutomatedClaim: Integration with MAS requires that we send a request for a collection ID and then keep poling periodically for status. This is implemented by storing a message in a queue with a specific delay. Once the delay elapses, the message triggers a query for status via the MAS API. This logic is implemented in "MasPollingProcessor". If the collection ID is not ready, the message is requeued with a delay. When the collection is ready, the message is forwarded to the internal mas-processing endpoint.

• configureMasProcessing: This route orchestrates the entire workflow by delegating the processing steps to other routes. It performs the following steps:

  • Calls the collect-evidence endpoint to collect evidence from different sources
  • Calls Health Assessment to assess the evidence (second yellow box in the diagram)
  • Conditionally calls "order exam" (third yellow box)
  • Generates and uploads the final PDF file (fourth yellow box)

• configureCollectEvidence: Collect evidence from the MAS and BIP APIs. Merge the two evidence objects and call Health Assessment via RabbitMQ.

• configureUploadPDF: This route corresponds to the penultimate yellow box in the diagram. It calls the service to generate PDF via Rabbit and the calls a BIP endpoint to upload the PDF>

• configureCompleteProcessing: This is the last processing step, corresponding to the last yellow box in the diagram. Both the off-ramp claims and the ones that have been processed converge on this route. Special Issue is removed via the BIP API. If there is sufficient evidence for fast-tracking, the claim is marked as RFD also via the BIP API.

• configureOrderExamStatus: This endpoint simply records an audit event to record the fact that the REST endpoint v2/examOrderingStatus has been called.

Auditing for VRO v2 is more generic than auditing for VRO v1. Instead of having a table structure mirroring claim requests, we have a generic event data model that can contain information about any type of event.

An audit trail is created for each API request. When an entry point is invoking a UUID is created to track the request. This ID can be used to identify all events connected to the request.

Since the workflow interacts with several services, it is unavoidable that exceptional conditions will occur from time to time. Exception handling is configured as part of auditing. Every exception is caught and fed into two streams: One stream sends a notification to a slack channel, whereas the other stream posts the exception in audit_event table in the database.

A description of the data model can be found here: Audit Data Model

The auditing framework is designed to satisfy the inconsonant objectives of modeling VRO claim requests and also be completely generic. It achieves this via a layer of abstraction:

• Any object that needs to be audited in the database must implement the Auditable interface:

public interface Auditable {

  String getEventId();

  String getDetails();

  String getDisplayName();
}

Auditable objects can be converted into AuditEvent objects by means of the method AuditEvent.fromAuditable(). Audit events can be triggered asynchronously via wireTap. For example:

  private void configureOrderExamStatus() {
    // This route does not do anything, but an audit event is persisted
    String routeId = "mas-exam-order-status";
    from(ENDPOINT_EXAM_ORDER_STATUS)
        .routeId(routeId)
        .wireTap(VroCamelUtils.wiretapProducer(EXAM_ORDER_STATUS_WIRETAP))
        .wireTap(ENDPOINT_AUDIT_WIRETAP)
        .onPrepare(auditProcessor(routeId, "Exam Order Status Called"))
        .log("Invoked " + routeId);
  }

The command wireTap(ENDPOINT_AUDIT_WIRETAP) sends a wiretap to the audit endpoint, the the onPrepare command is responsible from mapping the current object (an Auditable) to an AuditEvent.

Slack messages are also sent as part of the auditing framework, except in this case an AuditEvent is converted into a user-friendly string. Here are some examples:

AuditEvent{routeId='/automatedClaim', payloadType=Automated Claim, message='Claim with [collection id = 351] does not qualify for automated processing because it is missing anchors.}

AuditEvent{routeId='/automatedClaim', payloadType=Automated Claim, message='Claim with [collection id = 351] does not qualify for automated processing because it is missing anchors.}

AuditEvent{routeId='/automatedClaim', payloadType=Automated Claim, message='Claim with [collection id = 351], [diagnostic code = 7101], and [disability action type = FD] is not in scope.}

Exception occurred on route mas-claim-processing for Automated Claim(id = a14fd4e5-abe0-48b7-95df-3f9c85164a3a): Error in calling collection Status API .
Please check the audit store for more information.

Apache Camel makes some things easy and some things hard. One of the hard things is that Camel does not enforce type safety as a message travels from route to route. If the output type of a route does not match the input type of the next route, you are looking at a runtime exception.

I have found that the best way to debug Camel is to inject a processor between two routes and set a breakpoint to examine the contents of the message.

For example, consider the following snapshot:

      .routingSlip(method(slipClaimSubmitRouter, "routeHealthSufficiency"))
      .unmarshal(new JacksonDataFormat(AbdEvidenceWithSummary.class))
      .process(new HealthEvidenceProcessor()) 

A message is processed via a routingSlip, then converted from JSON to the type AbdEvidenceWithSummary which is the input type of HealthEvidenceProcessor. Suppose this conversion does not work for some reason and we want to know why.
We can interject a processor between the two steps of interest like so:

        .routingSlip(method(slipClaimSubmitRouter, "routeHealthSufficiency"))
        .process(
            new Processor() {
              @Override
              public void process(Exchange exchange) throws Exception {
                var message = exchange.getMessage();
                var body = message.getBody();
                System.out.println(body);
              }
            })
        .unmarshal(new JacksonDataFormat(AbdEvidenceWithSummary.class))
        .process(new HealthEvidenceProcessor()) // returns MasTransferObject

and in this way we can examine the details of the exchange, the message, and the message contents.

Mail Automation Systems (MAS) fast-tracks claims from CMP (Central Mail Portal). The fast-tracking capability will eventually be ported to VRO.

VRO will need to trigger OCR+NLP processing of eFolder documents for veterans of eligible claims to extract health data as evidence for fast-tracking.

• Would be preferable if there was a Lighthouse API for this.

To retrieve the OCR+NLP results, VRO will query MAS directly until the Claim Evidence API is available.

VRO may want to automatically order a medical exam using MAS's exam-ordering capability.

• According to Boise RO, the value of auto-ordering exams for CFI (claim for increase) was pretty straightforward and accurate.
• Uncertain if VRO should auto-order exams for new claims.

How to access MAS endpoints

1. Call the auth server API with the required scope, grant type, and credentials to obtain the JWT token.

   Token Endpoint: https://{baseurl}/pca/api/{environment}/token
   scope : openid
   grant_type : client_credentials
   client_id: {client id}
   client_secret: {client secret}
   

2. Check the status of a collection by invoking the "collection status" API with the JWT minted above as the bearer token. https://{baseurl}/pca/api/{environment}/pcCheckCollectionStatus

   Link to open api spec: https://github.com/department-of-veterans-affairs/abd-vro/wiki/MAS-api-spec-%28IBM-hosted-api%29

3. Call the "Collection Annotations" endpoint only if the collection is ready to be processed with the JWT minted above as the bearer token. https://{baseurl}/pca/api/{environment}/pcCheckCollectionStatus

   Link to open api spec: https://github.com/department-of-veterans-affairs/abd-vro/wiki/MAS-api-spec-%28IBM-hosted-api%29

4. Call the "Order Exam" endpoint if the evidence is not sufficient for the claim with the JWT minted above as the bearer token. https://{baseurl}/pca/api/{environment}/pcOrderExam

   Link to open api spec: https://github.com/department-of-veterans-affairs/abd-vro/wiki/MAS-api-spec-%28IBM-hosted-api%29

See EVSS

If a claim is fast-track-able, VRO will need to upload a generated PDF to eFolder and associate the PDF to the claim.

• VRO should use LH's new EVSS-replacement API (timeline? suitability for VRO?)

The following is ordered according to when it will be needed.

• ✅ health evidence data - LH Patient Health API (FHIR)
• mark claim as being assessed for fast-tracking so that users don't work on the claim
  • mark using a VBMS/BGS "station 398" and "Rating Decision Review - Level 1" special issue? Paul Shute says it's not a good long-term solution
  • LH doesn't currently have this capability
  • The RRD (Rapid Ready for Decision) special issue also needs to be set.
• upload PDF to eFolder and associate the PDF to the claim
  • LH is working on a new EVSS-replacement API (file upload service) to do this: timeline? suitability for VRO?
• claim details (diagnostic codes) using a claimID
  • LH Benefits Claims API response is missing diagnostic codes
• veteran details (SSN, name, DOB) to generate a PDF
  • LH Veteran Verification APIs doesn't return veteran information like SSN, name, DOB
• MAS claim notification to VRO (for VRO 2.0)
  • Expose VRO endpoints as a Lighthouse API for MAS to use?
• query VBMS and BGS to verify some things (MAS does this for CMP-submitted claims)
  • TBD: Need to determine MAS features that will be ported to VRO. Can we do the queries via Lighthouse?
• request OCR processing from MAS and retrieve OCR results from MAS
  • VRO will connect directly to MAS
• listen for BAM/BIA contention event notifications (for VRO 3.0)
  • VRO will need to set up a event subscriber directly with BIA's Kafka
• retrieve OCR results from BAM's Claims Evidence API
  • VRO will connect directly to CE API
• query API to map given veteran identifier into an ICN, which is needed to query LH health data
  • LH Benefits Claims API v2 accesses MPI to do this, but the veteran's SSN, name, and DOB is needed. Spoke with Derek but will need new "SR" to access MPI FHIR API to enable lookup by file number (a.k.a., BIRLS ID)
• mark claim as Ready for Decision (RFD)
  • This is not required for single issue CFI, but will be needed for handling multi-issue claims. We should be guided by the AIM project on when this is needed.

Not yet incorporated into ordered list above:

• veteran service history for presumptives
  • perhaps the Veteran Verification APIs, which uses eMIS, which is an API for VADIR (VA/DoD Identity Repository)
• Pact Act CorpBD flashes
  • requested to be added to LH Veteran Verification API but it doesn't currently support CCG (machine-to-machine) authentication

This pages answers questions like: Where is the software heading? How will we get there and in what order? What dependencies need to be satisfied?

High-level visualization:

• version 0.1: Minimally functional software; receives request, routes to dummy processor, returns response
  • includes (Ruby) PDF generator from RRD
  • (subsequent versions can be created to migrate other RRD functionalities to VRO (e.g., claim establishment, uploading docs for the claim)
• assess-data: Assess fast-tracking eligibility given all health data (as part of the request)
• assess-claim: Assess fast-tracking eligibility given claim (VRO must query for health data)
• use VRO from vets-api: RRD code (in vets-api) uses VRO API (deployed in prod)
• save-to-db: Save VRO claims processing to DB for reporting and diagnostics
• version 1.0: VRO actively used by va.gov-submitted claims
• (subsequent version will depend on readiness of presumptive claims, OCR data, CMP claims, and VBMS notifications)

Current Software State

• Software approach/architecture: Plan to Deploy to Lighthouse DI
• Software Design Document
• CamelApp demonstrates using Apache Camel for microservices to prove out the architecture

• VRO v2 Roadmap - iMVP (Integrated MVP)
• VRO v1 Roadmap - MVP

Architecture diagram (DrawIO diagram source)

What is the Future State of VRO? It depends. There are several external factors outside of the VRO team that influence what VRO will look like. One possibility is: Track this evolution in ticket #92.

• Problems being solved
• Timeline
• (draft) VRO versions

Links to documents, videos, and any resources regarding the Amida hand-off to OCTO.

• SD Elements + cATO handoff video / folder
• Camel Route video session 1 / video session 2 / folder
• BIP API video / folder
• Lighthouse API video / folder
• VRO Environment Cross walk Spreadsheet: VRO Environment Crosswalk

• MAS API video / folder
• RRD logic and PDF Generator video 1 / video 2 / video 2 (slack) / 101 notes (sharepoint) / 101 notes (google) / folder

(Much of the following was extracted from ABD Vision slides, drafted May 2022.)

• The initial VA.gov Benefits Automation prototype was built inside vets-api to prioritize speed to production and ability to learn/iterate quickly​
  • This wasn’t a long term home. Scaling inside vets-api would burden the normal 526 submission pathway and add unnecessary complexity. Benefits automation should be a separate, VA-owned service called by vets-api.​
• The MAS Benefits Automation logic is currently a managed service​ (owned by IBM contractors)
  • This creates a dependency with a proprietary codebase and reduces the ability to collaborate and learn.
• It is costly and confusing to build Benefits Automation processing in separate prototypes when they should be as consistent as possible regardless of claim source (i.e., mail, fax, or online).

We will​

• Dedicate resources to architecting and building a shared service that processes claims faster regardless of intake source​ (i.e., this VRO software)
• Cut off prototype duplication (i.e., stop feature development on the RRD prototype)
• Ensure all benefits automation logic exists in the shared service, which is open-source by default and owned by VA​ (i.e., VRO will have an API, hosted by LHDI)

Conceptual vision:

(Note that VRO may be notified of claims from CMP, VA.gov, and other claim sources by VBMS event Pub-Sub mechanism.)

Roadmap ​

• Deferring too much until after migration to Lighthouse DI. Existing pain points and areas for enhancement grow the longer we discuss and work toward the “long term solution.” Can we identify smaller, incremental steps that can reduce problems, add value, and be shipped sooner?
• Doing this architectural migration work at the same time we’re trying to ship as many presumptive features as possible w/ potential legislation implementation deadlines. When we migrate existing functionality to the new platform, it will temporarily slow development of new functionality.
• Scaling teams and cross-team operations in tandem with delivering the new platform alongside key new functionality to support presumptive claims. Can we clearly define areas of ownership and quickly align on standard ways of working through collaboration to establish a "one team" culture?

Communication is key!

VA Slack channel: #benefits-virtual-regional-office

(Refer to Roadmap for details on particular topics)

• Yoom - VRO vision and architecture
  • relevant skills: Java, Ruby, Groovy, Gradle, Spring, Docker, Bash
• Seth - full stack, SecRel
  • relevant skills: Python, Java, Kotlin, react, JS, CSS, Docker, AWS, FHIR/HL7, etc.
• Yang - full stack, vets-api
  • relevant skills: Python, Ruby, JS, Bash
• Mason - devops, full stack
  • relevant skills: Java, Python, Docker, AWS, Kubernetes
• Ilya - backend SDE for VRO(?)
  • relevant skills: Java Spring, Node.js, Ruby/Rails
• Cheng - full stack
  • relevant skills: Java, Spring, Javascript/Typescript, Docker, AWS
• Derek - backend
  • relevant skills: Java, Spring, AWS, Python
• Teja - full stack, devops, architecture
  • relevant skills: Typescript/JS, AWS, Python, Java, Ruby/Rails, NodeJS, Docker, React, etc

This page answers questions like: In what state is the software? What can the current software do? Where's the changelog?

Please add new significant changes and capabilities in reverse chronological order below. The headings below refer to the blocks illustrated in the Roadmap.

• QA release for VRO v1 Roadmap

• Integrate with PDF generation end points (TBD).
• Save assess-claim output to DB for reporting and diagnostics (TBD).
• Pull request #172 integrated with assess-claim end point on the input level. Whenever assess-claim end point is called the veteran and claim information is now saved to the database for reporting and diagnostics.
• Pull request #166 integrated the flyway scripts and db initiation to the code base.
• Pull request #123 started to integrate save-to-db functionality into the codebase.
• Save to DB integrated in camel route "access claim" (multiple PRs)

• The end-point \health-data-assessment is now ready for v1.0 and can be tried out from Swagger UI (local only until devops make dev and qa environments ready) for patients in Lighthouse Health API Sandbox. All the json field are also available from Swagger UI and matches fields used by va.gov. The following are active fixes and improvements.
  • Tests and linting for Data Access service (which is the code base for the Lighthouse Health API interaction) are not integrated to abd-vro build.
• Since current Hypertension PDF includes medications we added medications to health data for diagnostic code 7101. Pull request #181
• VRO Swagger documentation is updated in #169 and #170. The end point can be tried out from Swagger UI.
• Pull request #174 corrected a few fields in medication query to match fields used by va.gov.
• Queries for health data from Lighthouse Health API are integrated to the code base in #163. Authorization, blood pressure query for diagnostic code 7101, and medication query for diagnostic code 6602 are implemented. Conditions and procedures are also included but are preliminary. Blood pressure fields are identical to fields used by va.gov.

• The end-points \evidence-pdf (POST and GET) are now ready for v1.0 and can be tried out from Swagger UI (local only until devops make dev and qa environments ready). All the json field are also available from Swagger UI and matches to assess-claim. You can use output of assess-claim end-point \health-data-assessment to generate input for \evidence-pdf.
• JSON inputs to the endpoints are improved, Swagger documentation is added and full JSON object integration with assess-claim is achieved in Pull Requests #172 and #178.
• Python service code structure is improved and code coverage is added in Pull Request #168.
• PDF Generation started to use Redis instead of S3 and integrated to demo end points in Pull Request #143.
• Testing is integrated to CI/CD pipeline in Pull Request #137.
• V0.1 Generator is disabled in Pull Request #131.
• A series of Pull Requests (#126, #127, #128, #129, #130) integrated PDF generation into the codebase. They implement a stand alone Python service that generates PDFs for Hypertension and Asthma and saves it to S3.

This provides further assessment calculations on the data provided by assess-claims (such as predominant blood pressure reading). Is not necessary for V1.0 since it is not part of the current va.gov functinality.

• Creates a queue to which claim data is passed and responds with a decision on fast track-eligibility and supporting evidence. Messages to the queue must be routed with their VASRD DC code connected to the claim, which in this case is "7101". #112

• Initial VRO API with generate_summary_pdf and assess_data endpoints (PRs #71 and #109)
• Deployed RabbitMQ and Ruby microservice containers to LHDI's dev environment (PR #69)
• Ruby PDF generator (and health data assessor) ported from RRD prototype (#63)

• Added RabbitMQ and initial Camel routes (PR #50)
• Deployed to LHDI's dev environment (PR #47, along with #46 and #44)
• Enabled CircleCI checks to new pull requests (PR #16)
• Used LHDI's Java Starter Kit to populate the initial codebase (PR #8) using Java (AdoptOpenJDK) 17 and Gradle 7.4

See all merged PRs.