BIP APIs - department-of-veterans-affairs/abd-vro GitHub Wiki

As of the January 2024 deployment of the BIP shared microservice, VRO is only using the BIP Claims API, and while it is no longer using the BIP Claim Evidence API, the codebase for the Claims Evidence API is still present.

Information specific to one of the individual APIs can be found in the above pages. This page, in addition to providing these links, describes the details of VRO Mutual TLS implementation.

We will dive into the details of the TLS certificates momentarily, but it is first useful to understand the big picture of how these certs are used. First, there is the certificate management system that acts as the VA's primary Certificate Authority (CA). Our CA is provided by LHDI, a system called Venafi, which is deployed as a feature of our Kubernetes infrastructure. Through Venafi we provision and query for certificates and once provisioned, Venafi will manages the expiration of these certificates. Second, there is the generation process itself which is a bash script that lives in the VRO repo: scripts/svc-bip-api-certgen.sh. This script pulls the certs from Venafi and combines them with the intermediate certificates which it also downloads. The certs from Venafi, along with the Intermediate certificates are then crunched through a series of openssl commands to ultimately produce an output.json file. Third, there is HashiCorp Vault, which is where we store the generated truststore, keystore and the password used to seal these stores. Forth, and finally, there are the Kubernetes secrets that are used to supply the application with the truststore, keystore and password.

Please reference the following diagram as you navigate through the remainder of this documentation:

(note: this image was created with draw.io - the png file contains the draw.io source as embedded data to the png file)

VRO_SECRETS_BIP

The secrets that are pertinent to these integrations can be found in HashiCorp Vault under `vro-secrets/deploy/{ENV_NAME}/VRO_SECRETS_BIP and include:

  • BIP_CLAIM_SECRET
  • BIP_CLAIM_USERID
  • BIP_EVIDENCE_SECRET
  • BIP_EVIDENCE_USERID
  • BIP_KEYSTORE_BASE64
  • BIP_PASSWORD
  • BIP_TRUSTSTORE_BASE64

Mutual TLS (mTLS)

BIP APIs as well as other VA services require Mutual TLS, or mTLS. It is necessary for VRO to use a client certificate signed by a VA recognized Certificate Authority (CA) during https handshake. In addition VRO needs to use a VA recognized CA to validate BIP certificates during https handshake.

Certificate Management

Venafi is the VA's new Internal Management System which can be used to manage the PKI certificates. Within VRO Kubernetes cluster the process is further simplified and it is possible to generate the certificates within one of VRO Kubernetes namespaces. The client certificates that are being used currently by VRO has been generated using these instructions.

The certificates tls.key, tls.crt, and va.crt has been generated in va-abd-rrd-dev namespace and can be found in va-abd-rrd-dev-va-gov-tls secrets.

The private key tls.key is not encrypted so there are no passwords needed for steps described later in this document.

Details for tls.crt can be displayed using

openssl x509 -in tls.crt -text -noout

The certificate tls.crt expires in January 6th, 2024.

The certificate tls.crt contains both the client public key and an intermediate CA. Only the client public key is used and will be referred as tls_bip.crt.

The certificate va.crt is not currently used.

By default, certificates generated are valid for one year, and are set to auto-renew ⅔rds of the way through their lifecycle.

Production Use

A second set of certificates are generated in production. All the discussion in this document is identical for production certificates. The production certificate expires in March 17th, 2024.

Future Work

As discussed later in this document we do not yet know a way to use these certificates directly in the Java code and there are manual steps to make them usable. More discussions are needed with LHDI around the process on how to update them.

  • The certificates will be auto-renewed. What is the trigger to perform manual steps when the switch is made?
  • Should LHDI make these certificates available to containers without additional steps?
  • Can we forego container level https and move that complexity to API Gateway so that it is managed by LHDI?

At this time Java cert renewal is manual and steps in this document should be followed. We have time until January 6th, 2024 to make improvement.

Downloaded Certificates

During VRO implementation of mTLS we were not successful using the VA CA public keys in va.crt and tls.crt.

Instead we downloaded the public keys that were used by BIP APIs from Chrome on a GFE:

  • Open Chrome on your GFE
  • Open one of the end points in BIP Claims API or BIP Claim Evidence API. If you use a GET end point you should typically get a jwt missing error
  • Once you get the error click on the lock (View site information) in the url bar to bring up a pop-up window
  • Select "Connection is secure" menu item to bring up a second pop-up window. From this pop-up you can select "Certificate is valid" to bring up the "Certificate viewer".
  • Goto "Details" tab, select the certificate for root CA and "Export". Repeat for the intermediate CA

Two certificates downloaded are

  • VA-Internal-S2-RCA1-v1.crt (root)
  • VA-Internal-S2-ICA4.crt (intermediate)

These public keys are concatenated in a file named va_all.crt for later use.

Future Work

We have not retried to use the VA CA public keys in va.crt and tls.crt after we successfully used va_all.crt. This should be revisited in the future since it is possible that the steps we eventually ended up with could be succesful for public keys in va.crt and tls.crt as well.

Self-Signed Certificates

VRO generates a set of self-signed certificates using a script called build-certificates.sh. These certificates are used for local development and end-to-end tests to mock the actual certificates and mTLS based https handshake.

More details and what is being generated are documented in the script itself.

VRO mTLS Implementation

PKCS #12 Files

As of now we do not know a simple way to consume the certificates tls.key, tls_bip.crt, and va_all.crt directly in the Java code to utilize them in https calls. Java uses either JKS files or PKCS #12 files. Since PKCS #12 files are not Java specific, we prefer them over JKS files in VRO.

To generate the PKCS #12 file keystore.p12 for client certificates we use openssl

openssl pkcs12 -export -in tls_bip.crt -out keystore.p12 -name keystore -nodes -inkey tls.key

This commands asks for an "export password". This password is to be recorded as it needs to be later used as a secret.

To generate the PKCS #12 file truststore.p12 for VA CA certificates we use keytool

keytool -import -file va_all.crt -alias all_cas -keystore truststore.p12

This command asks for a "keystore password". For simplicity VRO uses the same the value of "export password".

Future Work

It should be possible either to use keystore or openssl exclusively here. But these PKCS #12 files worked and we did not try either route any further.

Kubernetes Secrets

VRO uses three environment variables to store the content of the [PKCS #12] files and the "export password".

  • BIP_KEYSTORE
  • BIP_TRUSTSTORE
  • BIP_PASSWORD

Since PKCS #12 files are binary we convert them to Base 64 using openssl

openssl base64 -in keystore.p12 -out keystore.b64
openssl base64 -in truststore.p12 -out truststore.b64

What is stored in the environment variables BIP_KEYSTORE and BIP_TRUSTSTORE are the content of the Base 64 files.

In our environments these environment variables are made available to the app container using the following secrets

  • bip.bipKeystore
  • bip.bipTruststore
  • bip.bipPassword

Manual Steps

In summary the manual steps to make the certificates available to the Java code are

  • Convert certificates to PKCS #12 files
  • Convert PKCS #12 files to Base 64 files
  • Store content of the Base 64 files in Kubernetes secrets
  • Restart the pod to make secret changes effective

If https handshake implementation remains in the Java code (as opposed to say moved to API Gateway) these manual steps need to be automated if certificate renewal is automated.

Java implementation and Code Walkthrough

Java uses Keystore objects to store certificate information in PKCS #12 files. VRO

  • Reads in the content of the PKCS #12 files and the password from the environment variables BIP_KEYSTORE, BIP_TRUSTSTORE, and BIP_PASSWORD through application.yml
  • Converts Base 64 content to binary content
  • Creates the keystore and trusstore objects as Keystore instances from the binary content and the password
  • Creates a custom RestTemplate bean that can be used to make the https requests

The properties in application.yml that corresponds to the environment variables are

  • keystore for BIP_KEYSTORE
  • truststore for BIP_TRUSTSTORE
  • truststore_password for BIP_PASSWORD

The RestTemplate bean that is used to make the https requests is implemented in BipApiConfig. This file also includes the generation of the Keystore objects from the application.yml properties. This RestTemplate bean is autowired in the rest of the code using the Qualifier bipCERestTemplate.

Curl Validation of Certificates

Validation of the certificates using the Java code has been problematic since the BIP APIs are not available outside of VA Firewall

  • Our GFE's typically do not have Java installed
  • Extra permissions are necessary to install and maintain Java on the GFE's
  • Additional security features in GFE's makes it difficult to make https calls from Java

curl has been an invaluable tool to test the validity of the certificates. In principle it should be possible to run curl in your GFE but running it from the app container in one of our environment was easier. We copied the certificates tls.key, tls_bip.crt, and va_all.crt to the container using cat <<EOF and copy&paste in the /tmp directory. You should be able to use kubectl cp with enough permissions. Once the files are in the container you can run

curl -X POST <https://......> --cacert va_all.crt --cert tls_bip.crt --key tls.key --verbose

To see the https handshake to verify the validity of the certificates. The request can be made to any end point of interest.