ProdTest Stress Test setup

The purpose of this documentation is to help serve as an SOP for testing new API endpoints within Caseflow by replicating load on the system in a way that is only seen in Production. The stress test below can help:

1. Validate new API endpoints
2. Benchmark new code being introduced to Caseflow
3. Look for weaknesses in Caseflow's current architecture

Due to the limited traffic in our testing environments, this stress test is the closest way to replicate production traffic in a safe and controlled way. With Caseflow being an enterprise level software, new code needs to be tested for Big O Notation concerns to prevent issues upon release to our production environment.

Test Setup

To stress test in prod test, you can write a script using Locust: https://locust.io/. Locust allows you to write tasks that generate API calls to simulate user traffic. These tasks will randomly call out one after another as long as the script is running (you can weight different tasks to run more often if needed though).

To set up this script, you first need Python 3.11 on your GFE: https://www.python.org/downloads/ After python is setup on your GFE, you can install locust by running pip install locust locust-plugins in either your terminal or PowerShell.

Building the Script

An example script is attached below. To build the script you will need:

1. The name of the file must be locustfile.py
2. CSV file(s) with prod test veteran identifiers (used in API call headers). These files can be generated from a Metabase query on your GFE.
3. Methods (notated with @task) that generate API calls
4. API key(s) for your headers.

API keys will need to be generated after a successful Prod Test deployment and added to the script. API keys can be generated in the Prod Test Rails console using the instructions on the Caseflow Wiki To generate an IDT token, use this in the Prod Test Rails console: key, token = Idt::Token.generate_one_time_key_and_proposed_token Idt::Token.activate_proposed_token(key, "CSS_ID_OF_ELIBIGLE_USER") # Token for IDT API token

To get an eligible user to generate your token, run: CSS_ID_OF_ELIGIBLE_USER = User.all.filter(&:intake_user?).first.css_id

Example Script

Example script in text file

Executing the script

To execute your script on your GFE:

1. Open up a terminal or PowerShell window.
2. Navigate to the folder with your locustfile.py file
3. To launch the script, run locust -H https://appeals.cf.prodtest.ds.va.gov/ to point your API calls to the prod test instance

While the Script Runs

While the script is running, Locust will generate a UI that is accessible at http://localhost:8089/. You can use the UI to set the number of users, the spawn rate of requests, and the host URL if needed. While the tasks are running in the background hitting the Prod Test endpoint, the Locust UI will also give you metrics about the number of requests and overall health of the prod test endpoint. Under the charts tab, you can have HTML charts that you can download to show the results of your test.