FACOLS - department-of-veterans-affairs/caseflow GitHub Wiki

Our README has basic FACOLS setup information. This page goes into more details and specifies some debugging steps as well as how to add new data to FACOLS.

FACOLS is our fake version of VACOLS which we use for local development and testing. We built it to run our test and development code in an environment as close to production as possible.

As part of our DMS backup of VACOLS, we have the following script which dumps the VACOLS schema. To build FACOLS we created our own Oracle image based on the information from here. We modified provided Dockerfile to include our own set of scripts and schema. Our Dockerfile is located here.

Running the image

We have an automated script that you can run with rake local:build that runs commands to bring up your development and test environment. To destroy it all you can run rake local:destroy.

This starts up the container, and applies the schema files. When it says it's ready, you should be able to connect and use it.

You can also download the FACOLS image instead if building it. Follow these steps:

Note: in this example, the latest image is facols:20190508. To find the latest image name, check our circleci config

  1. Clean up if you had a previous FACOLS setup: docker-compose down --remove-orphans
  2. Follow the instructions here to get MFA
  3. eval $(aws ecr get-login --no-include-email --region us-gov-west-1)
  4. docker login
  5. docker pull 008577686731.dkr.ecr.us-gov-west-1.amazonaws.com/facols:20190508
  6. docker tag 008577686731.dkr.ecr.us-gov-west-1.amazonaws.com/facols:20190508 vacols_db:latest
  7. docker-compose up -d
  8. Seed FACOLS: bundle exec rake local:vacols:seed

Rebuilding FACOLS Locally

  1. Clear your locally docker environment: bundle exec rake local:nuke WARNING This will destroy everything in your docker environment, proceed with caution
  2. Make changes to whatever SQL files are needed to be updated both for test and for dev that are located in the local/vacols folder and are prefixed with vacols_copy_
  3. Change to the facols directory to pick up contextual dependencies cd local/vacols
  4. Rebuild the image by overriding the command to pull down the image: ./build_push.sh remote

NOTE: By using the remote flag you will be pushing up a new version of the FACOLS image for other developers to use

  1. Once this finishes, run make reset and your environment will now contain the new version of the build

Development data pipeline

In order to develop against real data we created a data pipeline to pull data from a prod copy, sanitize, save it into CSVs and import it into FACOLS. This section explains how to add new data and new tables into our FACOLS records.

Circle CI

We use Circle CI to run all of our tests before devs can merge their changes. To enable Oracle on Circle, we publish our built image to ECR. This can be done with the following script. Our Circle CI configuration file then specifies this container to pull in and run against. The Circle CI config file also runs rake local:vacols:wait_for_connection in the main container, which pings FACOLS until it gets a response. This indicates the container is ready to run tests.

Crashes

Our Docker setup can be unstable on some machines. This can result in crashes. You might notice this when a request to the rails server hangs or a test hangs. For some people this happens regularly at 5pm EST/6pm EDT. You can try to restart the container: docker-compose restart. If that fails, restarting the docker service from the UI, and then running docker-compose restart when it comes back up should fix it.

Debugging FACOLS

We've encountered a few types of problems with the FACOLS setup.

  1. Something gets corrupted. This is the most common case. To solve this problem run docker system prune. This nukes your Docker setup so you can start over.

  2. Not enough memory/cpu. Unfortunately running the docker container is really heavy. Many devs have found the need to increase the amount of resources available to Docker. You can do this in the preferences menu from the Docker UI. You should try this if running docker system prune and then setting up the container didn't work. Increasing disk space, memory, or CPUs might help. We recommend the following settings:

Disk image size: 96.0 GB
CPUs: 4
Memory: 4.0 GiB
Swap: 1.0 GiB

This section explains more advanced FACOLS usage. Specifically:

  1. How to add new cases to FACOLS
  2. How to add new tables to FACOLS
    1. How sanitizaiton of data works

Adding new cases to FACOLS

We check FACOLS data directly into GitHub since the data has been scrubbed of any PII. The test data comes from the RDS instance named: dsva-appeals-vacols-uat-datasource-2017-12-13-11-20. This is a copy of UAT from December 2017 which means that it contains PII. We pull a subset of the data from this RDS instance, sanitize it, and dump it to CSV files that we check into local/vacols. The master list of cases in FACOLS is maintained in local/vacols/cases.csv. Here is an example excerpt from the file:

vacols_id vbms_key bgs_key used_in_app comments
3575931 static_documents reader queue Case assigned to attorney
3619838 no_categories reader queue Case assigned to attorney
3625593 random_documents reader queue Case assigned to attorney
  • vacols_id
    • This column represents the bfkey from the brieff table.
    • This is just a DB primary key and so is not considered PII.
  • vbms_key
    • Currently used to specify which documents we want to associate with this case. The choices are static_documents, no_categories, random_documents, redacted_documents.
    • In the future this field will be used to coordinate how this case associates with fake VBMS data.
  • bgs_key
    • Similar to vbms_key, but for setting up bgs, like has_many_ratings. See lib/fakes/bgs_service.rb
  • used_in_app
    • A list of which apps rely on this case.
    • If you no longer need a case, remove the app from this list.
    • If no apps are using it, it's safe to remove this row from the CSV.
    • If you start using a case, add your app's name to this list so no one else removes it from the list.
  • comments
    • Anything you'd like to add to give this case context.

To add a new case:

  1. Add a row to this above CSV.
    • To find a relevant vacols_id to use, you can run the rails console in the ssh_forwarding environment. This will enable you connect to the datasource RDS instance to peruse the available data. Instructions of connecting can be found here.
    • The ssh command to run to connect to the datasource RDS instance is: ssh -L 1526:dsva-appeals-vacols-uat-datasource-2017-12-13-11-20.cdqbofmbcmtd.us-gov-west-1.rds.amazonaws.com:1526 <Your UAT jumpbox ssh config>.
    • The username and password for the datasource RDS instance is in credstash
    export VACOLS_USERNAME=`credstash -t appeals-credstash get vacols.facols.db_username`
export VACOLS_PASSWORD=`credstash -t appeals-credstash get vacols.facols.db_password`
    • Once connected, you can run any queries you want to find useful cases, then get their bfkeys from the brieff table and copy them into the cases.csv file.
  2. Run the following to retrieve all the data for the cases you added: 
    RAILS_ENV=ssh_forwarding rake local:vacols:dump_data
    • This will dump data into the relevant CSV files in local/vacols so they can be imported later into FACOLS.
    • When this finishes running, you might (but are unlikely to) see red-text explaining that there may be PII in the data you pulled. If this is the case, we may need to white-list or fake out another field. Coordinate with Mark and Chris to make this happen.
  3. Run the following to import the dumped data into FACOLS: 
    rake local:vacols:seed
  4. Check in the resulting changes to the dumped-data CSVs.

This will seed FACOLS with the new data. Note when seeding data, all date-time fields are date shifted as if today was November, 1, 2017. In this way we can keep even old data current. After this you should be good to go! Note all PII will be sanitized, and staff ids will be scrambled, but you can backtrack any necessary information from the bfkey which stays the same.

Adding new Case Review to FACOLS

A VACOLS Case Review is comprised of a few different models from the VACOLS DB. The first thing to note is that a VACOLS Case Review is just an alias for a VACOLS Case Assignment (see Legacy Appeal). VACOLS Case Assignments is not a table of records that exists, but rather a collection of tables that have been joined into the Case Assignments view on the DB. The following models are involved in the VACOLS Case Assignment:

  • VACOLS::Case (also called brieff see Case)
    • Vacols ID (bfkey) required (Can be any unique number)
    • VBMS ID (bfcorlid) required (Must be a valid VBMS ID defined in vbms_setup)
    • Docket Date (bfd19) required
    • Assigned to Location Date (bfdloout) required
  • VACOLS::Decass
    • Assigned to Attorney Date (deasign)
    • Re-Assigned to Judge Date (dereceive)
    • Date Completed (decomp)
    • Document ID (dedocid)
    • Work Product (deprod)
    • Attorney ID (deatty)
  • VACOLS::Folder
    • Docket Number (tinum)
  • VACOLS::Staff
    • Added by First Name (snamef)
    • Added by Middle Name (snamemi)
    • Added by Last name (snamel)
    • Added by CSS ID (sdomainid)
    • Assigned by First Name (snamef)
    • Assigned by Last Name (snamel)
    • Written by First Name (snamef)
    • Written by Last Name (snamel)

In order to add a new Case Review, a unique record needs to be added to the FACOLS CSV file for each of the above tables (except for staff). The staff record should be whatever the slogid of the VACOLS::Staff to which you want to assign the case. After the seed data has been updated, you will need to associate the VACOLS::Case with a Legacy Appeal and VACOLS::Correspondent in the seeds,rb file.

Reference

Adding new tables to FACOLS

When adding a new table you must add code to sanitize any potential PII. Unfortunately, adding a new table is somewhat complicated, and if done incorrectly may leak PII, so please pair with either Mark or Chris on this process.

  1. Every table we add must have a wrapper class just like VACOLS::Case that defines the table for active record. Once that's established there are two different procedures to add two different types of tables.

    1. Adding a table that is related to the brieff table. For example, the hearsched table belongs to the brieff table. For these tables, we pull in rows that relate to the cases specified in the cases.csv file. To add the table, you'll need to add two things to the local:vacols:dump_data rake task defined in vacols.rake.
      1. Although technically optional, to reduce the number of queries you can add the table to the VACOLS::Case.includes statement.
      2. Add a line similar to write_csv(VACOLS::CaseHearing, cases.map(&:case_hearings), sanitizer). This writes the data from each case_hearings row that corresponds to a given case.
    2. Adding a table that is not related to the brieff table. For example the staff table has all the BVA staff members in it. For these kinds of tables, we must select which rows to bring in. What you choose depends on the use case. Some tables are completely dumped, while in others we only dump certain rows. Look at how we dump the tbsched table for reference: 
      write_csv(
   VACOLS::TravelBoardSchedule,
   VACOLS::TravelBoardSchedule.where("tbyear > 2016"),
   sanitizer
)

  2. Sanitize the data. The write_csv function takes in a sanitizer object. This sanitizer is defined in sanitizers.rb. In write_csv we call a sanitize method which uses meta-programming to call two methods specific to the class we're sanitizing:

    1. white_list_#{lower_case_class_name}: This method must return a string array of a set of fields to white list. All other fields are immediately nilled out. This is to ensure we don't actually write a field we do not intend to.
    2. sanitize_#{lower_case_class_name}: This allows us to fake out fields that were or were not white listed. If either of these methods are not defined, then sanitization will fail.

  3. Dump and seed the data:

    RAILS_ENV=ssh_forwarding rake local:vacols:dump_data
rake local:vacols:seed

  4. Check the resulting CSV updates into git.

  5. Whitelist fields with detected PII. When sanitization is run, we examine all white-listed fields for PII, including emails, phone numbers, sentences, and vet ids. If any field matches any of these RegExs then we print it in red at the end of the dump data job. This can alert you to a possible leak. To respond, you should either, stop white-listing the field if it is actually PII, or add the field to a special white-list for fields that detect PII but don't actually contain it. This white-list is in the ignore_pii_in_fields method.

Adding MST/PACT to FACOLS

If you have a build prior to the release of MST/PACT, you will need to rebuild your FACOLS image (Windows/Mac with Intel) or update your FACOLS VM setup with ISSMST and ISSPACT columns on the ISSUES table. Without the updated build, your local setup will have issues with legacy appeals. Setup steps can be found here

Manually seeding your local VACOLS container

To seed the VACOLS container with data you'll need to generate the data for the CSVs first.

  1. bundle install --with staging to get the necessary gems to connect to an Oracle DB
  2. rake local:vacols:seed to load the data from the CSV files into your local VACOLS
  3. rails s to start the server connected to local VACOLS or rails c to start the rails console connected to local VACOLS.

Certification Test Scenarios

BFKEY/VACOLS_ID Case
2367429 Ready to certify with all dates matching
2774535 Ready to certify with fuzzy-matched dates
2771149 Mismatched documents
3242524 Already certified