Exporting and Importing Appeals - TISTATechnologies/caseflow GitHub Wiki

Want to replicate sanitized real cases in your dev or test environment? If so, an initial implementation of this capability is in PR #15976. There's still more associated records (e.g., AttorneyCaseReview, EPEs, HLR, SC) and associated data from external sources (e.g., BGS, VACOLS) to include in the export, but it currently works for tasks, issues, and hearings -- see SanitizedJsonConfiguration for details.

As a developer or Bat Team member, you may want to recreate sanitized instances of prod data into your testing environment so that you can experiment with realistic data.

• The data can be imported as part of the setup for individual RSpec tests.
• The data can be manually imported into the dev DB in the Rails console.
• The data can be imported to be part of development environment's seed data so that it is available in everyone's dev and demo environments.

Example wins:

• Used the exporter to replicate an appeal locally to investigate a duplicate task problem. The appeal can be immediately fixed in prod, while I diagnose the problem locally using an RSpec created in PR #16154.
• [PROD FIX] Restore decision doc on dispatched appeal #16176
• Replicate bug in production: Case Movement user cannot reassign task #16206
• RSpec to replicate duplicate JudgeAssignTasks problem #16382

Courtesy of PR #16310, an appeal can be exported using the browser by navigating to /explain/appeals/<uuid>.json, e.g., http://localhost:3000/explain/appeals/31927be1-8447-44c4-ad66-8b73ae8bf401.json. Examine the JSON to ensure that there is no PII or sensitive data -- PII Handbook, then save the file for importing. Note: you must be logged in as a "System Admin".

Alternatively, navigate to /explain/appeals/<uuid> to see other useful visualizations of the appeal -- see https://github.com/department-of-veterans-affairs/caseflow/wiki/Explain-page-for-Appeals. Click on the blue 'sanitized json export' link to show the JSON file contents, which can then be examined for PII and then saved locally for importing.

SSM into a prod instance and note the instance ID -- you'll use it later to transfer the exported file.

# Maximize the verbosity to see what the exporter is sanitizing
sje=SanitizedJsonExporter.new(appeal, verbosity: 5) 

# Save to a file that is accessible by a non-root user like /tmp
sje.save("/tmp/appeal-#{appeal.id}.json", purpose: 'to diagnose stuck appeal')

Open the file and ensure that there is no PII or sensitive data -- PII Handbook.

Before disconnecting, make sure you have the instance ID. If you haven't already noted it, you can fetch it with curl http://169.254.169.254/latest/meta-data/instance-id while still logged into the instance.

Now, copy that file locally so it can be imported:

cd to/your/caseflow
INSTANCE=i-06b8441957026bb62
# mfasocks must be active
scp $INSTANCE:'/tmp/stuck-appeal-21.json' .

In your local Rails console or RSpec:

require "helpers/sanitized_json_configuration.rb"
require "helpers/sanitized_json_importer.rb"

...

# Maximize verbosity to see what the importer is doing
sji = SanitizedJsonImporter.from_file('stuck-appeal-21.json', verbosity: 5)
sji.import

Example: this RSpec in PR #16154

To include more associated records in the exported JSON, modify SanitizedJsonConfiguration and possibly SanitizationTransforms.

1. Add the ActiveRecord class to the @configuration that enumerates the sanitize_fields to be transformed by SanitizationTransforms and the retrieval command to retrieve associates records of that class. Note that the retrieval commands are run according to the order listed in the @configuration hash.
2. For each field/column listed in sanitize_fields, ensure that one of the methods in SanitizationTransforms will return a non-nil value, which will be used as the new value to replace the original sensitive value in the exported JSON. Note that the same original values will be transformed to the same new value for consistency, except when save_mapped_value? returns false.
3. The track_imported_ids boolean tells the importer to track the new record ids as it is importing and creating records. This is useful when one record references another records using a polymorphic association. Note that @reassociate_types may also need to be updated.
4. For certain unique records or singletons, find_existing_record may need to be updated.
5. Add RSpec tests with records having non-nil values to ensure exporting and importing works.