Convert MIMIC-IV into FHIR resources

This guide will run through converting data in the MIMIC-IV Clinical Database into resources following the FHIR format. Briefly, we:

1. Convert MIMIC-IV from its original schema into a set of tables with one FHIR resource per row
2. Export this data as ndjson
3. Load this data into HAPI FHIR
4. Validate the FHIR resources match the MIMIC FHIR profiles.

Prerequisites

Before getting started make sure you have MIMIC-IV loaded into an accessible PostgreSQL database. Follow this MIMIC-IV guide to set it up. You will need the mimic_hosp, mimic_icu, and mimic_ed schemas to be present and fully populated with MIMIC-IV data.

Your postgres user will need permission to create schemas and data - the below scripts will create and populate the fhir_trm and fhir_etl schemas.

Quickstart

1. Clone the repository locally:

git clone https://github.com/kind-lab/mimic-fhir.git

2. Generate the FHIR tables by running create_fhir_tables.sql found in the folder mimic-fhir/sql

psql -f create_fhir_tables.sql

3. Set up HAPI FHIR for use in validation/export

   • The first step in validation/export is getting the fhir server running. In our case we will use HAPI FHIR.
   • We made a fork of the jpa starter server: git clone https://github.com/kind-lab/hapi-fhir-jpaserver-starter.git
   • Create the postgres database hapi_r4 that will be used in HAPI:
     • From the terminal enter psql: psql
     • Create hapi database in sql: CREATE DATABASE hapi_r4;
     • Exit psql: \q
   • The application.yaml file in the hapi-fhir-jpaserver-starter project was modified to point to the mimic implementation guide
     • The mimic implementation guide is stored in the kindlab fhir-packages repo.
   • Start the HAPI FHIR server by going to the hapi-fhir-jpaserver-starter folder and running: mvn jetty:run
     • The initial loading of hapi fhir will be around 10-15 minutes, subsequent loads will be faster

4. Configure py_mimic_fhir package for use

   • Before starting the py_mimic_fhir package you need to add a .env file in mimic-fhir to match your local settings. An example .env.example file is available in the folder as reference. These environment variables will be used in the next step.
     • Export these environment variables to your terminal for ease of use. Run source .env
   • Next get the py_mimic_fhir package set up
     • Move into the mimic-fhir folder on your local machine
     • Install the package using pip install -e .

5. Post terminology to HAPI-FHIR using py_mimic_fhir

   • The default load of HAPI-FHIR with the mimic implementation guide does not fully expand all terminology. To ensure full expansion, we need to post the terminology directly. To do this
     • Ensure the environment variable MIMIC_TERMINOLOGY_PATH is set and pointing to the latest terminology files from mimic-profiles
     • Run the terminology post command in py_mimic_fhir: python py_mimic_fhir terminology --post

6. Load reference data bundles into HAPI-FHIR using py_mimic_fhir

   • Initialize data on the HAPI-FHIR server, so patient bundles can reference the data resources
   • The data tables for medication and organization only need to be loaded in once to your HAPI-FHIR server. To ensure these resources are loaded in, the first time you run mimic-fhir you must run:
     • python py_mimic_fhir validate --init

7. Validate mimic-fhir against mimic-profiles IG

   • After step 6 has been run once, you can proceeded to this step to validate some resources! In your terminal (with all the env variables) run: python py_mimic_fhir validate --num_patients 5
     • Any failed bundles will be written to your log folder specified in .env

8. Export mimic-fhir to ndjson

   • Using the py_mimic_fhir package you can export all the resources on the server to ndjson
   • Run python py_mimic_fhir export --export_limit 1
     • export_limit will reduce how much is written out to file. It limits how many binaries are written out. Each binary ~1000 resources. So in this case the limit of 1 will output 1000 resources into ndjsons
     • The outputted ndjson will be written to the MIMIC_JSON_PATH folder specified inthe .env

Useful wiki links

• The FHIR Conversion Asusmptions section covers assumptions made during the MIMIC to FHIR process.
• The HAPI FHIR Server Validation section walks through validating the MIMIC resources against various implementation guides using HAPI FHIR.
• The py_mimic_fhir CLI section details the arguments that can be used in the CLI
• The Bundle and Export section goes over the bundling process and export execution.