Flexporter - synthetichealth/synthea GitHub Wiki

Warning

🐉 Here be dragons. This feature is a very rough work in progress. Features that you expect may be missing, and features that do exist may be broken. Proceed with caution.

The Synthea Flexible Exporter a.k.a. "Flexporter" is a utility designed to make it easy to add new fields, values, and resource types to Synthea-generated FHIR, without having to modify the Synthea engine. Primary users of this feature are expected to be Implementation Guide authors and users, electronic clinical quality measure developers and testers, and anyone who needs just a slight tweak to the data exported by Synthea.

Approach

The basic idea of the Flexporter is that users will define a series of transformations to apply to Synthea data. A predefined set of transformations, such as "set field X to value Y" or "create a new resource" is intended to cover the majority of use cases while still remaining easy to use.

Usage

In Synthea

./run_synthea -fm mapping_file [-ig ig_folder]

Standalone

The Flexporter can also run standalone to post-process FHIR Bundles.

./run_flexporter -fm mapping_file -s source_fhir [-ig ig_folder]

https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/RunFlexporter.java

Mapping File

The format of the mapping file is as follows:

---
# name is just a friendly name for this mapping
name: Flexporter Mapping

# applicability determines whether this mapping applies to a given file.
# for now the assumption is 1 file = 1 synthea patient bundle.
# this should be a FHIRPath which should return a "truthy" value if the mapping applies
applicability: true

# actions is a list of Action objects. see below for details
actions:
 - name: Apply Profiles
...

Actions

https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/org/mitre/synthea/export/flexporter/Actions.java

Actions are driven by a key property name on the object. (technical note: this is because the YML library doesn't support discriminators like GSON does for JSON, think like how state types get mapped into a different class by type)

Apply Profiles

	Field Name	Type	Required	Description
🗝️	profiles	List	Yes	List of profile/applicability pairs

	profile	URL	Yes	Profile URL to be added to resources
	applicability	FHIRPath	Yes	FHIRPath to select resources that the profile should be added to

Example:
This example will apply the qicore-patient profile to all Patient resources, and qicore-encounter to all Encounters.

 - name: Apply Profiles
   profiles:
   - profile: http://hl7.org/fhir/us/qicore/StructureDefinition/qicore-patient
     applicability: Patient
   - profile: http://hl7.org/fhir/us/qicore/StructureDefinition/qicore-encounter
     applicability: Encounter

Set Values

	Field Name	Type	Required	Description
🗝️	set_values	List	Yes	List of profile/applicability pairs

	applicability	FHIRPath	Yes	FHIRPath to select resources to set fields on
	fields	List	Yes	List of location/value pairs to set

	location	FHIRPath	Yes	FHIRPath to the field to set the value on
	value	raw value, Value Function, object	Yes	The value to set on the target, or the function to fetch a value to set on the target
	transform	Value Transformation	No	Transformation to apply to the value before setting on the target

Example 1:
This example will read the value from Immunization.occurence[x] and set it on the Immunization.recorded field, for all Immunization resources.

 - name: testSetValues_getField
   set_values:
     - applicability: Immunization
       fields:
         - location: Immunization.recorded
           value: $getField([Immunization.occurrence])

Example 2:
This example will set a full coding, with code/system/display, to Patient.maritalStatus.coding.

 - name: testSetValues_object
   set_values:
     - applicability: Patient
       fields:
         - location: Patient.maritalStatus.coding
           value:
               system: http://snomedct.io
               code: "36629006"
               display: "Legally married (finding)"

Create Resource

	Field Name	Type	Required	Description
🗝️	create_resource	List	Yes	List of resource definitions to create

	resourceType	FHIR resource type	Yes	Resource type to create (Patient, Encounter, etc)
	based_on	object	No

	based_on.resource	FHIRPath	No	FHIRPath to select resources to base the new created resource off of. (Example use case, create an Appointment based on every Encounter)
	based_on.module	GMF Module Name	No	The name of a Synthea Module, where an instance of the current resource will be created for each instance of the patient passing through the state
	based_on.state	GMF Module State	No	The state to use as the basis for creating the current resource

	fields	List	Yes	List of location/value pairs to set on the created resource
	writeback	List (same options as fields)	No	List of location/value pairs to set on the selected based_on resource

	fields.location	FHIRPath	Yes	FHIRPath to the field to set the value on
	fields.value	raw value, Value Function, object	Yes	The value to set on the target, or the function to fetch a value to set on the target
	fields.transform	Value Transformation	No	Transformation to apply to the value before setting on the target

Example:
This example will create an instance of ServiceRequest for every instance of Procedure, with the following:

ServiceRequest.intent set to "plan"
ServiceRequest.encounter.reference set to the same as the source Procedure.encounter.reference
ServiceRequest.subject.reference set to a reference to the Patient resource
Procedure.basedOn.reference set to a reference to the newly created ServiceRequest

 - name: testCreateResources_createBasedOn
   create_resource: 
     - resourceType: ServiceRequest
       based_on:
         resource: Procedure
       fields:
         - location: ServiceRequest.intent
           value: plan
         - location: ServiceRequest.encounter.reference
           value: $getField([Procedure.encounter.reference])
         - location: ServiceRequest.subject.reference
           value: $findRef([Patient])
       writeback:
         - location: Procedure.basedOn.reference
           value: $setRef([ServiceRequest])

Keep Resources

	Field Name	Type	Required	Description
🗝️	keep_resources	List<FHIRPath>	Yes	List of FHIRPath to select resources that should be retained in the bundle, anything not selected by a rule will be removed

Example:
This example will keep only the Patient, Encounter, and Condition resources. All other resources will be removed.

 - name: testKeepResources
   keep_resources:
     - Patient
     - Encounter
     - Condition

Delete Resources

	Field Name	Type	Required	Description
🗝️	delete_resources	List<FHIRPath>	Yes	List of FHIRPath to select resources that should be deleted in the bundle, anything selected by a rule will be removed

Example:
This example deletes the Provenance resource:

 - name: testDeleteResources
   delete_resources:
     - Provenance

Execute Script

If the above options aren't sufficient, the Flexporter allows for executing arbitrary JavaScript code against the Bundle. (The ability to load arbitrary libraries is not [yet] supported)

	Field Name	Type	Required	Description
🗝️	execute_script	List	Yes	List of script definitions to execute (in order)

	function	JavaScript code (string)	Yes	Javascript code including the function to be executed. More than one function may be defined in this field at a time
	function_name	string	Yes	Name of the function to be invoked
	apply_to	`"resources"` or `"bundle"`	Yes	Whether the function should be invoked on the "bundle" as a whole, or on the "resources" each individually

Example:
This example executes two scripts. The first adds a dummy profile to the first resource in the Bundle, the second sets the birthDate on any Patient resources in the Bundle to '2022-02-22'.

 - name: testExecuteScript
   execute_script:
     - apply_to: bundle
       function_name: apply
       function: |
          function apply(bundle) {
            bundle['entry'][0]['resource']['meta'] = {profile: ['http://example.com/dummy-profile']}
          }
     - apply_to: resource
       function_name: apply2
       function: |
         function apply2(resource, bundle) {
           if (resource.resourceType == 'Patient') {
             resource.birthDate = '2022-02-22';
           }
         }

Value Functions

Function	Arguments	Description
`getField`	FHIRPath pointing to field	Get value from a field from current or based-on resource
`findValue`	FHIRPath pointing to resource/field	Find a value from another resource within the entire Bundle
`setRef`		Create a reference to the current resource
`findRef`	FHIRPath pointing to resource	Find a reference to another resource in the bundle
`getAttribute`	Attribute key	Read value from a Synthea attribute. Note this can only be used when running in Synthea, not standalone.
`randomCode`	ValueSet URL	Pick a random code from the target ValueSet

Basic naming scheme here is "set" and "get" refer to a particular resource, (ie, the current one, or the resource a new one is being based on) and "find" refers to searching the entire bundle. The syntax for value functions is function([argument1,argument2,...]) where arguments are included without quotes.

Examples

 -  create_resource: 
     - resourceType: ServiceRequest
       based_on: 
         resource: Procedure
       fields:
         - location: ServiceRequest.intent
           value: plan
         - location: ServiceRequest.encounter.reference
           value: $getField([Procedure.encounter.reference])
         - location: ServiceRequest.subject.reference
           value: $findRef([Patient])
         - location: ServiceRequest.code.coding
           value: $randomCode([http://hl7.org/fhir/us/mcode/ValueSet/mcode-laterality-vs])

Value Transforms

https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/org/mitre/synthea/export/flexporter/ValueTransforms.java

Function	Description
`toInstant`	Convert the given date/time into an instant: `YYYY-MM-DDThh:mm:ss.sss+zz:zz`. If the given item doesn't have that level of precision, random values will populate the missing pieces.
`toDate`	Convert the given date/time into a date: `YYYY-MM-DD`. If the given item doesn't have that level of precision, random values will populate the missing pieces.
`toDateTime`	Convert the given date/time into a dateTime: `YYYY-MM-DDThh:mm:ss+zz:zz`. If the given item doesn't have that level of precision, random values will populate the missing pieces.
`toTime`	Convert the given date/time into a time: `hh:mm:ss`. If the given item doesn't have that level of precision, random values will populate the missing pieces.

Full Examples

mCODE

actions:
 - name: Apply Profiles
   profiles:
   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-patient
     applicability: Patient

   - profile: http://example.com/bloodpressure
     applicability: Observation.code.coding.where($this.code = '85354-9')

   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-stage-group
     applicability: Observation.code.coding.where($this.code = '21908-9')

   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-primary-tumor-category
     applicability: Observation.code.coding.where($this.code = '21905-5')

   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-regional-nodes-category
     applicability: Observation.code.coding.where($this.code = '21906-3')

   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-distant-metastases-category
     applicability: Observation.code.coding.where($this.code = '21907-1')


 - name: Create TNM References
   set_values:
     - applicability: Observation.code.coding.where($this.code = '21908-9')
       fields:
         - location: Observation.hasMember.where(display='Tumor Category').reference
           value: $findRef([Observation.code.coding.where($this.code = '21905-5')])
         - location: Observation.hasMember.where(display='Nodes Category').reference
           value: $findRef([Observation.code.coding.where($this.code = '21906-3')])
         - location: Observation.hasMember.where(display='Metastases Category').reference
           value: $findRef([Observation.code.coding.where($this.code = '21907-1')])

Known Limitations

ValueSets can only be loaded from the IG when they are fully defined and expanded within the IG. The existing Synthea terminology support works, but you have to provide your own terminology server.
Resources removed by keep_resources or delete_resources will still be referenced by other resources. Need to define the best approach for what to do here. (Maybe options? "Cascade" = delete resources that reference deleted resources, "dangle" = leave dangling references, "prune field" = remove the reference field but leave the rest of the resource)
Copying objects from one field to another doesn't currently work, only primitives. The workaround for now is to list out all the individual sub-fields, but this doesn't work well when the fields that are populated might change. One hack I tried to implement was to deconstruct the provided object into its component fields, but that didn't work because HAPI doesn't provide a way to serialize generic FHIR objects, only Resources. (Ex, you can't serialize a CodeableConcept) See Actions.createFhirPathMapping https://github.com/synthetichealth/synthea/blob/d08c545d168610df7ba42326477a14e17daaa78c/src/main/java/org/mitre/synthea/export/flexporter/Actions.java#L176
Sometimes YAML needs to be told what an object is. For example things that look like dates will become java Date objects unless explicitly marked as strings:

 - name: testSetValues
   set_values:
     - applicability: Patient
       fields:
         - location: Patient.birthDate
           value: !!str "1987-06-05"

Technical Notes

At the core of the flexporter is the CustomFHIRPathResourceGeneratorR4 class - this is based on a community contribution within HAPI which we've added some additional functionality to. It does some magic using HAPI's internal reflection (on top of Java reflection), so it can be a bear to wrap your head around. Debugging and stepping through examples makes it easier. https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/org/mitre/synthea/export/flexporter/CustomFHIRPathResourceGeneratorR4.java