OMOP Pipeline User Guide - Analyticsphere/ehr-pipeline-documentation GitHub Wiki

• 0.1 Incoming Data Delivery Requirements
• 0.2 Pipeline Configuration Options
• 0.3 Study Site Configuration
• 0.4 Pipeline Operation
• 0.5 Pipeline Logging
• 0.6 OMOP Pipeline Directory Structure

• 1.1 Folder Structure Requirements
• 1.2 Discovery Process
• 1.3 Processing Decision

• 2.1 Processing Parquet Input
• 2.2 Processing CSV and CSV.GZ Input

• 3.1 Table Name Validation
• 3.2 Column Name Validation

• 4.1 Core Normalization Functions
• 4.2 Invalid Row Processing
• 4.3 Connect ID Handling
• 4.4 Normalization Output

• 5.1 Upgrade Determination
• 5.2 Table Handling During Upgrade
• 5.3 Upgrade Implementation

• 7.1 Purpose
• 7.2 Hash Construction
• 7.3 Affected Columns and Excluded Tables

• 8.1 Exclusion Rules
• 8.2 Connect Data Export
• 8.3 Table-Level Filtering Behavior
• 8.4 Connect Reporting Artifacts

• 9.1 Overview and Purpose
• 9.2 Harmonization Stages
• 9.3 Domain-Based Table Reassignment
• 9.4 Consolidation and Primary Key Deduplication
• 9.5 Harmonization Outputs

• 10.1 Task Configuration and Execution Order
• 10.2 SQL Placeholders
• 10.3 Post-Processing Artifacts and Deduplication
• 10.4 Vocabulary Write Protection

• 11.1 Purpose of Derived Tables
• 11.2 Generated Tables
• 11.3 Implementation Approach

• 12.1 Data Loading Sequence

• 13.1 Delivery Report CSV
• 13.2 Data Quality Dashboard (DQD)
• 13.3 Achilles
• 13.4 PASS
• 13.5 Atlas Results Tables
• 13.6 HTML Delivery Report

The OMOP pipeline is an automated workflow that prepares OMOP deliveries for use in BigQuery and then runs downstream analyses and reporting.

The pipeline is made up of three deployed components:

• ccc-omop-file-processor: Python service that validates, standardizes, harmonizes, post-processes, and loads OMOP files (link)
• ccc-omop-analyzer: R-based service that runs DQD, Achilles, PASS, Atlas results setup, and HTML report generation (link)
• ccc-orchestrator: Airflow workflows that coordinate the pipeline run (link)

These components are deployed and maintained separately, but work together to complete a single pipeline run.

The ccc-omop-file-processor and ccc-omop-analyzer components expose API endpoints that perform specific tasks within a pipeline run.

The ccc-orchestrator component calls those endpoints in the correct order and monitors the run as it progresses. That ordered Airflow workflow is called a Directed Acyclic Graph, or DAG. In this guide, "DAG" and "pipeline" may be used interchangeably.

Connect deliveries must use one consistent set of technical choices within a single delivery. File format, CDM version, and date/datetime formatting must be consistent across all files in that delivery. These choices may change in later deliveries.

The pipeline supports OMOP data files that are individual flat files of formats:

• .csv
• .csv.gz
• .parquet

• UTF-8 or ASCII encoded
• Comma-delimited
• RFC 4180-style escaping for quotes, commas, backslashes, and other special characters within field values
• Consistent line endings using either LF or CRLF
• Field values must be single-line values with no embedded line breaks or carriage returns
• Exactly one header row, positioned as the first row

• File names must exactly match the lowercase OMOP table name plus the correct extension (i.e. person.csv)
• Column names should be lowercase and match the OMOP CDM exactly

• Date and datetime values must use one consistent format across all files in a delivery
• ISO 8601 formats are strongly preferred, for example 2024-03-15 and 2024-03-15 14:30:00

• All OMOP tables in a delivery must use the same CDM version
• Deliveries may use OMOP CDM 5.3, 5.3.1, or 5.4

• Every OMOP table that contains person_id must include both the Study ID (generated by the study site) and the Connect ID (generated by the Connect Coordinating Center)
• An additional connect_id column must be included to store the Connect ID
• Exclude participants whose Connect ID is unknown or cannot be provided

Configuration is split across the three pipeline components. At a high level, the available configuration includes:

• ccc-orchestrator: Cloud Composer environment variables that define which processor and analyzer services to call, which Connect dataset to query, and which target OMOP CDM version and target vocabulary version the pipeline should standardize to
• ccc-omop-file-processor: Cloud Run service and job deployment settings, storage behavior, vocabulary file location, logging configuration, and temporary DuckDB storage configuration
• ccc-omop-analyzer: analyzer service and job deployment settings, authentication and secret configuration, artifact output locations, and job inputs for DQD, Achilles, PASS, Atlas results setup, and report generation

For the complete option lists and current variable names, see the README files for each of the individual components.

Site-specific information is stored in dags/dependencies/ehr/config/site_config.yml in the Airflow DAG project. The DAG reads this file to determine where to find deliveries, which OMOP version was delivered, how to parse dates, where to load BigQuery tables, how to query Connect data, and which optional post-processing tasks to run.

The fields currently read by the DAG are shown below:

site:
  'synthea_54':
    display_name: 'Synthea Synthetic Data'
    gcs_bucket: 'synthea_cdm54'
    file_delivery_format: '.csv'
    project_id: 'nih-nci-dceg-connect-dev'
    cdm_bq_dataset: 'synthea_cdm54'
    analytics_bq_dataset: 'synthea_atlas_results'
    cdm_version: '5.4'
    date_format: '%Y-%m-%d'
    datetime_format: '%Y-%m-%d %H:%M:%S'
    overwrite_site_vocab_with_standard: true
    site_connect_id: 13
    post_processing: ['remove_text-to-concept_measurements']

Field definitions:

• display_name: Human-readable label used in reports and analyzer outputs
• gcs_bucket: Bucket name only, without gs://
• file_delivery_format: Which file extension the DAG will look for in the delivery folder
• project_id: GCP project for BigQuery and Cloud Run job execution
• cdm_bq_dataset: BigQuery dataset that receives the OMOP CDM tables. Can be the same as analytics_bq_dataset
• analytics_bq_dataset: BigQuery dataset used by DQD, Achilles, and Atlas results tables. Can be the same as cdm_bq_dataset
• cdm_version: OMOP CDM version delivered by the site
• date_format and datetime_format: Site-specific parsing formats used during normalization
• overwrite_site_vocab_with_standard:
  • true: load the configured target vocabulary into BigQuery and skip site-delivered vocabulary files
  • false: do not load the target vocabulary; site-delivered vocabulary files are loaded if present
• site_connect_id: Site identifier used in the Connect export query
• post_processing: Optional list of post-processing task names to run after vocabulary harmonization. Each name must correspond to a SQL file at reference/sql/post_processing/<task_name>.sql in the file processor repo. Tasks for a given site run in series in the configured order; different sites' tasks run in parallel. Omit the field, leave it empty, or list only empty strings to skip post-processing for the site. See Section 10 for details.

The pipeline follows an API-driven architecture orchestrated by Airflow. The production DAG runs daily and can also be triggered manually from Airflow. A non-exhaustive list of tasks executed by the pipeline, in order, includes:

1. Check processor service health
2. If not already built, create optimized vocabulary files for the target vocabulary
3. Find the latest date-based delivery for each site
4. Query the BigQuery pipeline log table to decide whether each delivery needs processing; if so:
5. Create artifact directories and build file configuration objects
6. Convert incoming files to working Parquet files
7. Validate table names and column names against the delivered OMOP CDM version
8. Normalize data types, fill defaults, and isolate invalid rows
9. Upgrade delivered CDM files to the target CDM version when required
10. Populate or rewrite the cdm_source table for the delivery
11. Globalize natural-key columns (PK and FK) across sites
12. Export Connect participant status and remove rows for patients who do not meet Connect eligibility rules
13. Run eight vocabulary harmonization stages on clinical tables
14. Apply user-curated post-processing tasks (when configured for the site)
15. Generate derived OMOP tables
16. Clear the BigQuery CDM dataset and load harmonized, vocabulary, remaining delivered, and derived tables
17. Run the cleanup stage to load cdm_source and create or rewrite remaining BigQuery tables via the OMOP DDL
18. Generate the delivery report CSV
19. Run DQD, Achilles, and PASS in parallel
20. Create Atlas results tables and generate the interactive HTML delivery report
21. Mark the delivery complete in the pipeline log table

Multiple sites can be processed within a single pipeline run, and multiple files are processed in parallel during runtime.

The pipeline records execution state in three places:

• Airflow task logs
• Cloud Run service and job logs
• a BigQuery logging table managed by the processor service

The BigQuery logging table tracks one row per site + delivery_date. The key states are:

• started: written when get_unprocessed_files begins processing a delivery
• running: refreshed by most downstream tasks while work is in progress
• error: written when a task fails
• completed: written only after the analyzer stage, Atlas table creation, and HTML report generation succeed

The delivery is selected for processing when:

• no log row exists for the site and delivery date, or
• the existing row has status error

The delivery is skipped when the latest log row has status:

• started
• running
• completed

Operational notes:

• Reprocessing a delivery usually means removing or correcting its BigQuery log row and rerunning the workflow
• The pipeline log table is updated through BigQuery DML, so concurrent writes can still cause transient Too many DML statements outstanding against table errors during busy runs

The processor creates the following directory layout under each delivery:

gs://{site_bucket}/{YYYY-MM-DD}/
└── artifacts/
    ├── converted_files/
    │   ├── person.parquet
    │   ├── condition_occurrence.parquet
    │   └── ...
    ├── invalid_rows/
    │   ├── person.parquet
    │   ├── condition_occurrence.parquet
    │   └── ...
    ├── connect_data/
    │   └── participant_status.parquet
    ├── harmonized_files/
    │   ├── condition_occurrence/
    │   │   ├── condition_occurrence_source_target_remap.parquet
    │   │   ├── condition_occurrence_target_remap.parquet
    │   │   ├── condition_occurrence_source_concept_backfill.parquet
    │   │   └── condition_occurrence_domain_check.parquet
    │   └── ...
    ├── omop_etl/
    │   ├── condition_occurrence/
    │   │   └── condition_occurrence.parquet
    │   └── ...
    ├── post_processing/
    │   └── {task_name}/
    │       └── tmp/
    │           └── {table}_pre.parquet
    ├── derived_files/
    │   ├── condition_era.parquet
    │   ├── drug_era.parquet
    │   └── observation_period.parquet
    ├── delivery_report/
    │   ├── delivery_report_{site}_{date}.csv
    │   ├── omop_delivery_report.html
    │   └── tmp/
    │       ├── delivery_report_part_{uuid1}.parquet
    │       └── ...
    ├── dqd/
    │   ├── dqdashboard_results.json
    │   ├── dqdashboard_results.csv
    │   └── errors/
    ├── achilles/
    │   ├── achilles_results.csv
    │   └── results/
    └── pass/
        ├── pass_overall.csv
        ├── pass_table_level.csv
        ├── pass_field_level.csv
        ├── pass_composite_overall.csv
        └── pass_composite_components.csv

Key points:

• converted_files/ is the main working area for file-level processing. Conversion, normalization, CDM upgrade, cdm_source population, natural-key globalization, and Connect filtering all write back into this area.
• harmonized_files/ stores intermediate vocabulary harmonization outputs by source table.
• omop_etl/ stores the final, consolidated harmonized tables that are loaded to BigQuery and used to generate derived tables.
• post_processing/ holds per-task row-identity snapshots used to diff the on-disk state before and after each post-processing task runs. Snapshots are removed at the end of the task.
• derived_files/ stores generated OMOP tables such as condition_era, drug_era, and the standardized observation_period.
• delivery_report/tmp/ stores small Parquet artifacts that are later consolidated into the final CSV report.
• dqd/, achilles/, and pass/ are populated by the analyzer stage, not by the processor stage.

The artifact directory structure is created by the create_artifact_directories processor endpoint. Existing files in those directories are removed before reuse.

The pipeline runs daily to check for new deliveries. Automatic discovery and execution relies on this file and directory structure:

gs://{site_bucket}/{YYYY-MM-DD}/{files}

Where:

• {site_bucket} is the site bucket configured in site_config.yml
• {YYYY-MM-DD} is the delivery folder name
• {files} are the delivered OMOP files using the configured file_delivery_format

Only the most recent top-level folder that parses as YYYY-MM-DD is considered for each site.

The file discovery phase begins in id_sites_to_process and get_unprocessed_files.

id_sites_to_process does the following:

1. Reads the site list from site_config.yml
2. Calls create_optimized_vocab once for the target vocabulary version
3. Finds the latest delivery folder for each site
4. Queries the BigQuery pipeline log table for that site and delivery date
5. Returns only the deliveries that are new or previously failed

get_unprocessed_files then:

1. Writes the started log entry for each selected site delivery
2. Creates the artifact directories in the delivery folder
3. Calls get_file_list with the site bucket, delivery date, and configured file extension
4. Builds one FileConfig object per file, containing information from the site_config.yml file

If there are no deliveries to process, the end_if_all_processed task short-circuits the workflow by skipping the remaining tasks.

Incoming files are standardized into Parquet by the convert_file DAG task, which executes the process_incoming_file processor logic through a Cloud Run job. All converted files are written to artifacts/converted_files/. Note that the note_nlp.offset field has special handling logic in the pipeline, as offset is a reserved keyword.

All converted Parquet files, regardless of incoming data file type:

• use lowercase file names
• use lowercase, cleaned column names
• store all columns as strings at this stage

Incoming Parquet files are validated for readability and then copied into artifacts/converted_files/.

Incoming .csv and .csv.gz files are converted to Parquet using DuckDB. During this conversion, invalid characters, erroneous formatting, and other common CSV issues are corrected.

The pipeline detects the file encoding before reading the CSV, and then attempts to convert to Parquet using strict parsing rules. If that fails, it retries with more permissive DuckDB CSV options:

• store_rejects=True
• ignore_errors=True
• parallel=False

If conversion fails with the permissive rules, the entire task is failed.

Converted Parquet files are validated by the validate_file task. Validation compares the converted file against the delivered OMOP CDM version declared for the site.

Validation creates report artifacts but does not itself transform the file.

The processor derives the OMOP table name from the file name and checks whether that table exists in the CDM schema JSON for the delivered version.

Artifacts are created for:

• valid table names
• invalid table names

If the table name is valid, the processor compares the file's columns to the schema definition for that table.

Artifacts are created for:

• valid column names
• invalid column names
• missing schema columns

These artifacts are stored in delivery_report/tmp/ and later merged into the final delivery report CSV.

The normalize_file task runs the processor's normalization logic through a Cloud Run job. It rewrites the working Parquet file in artifacts/converted_files/ so downstream tasks can assume a consistent schema and consistent data types.

Normalization performs these operations:

1. Data type conversion

   • casts each OMOP column to its target type
   • tries to parse date and datetime columns with the site's configured formats
   • falls back to default values when a required field cannot be parsed

2. Schema completion

   • adds missing OMOP columns
   • fills missing required columns with placeholder defaults
   • fills missing _concept_id columns with 0
   • drops extra columns that are not part of the OMOP table schema

3. Column standardization

   • lowercases column names
   • writes columns in consistent OMOP schema order

4. Primary key generation

   • for tables with surrogate primary keys, builds a deterministic composite key from the other column values
   • Note: primary keys at this stage can still collide when rows are duplicated or moved during harmonization, so a later harmonization step deduplicates them

5. Special-case handling for person.birth_datetime

   • DQD and Achilles require birth_datetime to be populated, but many sites deliver only the integer year_of_birth / month_of_birth / day_of_birth components. The normalizer back-fills birth_datetime so the column is always non-null after this step.
   • When the source file already has a birth_datetime column, the normalizer tries (in order) to parse the value with the site's configured datetime_format, then with DuckDB's default DATETIME cast, and falls through to a calculated value only if both parses fail.
   • The calculated value is CONCAT(year_of_birth, '-', month_of_birth, '-', day_of_birth, ' 00:00:00'), with missing components defaulting to 1900, 01, and 01 respectively. So a row with only year_of_birth = 1985 resolves to 1985-01-01 00:00:00, and a row with no birth fields at all resolves to 1900-01-01 00:00:00.
   • The time component is always 00:00:00 — no timezone is applied.

Normalization also splits rows into valid and invalid groups.

A row is considered invalid when a required field cannot be cast to its appropriate type after applying the pipeline's parsing and fallback logic. In contrast:

• if a required column is missing entirely, the pipeline inserts a default value
• if a required column is present but NULL, the pipeline inserts a default value

Those rows are not treated as invalid.

For invalid rows, the processor:

• writes the rejected rows to artifacts/invalid_rows/{table}.parquet
• removes those rows from the normalized working table

Report artifacts for valid and invalid row counts are created as well.

Normalization also looks for site-delivered Connect identifiers. If any column name contains connectid or connect_id, that column is treated as the authoritative identifier and is used to populate person_id.

This happens in all OMOP tables that contain a person_id column.

After normalization:

• the normalized table overwrites the file in artifacts/converted_files/
• invalid rows are written to artifacts/invalid_rows/. If there are no invalid rows, a file with 0 rows is still created.
• row-count artifacts are written to delivery_report/tmp/

The cdm_upgrade task upgrades delivered OMOP files to the configured target CDM version. In the current pipeline, the supported upgrade path is from 5.3 to 5.4.

For each file:

• if the delivered CDM version already matches the target CDM version, the task does nothing
• otherwise, the processor checks whether that table changed between the two versions

The current 5.3 -> 5.4 handling is:

1. Removed tables

   • attribute_definition is deleted from the processed delivery because it does not exist in CDM 5.4

2. Changed tables

   • Version-specific SQL is applied to these tables:
     • visit_occurrence
     • visit_detail
     • procedure_occurrence
     • device_exposure
     • measurement
     • observation
     • note
     • location
     • metadata
     • cdm_source

3. New tables

   • These are not generated during file upgrade. They are created later in BigQuery during dataset finalization.
     • episode
     • episode_event
     • cohort

4. Unchanged tables

   • tables that are same in both 5.3 and 5.4 pass through without modification

Upgrade SQL scripts are stored under reference/sql/cdm_upgrade/.

The processor:

1. selects the script for the relevant table and version transition
2. runs the SQL against the normalized working Parquet file
3. overwrites the same working Parquet file in artifacts/converted_files/

If the table was removed in the target CDM version, the processor deletes the processed Parquet artifact (not the source file) instead of rewriting it.

After per-file CDM upgrades finish, the populate_cdm_source_file task runs once per site delivery. It ensures every delivery has a well-formed single-row cdm_source.parquet artifact, even when the site did not deliver one. The cdm_source table itself is unrelated to the CDM version upgrade — it is OMOP metadata describing the delivery (source release date, CDM holder, version identifiers, etc.) — and is populated as a separate pipeline step.

The task creates or rewrites cdm_source.parquet when any of the following is true:

• the file does not exist
• the file exists but contains zero rows
• the file exists but contains more than one row

If the file already exists with exactly one row, the task keeps every site-delivered column except source_release_date and cdm_release_date, which it always rewrites:

• source_release_date keeps the site's value when it parses as a valid date; otherwise it falls back to the delivery date
• cdm_release_date is unconditionally set to the delivery date

A "Source system extraction date" report artifact is always written; if the resolved source_release_date cannot be parsed, the artifact falls back to the delivery date.

The task uses the DAG-configured target CDM version and target vocabulary version to populate cdm_version, cdm_version_concept_id, and vocabulary_version consistently with what the pipeline will load to BigQuery.

The globalize_natural_keys task rewrites natural-key columns in each processed Parquet file so values are globally unique across sites. This step runs per file after populate_cdm_source_file and before Connect participant filtering.

Natural keys delivered by sites (for example visit_occurrence_id, provider_id, care_site_id) are only guaranteed to be unique within a single site's source system. When data from multiple sites is later combined or queried together in BigQuery, identifiers from different sites can collide. Globalizing these columns up front guarantees uniqueness across the whole study and removes the need for ad-hoc disambiguation downstream.

For each in-scope column, every non-null value is replaced with a deterministic 64-bit hash:

hash(CONCAT(value, site)) % 9223372036854775807

NULL values are preserved as NULL. The hash shape matches the surrogate-key hash used during normalization, so foreign-key joins across tables continue to work after the rewrite (a visit_occurrence_id minted in visit_occurrence lines up with the same visit_occurrence_id referenced from, e.g., condition_occurrence).

Columns rewritten when present in the file:

• visit_occurrence_id, preceding_visit_occurrence_id
• visit_detail_id, preceding_visit_detail_id, parent_visit_detail_id
• provider_id, care_site_id, location_id
• episode_id

Excluded by policy:

• All OMOP vocabulary tables (concept, vocabulary, domain, concept_class, relationship, concept_relationship, concept_synonym, concept_ancestor, drug_strength)
• The person_id column, in every table where it appears (including person.person_id). The person table itself IS rewritten — its location_id, provider_id, and care_site_id FK columns are globalized along with the parent tables so cross-table joins remain intact.

Tables that contain none of the in-scope columns are skipped with a no-op success.

Data for patients who are not consented and verified Connect participants, or whose participation status prohibits use of their EHR data (i.e. revoked HIPAA authorization, withdrew, and/or requested data destruction), are removed. Connect participant filtering runs after normalization, CDM upgrade, and natural-key globalization so that filtering rules are applied to a standardized, consistent data structure with globally unique identifiers already in place.

Filtering runs once per pipeline execution and uses the participant information that is current at the time of execution. If a participant's status later changes, the pipeline can be rerun against an already processed delivery to apply the updated status. Additional filtering outside of the EHR pipeline is completed prior to data being released for research.

The participant filter removes rows when any one of the following is true:

• the participant's Connect ID is missing, non-numeric, or -1 (the pipeline default value)
• the participant's Connect ID in the EHR data is absent from the Connect BigQuery Participant table for that site
• the participant is not Verified
• the participant has withdrawn consent
• the participant has revoked HIPAA authorization
• the participant has requested data destruction

Internally, the processor applies the following Connect concept ID rules:

• verification status (821247024) must equal 197316935
• exclusion flags for HIPAA revocation, withdrawn consent, and data destruction requests (773707518, 747006172, and 831041022, respectively) are triggered when their concept ID equals 353358909

The retrieve_connect_data task runs once per site delivery and calls the processor's get_connect_data logic. It queries the Connect BigQuery Participant table for the current participant list associated with the site, along with their verification status and participation variables, then writes the result to artifacts/connect_data/participant_status.parquet.

This task also generates report artifacts identifying participants who should be excluded per the rules above, or should be included in the EHR delivery but are missing.

The filter_participants task runs once per file against the working Parquet file in artifacts/converted_files/. Filtering is applied to tables that contain a Connect ID; tables without a Connect ID are not filtered. Tables without a Connect ID contain metadata or vocabulary information - not clinical data.

Rows are retained only if they belong to a patient that can be verified as a Connect participant, and the participant does not meet any exclusion rule.

Report artifacts are generated describing the number of rows removed; artifacts are generated even if 0 rows are removed.

The Connect filtering stage adds report artifacts for:

• counts of rows removed (by table) because the Connect ID was missing or invalid
• counts of rows removed (by table) because the Connect ID in the EHR data was not found in the Connect Participant BigQuery table
• counts of rows removed (by table) because at least one of the participant exclusion rules applied
• a list of Connect IDs present in the OMOP delivery but missing from the Connect Participant BigQuery table
• a list of eligible Connect IDs missing from the delivery

Vocabulary harmonization standardizes clinical concept usage to the configured target vocabulary version. Harmonization applies only to these clinical tables:

• visit_occurrence
• condition_occurrence
• drug_exposure
• procedure_occurrence
• device_exposure
• measurement
• observation
• note
• specimen

Vocabulary harmonization exists because concept meanings, mappings, and domains change across vocabulary releases. The goal is to produce a final dataset whose clinical tables are aligned to one target vocabulary version, even when sites delivered data built against older vocabulary releases.

The vocabulary harmonization process is split into eight stages that run in the production DAG:

1. source_target

   • remaps source concept IDs to updated target mappings. Run per file

2. target_remap

   • remaps non-standard target concepts when a newer standard mapping exists. Run per file

3. source_concept_backfill

   • sets the primary _concept_id to _source_concept_id when the concept ID is zero, the source concept ID is non-zero, and the source concept exists in the vocabulary. Run per file

4. domain_check

   • verifies that the concept domain still matches the OMOP table where the row currently lives, and writes a target_table value to each row. Run per file

5. omop_etl

   • transforms rows into their destination OMOP tables based on target_table. Run per file

6. consolidate_etl

   • merges per-file ETL outputs into one consolidated table per destination OMOP table. Run per delivery

7. discover_tables_for_dedup

   • inspects consolidated ETL outputs and identifies which destination tables need primary key deduplication. Run per delivery

8. deduplicate_single_table

   • rewrites each identified destination table so primary keys are unique. Run per discovered destination table

Two additional steps — target_replacement and secondary_concept_backfill — exist as processor endpoints but are intentionally excluded from the current DAG chain. They may be reintroduced in a future release.

During domain_check, the harmonizer assigns each row a target_table based on the current domain of its harmonized target concept. During omop_etl, rows are written into the destination OMOP table named in target_table.

If the current concept domain still matches the source table, the row stays in that table. If the concept now belongs to a different supported OMOP domain, the row is written to the corresponding destination table. If the domain is unknown or does not map to one of the supported harmonized domains, target_table defaults to the source table.

This means harmonization can create destination tables that were not present in the original delivery. For example:

• a site does not deliver a note table
• a vocabulary update changes some harmonized concepts to the Note domain
• omop_etl writes those rows to the note destination table

As a result, the harmonized output can include a valid note table even though the site did not originally deliver one.

The first five stages operate per source file. After that:

• consolidate_etl merges all destination fragments for one site delivery
• discover_tables_for_dedup writes a temporary table-config JSON file so the DAG can fan out deduplication work
• deduplicate_single_table runs in parallel across the discovered destination tables

These site-level consolidation and table discovery processes are required because domain_check and omop_etl can generate tables that were not in the site's original delivery; these newly generated tables require FileConfig objects in order to be processed by the DAG.

The harmonization stage produces two distinct artifact areas:

• artifacts/harmonized_files/ - per-source intermediate mapping outputs
• artifacts/omop_etl/ - final consolidated destination tables used for: - BigQuery loading - derived table generation - downstream reporting

Post-processing applies user-curated SQL tasks to the on-disk OMOP artifacts after vocabulary harmonization and before derived-table generation. It exists to handle site- or study-specific transformations that fall outside the standardized pipeline stages — for example, removing measurement rows that map to unwanted concepts, or updating values that occur in one or a small number of sites.

A site opts in to post-processing through the post_processing field in site_config.yml. Each entry names a SQL file that must exist at reference/sql/post_processing/<task_name>.sql in the file processor repo. The DAG fails fast if the script is missing.

Execution model:

• Per-site serial. Tasks for a single site run sequentially in the order they appear in site_config.yml. The next task does not start until the previous one returns. This matters because post-processing tasks share the same Parquet artifacts; a later task may depend on the state produced by an earlier one.
• Cross-site parallel. Different sites' chains run independently in parallel.
• Fail-fast within a chain. If any task in a site's chain fails, the remaining tasks for that site are not attempted.
• No-op when unconfigured. Empty or whitespace-only entries are filtered out, so stubs like post_processing: [''] collapse to no work. When no site has any tasks configured, the post-processing task group expands to zero instances and the DAG flows straight through.

Task SQL files use placeholders that the processor expands to the correct Parquet artifact paths at runtime. The available placeholders are:

Derived tables (condition_era, drug_era, observation_period) are not exposed as placeholders, because they are regenerated immediately after post-processing.

For inserts into surrogate-key tables, task authors are expected to mint primary keys using the same canonical hash formula used elsewhere in the pipeline (see Section 7.2), salted with @SITE. Updates are expressed as delete-plus-insert: because primary keys depend on row content, an "updated" row will have a different primary key from the row it replaces, and the post-processing diff correctly reports this as one added and one removed row.

Foreign-key referential integrity across tables is the task author's responsibility. The pipeline does not cascade deletes or repair orphaned references.

Before each task runs, the processor takes a row-identity snapshot of every non-vocabulary OMOP table on disk (the PK column when present; otherwise a content hash). Snapshots are written under artifacts/post_processing/<task_name>/tmp/.

After the task SQL executes, the processor:

• Diffs the post-task state against each snapshot to compute rows added and rows removed per table
• Emits three report artifacts per affected table: rows added, rows removed, and table affected
• Re-runs primary-key deduplication on every affected surrogate-key table, using the same deduplication step that runs at the end of vocabulary harmonization
• Cleans up the snapshot files

Surrogate-key tables are auto-deduplicated. Natural-key and derived tables are not.

Vocabulary Parquet files are treated as read-only. A task that attempts to write to any OMOP vocabulary table (concept, vocabulary, domain, concept_class, relationship, concept_relationship, concept_synonym, concept_ancestor, drug_strength) or to the optimized vocabulary lookup is rejected with a 400 before any DuckDB execution happens. The guard inspects the rendered SQL for COPY ... TO '...<vocab>.parquet' patterns and catches both placeholder-resolved paths and hard-coded paths.

Reading vocabulary via read_parquet('@CONCEPT') and similar inside a SELECT is unaffected — only writes are blocked.

The DAG generates derived OMOP tables after post-processing is complete, so derived tables reflect the final post-processed state of the harmonized clinical tables.

Derived OMOP tables are part of the standard CDM, but they are produced from other OMOP tables rather than loaded directly from source systems. The pipeline generates these according to OHDSI and THEMIS guidelines.

The pipeline generates these tables:

1. condition_era

   • requires condition_occurrence
   • groups related condition records into eras

2. drug_era

   • requires drug_exposure
   • groups drug exposures into eras

3. observation_period

   • requires person, visit_occurrence, and death
   • is always standardized by the pipeline, even when a site delivered its own observation_period

observation_period uses one of three SQL paths:

• visit_occurrence plus death, if both are present
• visit_occurrence only, if death is absent
• if neither table is present, a generic observation period is created

Because load_derived_tables runs after load_remaining, the generated observation_period replaces any site-delivered observation_period table in BigQuery.

Derived table generation:

1. checks whether the required source tables exist
2. reads harmonized tables from artifacts/omop_etl/ when the source table was vocabulary-harmonized
3. reads working files from artifacts/converted_files/ or artifacts/omop_etl/
4. executes the relevant SQL script from reference/sql/derived_tables/
5. writes the result to artifacts/derived_files/{table}.parquet

Current implementation details:

• drug_era uses a two-part SQL flow because it is more resource-intensive than the other derived tables
• when a required source table is missing, the processor logs a warning and skips writing that derived table. The task does not fail.

The BigQuery load order is:

1. prepare_bq

   • deletes all tables in the site's CDM dataset before loading the new delivery

2. load_harmonized_tables

   • loads consolidated artifacts/omop_etl/ tables to BigQuery
   • skips cleanly when no harmonized tables were created for the delivery

3. load_target_vocab

   • loads target vocabulary tables only when overwrite_site_vocab_with_standard is true

4. load_remaining

   • loads the remaining processed Parquet files from artifacts/converted_files/
   • skips:
     • vocabulary tables when standard vocabulary loading is enabled
     • clinical tables already loaded from omop_etl/
     • cdm_source, which is loaded later

5. load_derived_tables

   • loads all Parquet files in artifacts/derived_files/
   • replaces same-named tables already in BigQuery, including observation_period

6. cleanup

   • creates missing OMOP tables via the OMOP DDL (including 5.4-only tables such as episode, episode_event, and cohort) and rewrites existing tables in place to cast date and datetime columns to the expected BigQuery types
   • then loads cdm_source.parquet

After cleanup, the DAG generates the delivery report CSV and then starts the analyzer phase.

After the CDM dataset is finalized in BigQuery, the pipeline produces two kinds of reporting outputs:

• a delivery-level CSV generated by ccc-omop-file-processor
• downstream analyzer outputs generated by the separately deployed ccc-omop-analyzer

The analyzer phase uses a small set of R packages, including OHDSI DataQualityDashboard, OHDSI Achilles, PASS, and omopDeliveryReport. It also creates the BigQuery results tables needed for OHDSI ATLAS.

Analyzer outputs land in two places — GCS (for files) and BigQuery (for queryable tables):

DQD, Achilles, and PASS run in parallel after the delivery report CSV completes. Both Atlas results table creation and HTML report generation depend on all three finishing, so a failure in any of the three blocks both downstream steps.

After cleanup, the orchestrator's reporting task group produces the final delivery report CSV in artifacts/delivery_report/. This step is executed by ccc-omop-file-processor. The remaining outputs in this section are produced by ccc-omop-analyzer.

The reporting task group runs in two layers, both backed by the file processor's /generate_delivery_report_csv endpoint:

1. generate_report_artifact runs in parallel, once per (site, artifact_type) combination, with each instance generating a single section of the report from the temporary Parquet artifacts written throughout pipeline execution.
2. consolidate_report runs once per site after all artifact sections complete, and assembles them into the final CSV.

The CSV serves as a structured summary of the delivery and is one input to the final HTML report.

The final file name is:

artifacts/delivery_report/delivery_report_{site}_{delivery_date}.csv

After the report CSV is created, the DAG triggers the analyzer job ccc-omop-analyzer-dqd-job to run OHDSI DataQualityDashboard.

DQD runs roughly 2,500 standardized data quality checks against the finalized OMOP dataset and writes these artifacts to GCS:

• artifacts/dqd/dqdashboard_results.json
• artifacts/dqd/dqdashboard_results.csv
• artifacts/dqd/errors/*.txt when DQD produces error files

The dqdashboard_results.csv output is also written to the analytics BigQuery dataset as a table named dqdashboard_results.

The DAG also triggers ccc-omop-analyzer-achilles-job to run OHDSI Achilles.

Achilles produces database characterization outputs and writes results tables to the analytics BigQuery dataset. These tables are what the Atlas results table creation step (§13.5) reads from, and are also consumed directly by ATLAS and related OHDSI tooling.

The Achilles job also generates these artifacts in GCS:

• artifacts/achilles/achilles_results.csv
• artifacts/achilles/results/**/*.json

The DAG also triggers ccc-omop-analyzer-pass-job to run PASS.

PASS evaluates data fitness-for-purpose across six evidence-based dimensions: accessibility, provenance, standards, concept diversity, source diversity, and temporal coverage.

Current PASS outputs are written to artifacts/pass/:

• pass_composite_overall.csv — single weighted composite score for the delivery; the headline number when summarizing PASS results
• pass_composite_components.csv — per-metric contributions to the composite
• pass_overall.csv — scores by metric with 95% confidence intervals
• pass_table_level.csv — scores by table
• pass_field_level.csv — field-level detail

Once DQD, Achilles, and PASS finish successfully, the analyzer service endpoint /create_atlas_results_tables creates additional BigQuery tables used by OHDSI ATLAS. These tables are derived from the Achilles output written in §13.3, which is why Atlas table creation depends on the Achilles job completing.

After the Atlas results tables are created, the analyzer service endpoint /generate_delivery_report uses omopDeliveryReport to combine the following into a single HTML delivery report:

• the delivery report CSV
• the DQD results
• the PASS outputs

The output is written to:

artifacts/delivery_report/omop_delivery_report.html

After both analyzer service calls complete, mark_delivery_complete writes the completed status to the pipeline log table.