9. DataSync - CCRGeneticsBranch/oncogenomics GitHub Wiki

Data sync

Description of how Oncogenomics syncs data daily

On Biowulf

  • At 23:00 daily, a cron job script /data/khanlab/projects/updatedSync/check_processed_runs.sh is launched to gather all newly processed cases from pipeline.
  • The path of project location is defined in /data/khanlab/projects/updatedSync/projects.txt
Project Location
compass_tso500 /data/Compass/Analysis/ProcessedResults_NexSeq/OncoPilot/
compass_exome /data/Compass/Analysis/ProcessedResults_NexSeq/ExomeRNA_Results/
khanlab3 /data/khanlab3/processed_DATA/
clinomics /data/khanlab2/ProcessedResults_TGEN/
wulab /data/Wulab/processed_DATA/
Acc_19 /data/Acc_19/processed_DATA/
BeatCC /data/khanlab3/BeatCC/processed_DATA/
processed_DATA /data/khanlab/projects/processed_DATA/
roper /data/RoperData/processed_DATA/
alex /data/AlexanderP3/processed_DATA/
gb /data/khanlab3/gb_omics/processed_DATA/
k2_data /data/khanlab3/k2_data_tmp/
khanlab2 /data/khanlab2/processed_DATA/
NF_benchmarking /data/khanlab2/NF_benchmarking/results.processed_data/
GBNCI /data/GBNCI/projects/Sanam_Tumor-normal_processed_data/

  • For each project, three files will be generated in /data/khanlab/projects/updatedSync/update_list2:

    1. new_list_[project name].txt
    2. today_list_[project name].txt
    3. yesterday_list_[project name].txt

  • The logic of check_processed_runs.sh:

    1. Copy today_list.txt to yesterday list.txt
    2. Save timestamp (with format "Y") of all successful.txt and failed_delete.txt to today_list.txt using stat command
    3. Save the difference between today_list.txt and yesterday list.txt to new_list.txt

On Frederick server (fsabcl-onc03d)

  • At 2:00 daily, a cron job script "/var/www/html/clinomics/app/scripts/backend/syncProcessedResults.sh all db all" is launched to sync and upload the data to database.

    • Sync "new_list.txt" on biowulf (defined in /var/www/html/clinomics/app/scripts/backend/project_mapping.txt) to /var/www/html/onco.data/ProcessedResults/update_list
    • For each case, sync data to /var/www/html/onco.data/ProcessedResults/
    • Run script /var/www/html/clinomics/app/scripts/backend/deleteCase.pl to delete cases with "failed_delete.txt"
    • Run script /var/www/html/clinomics/app/scripts/backend/loadVarPatients.pl to upload cases with "successful.txt" to database
    • Run script /var/www/html/clinomics/app/scripts/backend/refreshViews.pl to refresh materialized views on Oracle database
    • Run script /var/www/html/clinomics/app/scripts/backend/updateVarCases.pl to determine the case name (defined in master file) for the processed cases

  • At 5:00 daily, a cron job script "/var/www/html/clinomics/app/scripts/backend/syncProcessedResults.sh all bam all" is launched to sync BAM files.

  • At 10:00 daily, a cron job script "/var/www/html/clinomics/app/scripts/backend/syncProcessedResults.sh all tier all" is launched to update variants tiering based on AVIA annotation. (This should be launched after AVIA processing is finished)