Step 6 Evaluation Against the Truth Set - alyrcoun/EPP622-2026-SNPCalling GitHub Wiki
Analysis Overview:
Now that we have performed variant calling and filtering, it's time to evaluate SNPs.
Navigate to the bcftools_results directory and create symbolic links to the python evaluation script and ground-truth VCF file
cd /lustre/isaac24/proj/UTK0505/alyrcoun/epp_622/test2/test2_analyses/bcftools_results
# Create symbolic links for evaluate_snps_v4.py and truth_snps.vcf
ln -s /lustre/isaac24/proj/UTK0505/test2/check/evaluate_snps_v4.py .
ln -s /lustre/isaac24/proj/UTK0505/test2/check/truth_snps.vcf .
# 1. Run evaluation on raw variant calls
# python3 runs Python and tells it to execute a custom script
# --truth truth_snps.vcf specifies the benchmark dataset containing known ground-truth SNPS
# --calls raw_calls.vcf.gz points to the target VCF file being benchmarked (gentle and strict in place of raw for those types)
# --all-samples tells the script to evaluate the metrics for every individual sample in the VCF simultaneously
# --tag raw(or strict or gentle) assigns a prefix label to the output file so it's easier to identify type
# --outdir eval_raw (or strict or gentle) creates a dedicated folder to organize and store the evaluation summaries, making sure files aren't overwritten
python3 evaluate_snps_v4.py \
--truth truth_snps.vcf \
--calls raw_calls.vcf.gz \
--all-samples \
--tag raw \
--outdir eval_raw
# 2. Run evaluation on gentle filtered calls
python3 evaluate_snps_v4.py \
--truth truth_snps.vcf \
--calls filtered_gentle.vcf.gz \
--all-samples \
--tag gentle \
--outdir eval_gentle
# 3. Run evaluation on stringent filtered calls
python3 evaluate_snps_v4.py \
--truth truth_snps.vcf \
--calls filtered_stringent.vcf.gz \
--all-samples \
--tag strict \
--outdir eval_strict
Directory structure
Run tree starting at the beginning of the entire analysis to document a tidy pipeline layout:
cd /lustre/isaac24/proj/UTK0505/alyrcoun/epp_622/test2/test2_analyses
tree .
# Output of tree command:
├── bcftools_results
│ ├── Sample_A.sorted.bam -> ../bwamem2_results/Sample_A.sorted.bam
│ ├── Sample_A.sorted.bam.bai -> ../bwamem2_results/Sample_A.sorted.bam.bai
│ ├── Sample_B.sorted.bam -> ../bwamem2_results/Sample_B.sorted.bam
│ ├── Sample_B.sorted.bam.bai -> ../bwamem2_results/Sample_B.sorted.bam.bai
│ ├── Sample_C.sorted.bam -> ../bwamem2_results/Sample_C.sorted.bam
│ ├── Sample_C.sorted.bam.bai -> ../bwamem2_results/Sample_C.sorted.bam.bai
│ ├── Sample_D.sorted.bam -> ../bwamem2_results/Sample_D.sorted.bam
│ ├── Sample_D.sorted.bam.bai -> ../bwamem2_results/Sample_D.sorted.bam.bai
│ ├── bam_list.txt
│ ├── eval_gentle
│ │ ├── Sample_A_gentle_false_negatives.tsv
│ │ ├── Sample_A_gentle_false_positives.tsv
│ │ ├── Sample_A_gentle_genotype_confusion.tsv
│ │ ├── Sample_B_gentle_false_negatives.tsv
│ │ ├── Sample_B_gentle_false_positives.tsv
│ │ ├── Sample_B_gentle_genotype_confusion.tsv
│ │ ├── Sample_C_gentle_false_negatives.tsv
│ │ ├── Sample_C_gentle_false_positives.tsv
│ │ ├── Sample_C_gentle_genotype_confusion.tsv
│ │ ├── Sample_D_gentle_false_negatives.tsv
│ │ ├── Sample_D_gentle_false_positives.tsv
│ │ ├── Sample_D_gentle_genotype_confusion.tsv
│ │ └── summary_all_runs.tsv
│ ├── eval_raw
│ │ ├── Sample_A_raw_false_negatives.tsv
│ │ ├── Sample_A_raw_false_positives.tsv
│ │ ├── Sample_A_raw_genotype_confusion.tsv
│ │ ├── Sample_B_raw_false_negatives.tsv
│ │ ├── Sample_B_raw_false_positives.tsv
│ │ ├── Sample_B_raw_genotype_confusion.tsv
│ │ ├── Sample_C_raw_false_negatives.tsv
│ │ ├── Sample_C_raw_false_positives.tsv
│ │ ├── Sample_C_raw_genotype_confusion.tsv
│ │ ├── Sample_D_raw_false_negatives.tsv
│ │ ├── Sample_D_raw_false_positives.tsv
│ │ ├── Sample_D_raw_genotype_confusion.tsv
│ │ └── summary_all_runs.tsv
│ ├── eval_strict
│ │ ├── Sample_A_strict_false_negatives.tsv
│ │ ├── Sample_A_strict_false_positives.tsv
│ │ ├── Sample_A_strict_genotype_confusion.tsv
│ │ ├── Sample_B_strict_false_negatives.tsv
│ │ ├── Sample_B_strict_false_positives.tsv
│ │ ├── Sample_B_strict_genotype_confusion.tsv
│ │ ├── Sample_C_strict_false_negatives.tsv
│ │ ├── Sample_C_strict_false_positives.tsv
│ │ ├── Sample_C_strict_genotype_confusion.tsv
│ │ ├── Sample_D_strict_false_negatives.tsv
│ │ ├── Sample_D_strict_false_positives.tsv
│ │ ├── Sample_D_strict_genotype_confusion.tsv
│ │ └── summary_all_runs.tsv
│ ├── evaluate_snps_v4.py -> /lustre/isaac24/proj/UTK0505/test2/check/evaluate_snps_v4.py
│ ├── filtered_MQ40_DP10.stats.txt
│ ├── filtered_MQ40_DP10.vcf.gz
│ ├── filtered_gentle.stats.txt
│ ├── filtered_gentle.vcf.gz
│ ├── filtered_gentle.vcf.gz.csi
│ ├── filtered_stringent.stats.txt
│ ├── filtered_stringent.vcf.gz
│ ├── filtered_stringent.vcf.gz.csi
│ ├── raw_calls.stats.txt
│ ├── raw_calls.vcf.gz
│ ├── sacCer3.fa -> ../bwamem2_results/sacCer3.fa
│ ├── sacCer3.fa.fai
│ └── truth_snps.vcf -> /lustre/isaac24/proj/UTK0505/test2/check/truth_snps.vcf
├── bwamem2_results
│ ├── Sample_A.coverage.txt
│ ├── Sample_A.flagstat.txt
│ ├── Sample_A.sorted.bam
│ ├── Sample_A.sorted.bam.bai
│ ├── Sample_A.stats.txt
│ ├── Sample_A_R1.trimmed.fastq.gz -> ../fastp_results/fastp_trimmed/Sample_A_R1.trimmed.fastq.gz
│ ├── Sample_A_R2.trimmed.fastq.gz -> ../fastp_results/fastp_trimmed/Sample_A_R2.trimmed.fastq.gz
│ ├── Sample_B.coverage.txt
│ ├── Sample_B.flagstat.txt
│ ├── Sample_B.sorted.bam
│ ├── Sample_B.sorted.bam.bai
│ ├── Sample_B.stats.txt
│ ├── Sample_B_R1.trimmed.fastq.gz -> ../fastp_results/fastp_trimmed/Sample_B_R1.trimmed.fastq.gz
│ ├── Sample_B_R2.trimmed.fastq.gz -> ../fastp_results/fastp_trimmed/Sample_B_R2.trimmed.fastq.gz
│ ├── Sample_C.coverage.txt
│ ├── Sample_C.flagstat.txt
│ ├── Sample_C.sorted.bam
│ ├── Sample_C.sorted.bam.bai
│ ├── Sample_C.stats.txt
│ ├── Sample_C_R1.trimmed.fastq.gz -> ../fastp_results/fastp_trimmed/Sample_C_R1.trimmed.fastq.gz
│ ├── Sample_C_R2.trimmed.fastq.gz -> ../fastp_results/fastp_trimmed/Sample_C_R2.trimmed.fastq.gz
│ ├── Sample_D.coverage.txt
│ ├── Sample_D.flagstat.txt
│ ├── Sample_D.sorted.bam
│ ├── Sample_D.sorted.bam.bai
│ ├── Sample_D.stats.txt
│ ├── Sample_D_R1.trimmed.fastq.gz -> ../fastp_results/fastp_trimmed/Sample_D_R1.trimmed.fastq.gz
│ ├── Sample_D_R2.trimmed.fastq.gz -> ../fastp_results/fastp_trimmed/Sample_D_R2.trimmed.fastq.gz
│ ├── multiqc_data
│ │ ├── llms-full.txt
│ │ ├── multiqc.log
│ │ ├── multiqc.parquet
│ │ ├── multiqc_citations.txt
│ │ ├── multiqc_data.json
│ │ ├── multiqc_general_stats.txt
│ │ ├── multiqc_samtools_coverage.txt
│ │ ├── multiqc_samtools_flagstat.txt
│ │ ├── multiqc_samtools_stats.txt
│ │ ├── multiqc_software_versions.txt
│ │ ├── multiqc_sources.txt
│ │ ├── samtools-coverage-table.txt
│ │ ├── samtools-coverage_BQ.txt
│ │ ├── samtools-coverage_Bases.txt
│ │ ├── samtools-coverage_Coverage.txt
│ │ ├── samtools-coverage_MQ.txt
│ │ ├── samtools-coverage_Mean_depth.txt
│ │ ├── samtools-coverage_Reads.txt
│ │ ├── samtools-flagstat-pct-table.txt
│ │ ├── samtools-flagstat-table.txt
│ │ ├── samtools-stats-dp.txt
│ │ ├── samtools_alignment_plot.txt
│ │ └── samtools_insert_size.txt
│ ├── multiqc_report.html
│ ├── run_bwa.sh
│ ├── sacCer3.fa -> /lustre/isaac24/proj/UTK0505/test2/ref/sacCer3.fa
│ ├── sacCer3.fa.0123 -> /lustre/isaac24/proj/UTK0505/test2/ref/sacCer3.fa.0123
│ ├── sacCer3.fa.amb -> /lustre/isaac24/proj/UTK0505/test2/ref/sacCer3.fa.amb
│ ├── sacCer3.fa.ann -> /lustre/isaac24/proj/UTK0505/test2/ref/sacCer3.fa.ann
│ ├── sacCer3.fa.bwt.2bit.64 -> /lustre/isaac24/proj/UTK0505/test2/ref/sacCer3.fa.bwt.2bit.64
│ ├── sacCer3.fa.fai -> /lustre/isaac24/proj/UTK0505/test2/ref/sacCer3.fa.fai
│ ├── sacCer3.fa.pac -> /lustre/isaac24/proj/UTK0505/test2/ref/sacCer3.fa.pac
│ └── stats.sh
├── eval
├── fastp_results
│ ├── Sample_A.fastp.html
│ ├── Sample_A.fastp.json
│ ├── Sample_A_R1.fastq.gz -> ../../raw_data_test2/Sample_A_R1.fastq.gz
│ ├── Sample_A_R2.fastq.gz -> ../../raw_data_test2/Sample_A_R2.fastq.gz
│ ├── Sample_B.fastp.html
│ ├── Sample_B.fastp.json
│ ├── Sample_B_R1.fastq.gz -> ../../raw_data_test2/Sample_B_R1.fastq.gz
│ ├── Sample_B_R2.fastq.gz -> ../../raw_data_test2/Sample_B_R2.fastq.gz
│ ├── Sample_C.fastp.html
│ ├── Sample_C.fastp.json
│ ├── Sample_C_R1.fastq.gz -> ../../raw_data_test2/Sample_C_R1.fastq.gz
│ ├── Sample_C_R2.fastq.gz -> ../../raw_data_test2/Sample_C_R2.fastq.gz
│ ├── Sample_D.fastp.html
│ ├── Sample_D.fastp.json
│ ├── Sample_D_R1.fastq.gz -> ../../raw_data_test2/Sample_D_R1.fastq.gz
│ ├── Sample_D_R2.fastq.gz -> ../../raw_data_test2/Sample_D_R2.fastq.gz
│ ├── fastp_trimmed
│ │ ├── Sample_A_R1.trimmed.fastq.gz
│ │ ├── Sample_A_R2.trimmed.fastq.gz
│ │ ├── Sample_B_R1.trimmed.fastq.gz
│ │ ├── Sample_B_R2.trimmed.fastq.gz
│ │ ├── Sample_C_R1.trimmed.fastq.gz
│ │ ├── Sample_C_R2.trimmed.fastq.gz
│ │ ├── Sample_D_R1.trimmed.fastq.gz
│ │ └── Sample_D_R2.trimmed.fastq.gz
│ ├── run1_fastp.sh
│ ├── run2_fastp.sh
│ ├── run_fastp.sh
│ └── trimmed_fastqc
│ ├── Sample_A_R1.trimmed_fastqc.html
│ ├── Sample_A_R1.trimmed_fastqc.zip
│ ├── Sample_A_R2.trimmed_fastqc.html
│ ├── Sample_A_R2.trimmed_fastqc.zip
│ ├── Sample_B_R1.trimmed_fastqc.html
│ ├── Sample_B_R1.trimmed_fastqc.zip
│ ├── Sample_B_R2.trimmed_fastqc.html
│ ├── Sample_B_R2.trimmed_fastqc.zip
│ ├── Sample_C_R1.trimmed_fastqc.html
│ ├── Sample_C_R1.trimmed_fastqc.zip
│ ├── Sample_C_R2.trimmed_fastqc.html
│ ├── Sample_C_R2.trimmed_fastqc.zip
│ ├── Sample_D_R1.trimmed_fastqc.html
│ ├── Sample_D_R1.trimmed_fastqc.zip
│ ├── Sample_D_R2.trimmed_fastqc.html
│ ├── Sample_D_R2.trimmed_fastqc.zip
│ ├── multiqc_data
│ │ ├── fastqc-status-check-heatmap.txt
│ │ ├── fastqc_adapter_content_plot.txt
│ │ ├── fastqc_per_base_n_content_plot.txt
│ │ ├── fastqc_per_base_sequence_quality_plot.txt
│ │ ├── fastqc_per_sequence_gc_content_plot_Counts.txt
│ │ ├── fastqc_per_sequence_gc_content_plot_Percentages.txt
│ │ ├── fastqc_per_sequence_quality_scores_plot.txt
│ │ ├── fastqc_sequence_counts_plot.txt
│ │ ├── fastqc_sequence_duplication_levels_plot.txt
│ │ ├── fastqc_sequence_length_distribution_plot.txt
│ │ ├── llms-full.txt
│ │ ├── multiqc.log
│ │ ├── multiqc.parquet
│ │ ├── multiqc_citations.txt
│ │ ├── multiqc_data.json
│ │ ├── multiqc_fastqc.txt
│ │ ├── multiqc_general_stats.txt
│ │ ├── multiqc_software_versions.txt
│ │ └── multiqc_sources.txt
│ └── multiqc_report.html
└── fastqc_results
├── Sample_A_R1.fastq.gz -> /lustre/isaac24/proj/UTK0505/test2/reads/Sample_A_R1.fastq.gz
├── Sample_A_R1_fastqc.html
├── Sample_A_R1_fastqc.zip
├── Sample_A_R2.fastq.gz -> /lustre/isaac24/proj/UTK0505/test2/reads/Sample_A_R2.fastq.gz
├── Sample_A_R2_fastqc.html
├── Sample_A_R2_fastqc.zip
├── Sample_B_R1.fastq.gz -> /lustre/isaac24/proj/UTK0505/test2/reads/Sample_B_R1.fastq.gz
├── Sample_B_R1_fastqc.html
├── Sample_B_R1_fastqc.zip
├── Sample_B_R2.fastq.gz -> /lustre/isaac24/proj/UTK0505/test2/reads/Sample_B_R2.fastq.gz
├── Sample_B_R2_fastqc.html
├── Sample_B_R2_fastqc.zip
├── Sample_C_R1.fastq.gz -> /lustre/isaac24/proj/UTK0505/test2/reads/Sample_C_R1.fastq.gz
├── Sample_C_R1_fastqc.html
├── Sample_C_R1_fastqc.zip
├── Sample_C_R2.fastq.gz -> /lustre/isaac24/proj/UTK0505/test2/reads/Sample_C_R2.fastq.gz
├── Sample_C_R2_fastqc.html
├── Sample_C_R2_fastqc.zip
├── Sample_D_R1.fastq.gz -> /lustre/isaac24/proj/UTK0505/test2/reads/Sample_D_R1.fastq.gz
├── Sample_D_R1_fastqc.html
├── Sample_D_R1_fastqc.zip
├── Sample_D_R2.fastq.gz -> /lustre/isaac24/proj/UTK0505/test2/reads/Sample_D_R2.fastq.gz
├── Sample_D_R2_fastqc.html
├── Sample_D_R2_fastqc.zip
├── link_data.sh
├── multiqc_data
│ ├── fastqc-status-check-heatmap.txt
│ ├── fastqc_adapter_content_plot.txt
│ ├── fastqc_overrepresented_sequences_plot.txt
│ ├── fastqc_per_base_n_content_plot.txt
│ ├── fastqc_per_base_sequence_quality_plot.txt
│ ├── fastqc_per_sequence_gc_content_plot_Counts.txt
│ ├── fastqc_per_sequence_gc_content_plot_Percentages.txt
│ ├── fastqc_per_sequence_quality_scores_plot.txt
│ ├── fastqc_sequence_counts_plot.txt
│ ├── fastqc_sequence_duplication_levels_plot.txt
│ ├── fastqc_top_overrepresented_sequences_table.txt
│ ├── llms-full.txt
│ ├── multiqc.log
│ ├── multiqc.parquet
│ ├── multiqc_citations.txt
│ ├── multiqc_data.json
│ ├── multiqc_fastqc.txt
│ ├── multiqc_general_stats.txt
│ ├── multiqc_software_versions.txt
│ └── multiqc_sources.txt
└── multiqc_report.html
Results
| Sample | Filter | TP | FP | FN | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|
| A | raw | 4,875 | 0 | 125 | 1.0000 | 0.9750 | 0.9873 |
| A | gentle | 4,864 | 0 | 136 | 1.0000 | 0.9728 | 0.9862 |
| A | strict | 4,772 | 0 | 228 | 1.0000 | 0.9544 | 0.9767 |
| B | raw | 4,422 | 23 | 578 | 0.9948 | 0.8844 | 0.9364 |
| B | gentle | 4,419 | 0 | 581 | 1.0000 | 0.8838 | 0.9383 |
| B | strict | 4,347 | 0 | 653 | 1.0000 | 0.8694 | 0.9301 |
| C | raw | 4,871 | 2 | 129 | 0.9996 | 0.9742 | 0.9867 |
| C | gentle | 4,865 | 0 | 135 | 1.0000 | 0.9730 | 0.9863 |
| C | strict | 4,772 | 0 | 228 | 1.0000 | 0.9544 | 0.9767 |
| D | raw | 4,859 | 15 | 141 | 0.9969 | 0.9718 | 0.9842 |
| D | gentle | 4,851 | 1 | 149 | 0.9998 | 0.9702 | 0.9848 |
| D | strict | 4,761 | 0 | 239 | 1.0000 | 0.9522 | 0.9755 |
In the raw calls, Sample B, C, and D had a lot of false positives. Gentle filtering removed 100% of those in Sample B and C but not all in Sample D. Plus it didn't cost as many real variants (FNs only increased slightly). Strict filtering threw away more than 90 real true variants per sample and did not remove any extra false positives because gentle filtering already cleaned them up to 100% precision.
Gentle filtering is the best choice if you want to be precise without unneccessarily sacrificing recall.
Results Output
====================================================================== Sample_A_raw (VCF column: Sample_A)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,875 false positives 0 false negatives 125 precision 1.0000 recall 0.9750 F1 0.9873 recall (het) 0.9739 <- where low depth bites recall (hom) 0.9765
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9990 (4,870/4,875) ref dropout 5 true het called 1/1 (detected, genotyped WRONG) allelic dropout 77 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_raw/Sample_A_raw_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 125 of which HARD sites 1 (0.2% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,920 hom-ref genotypes 38 this sample was 0/0 here missing genotypes 7 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_raw/Sample_A_raw_false_negatives.tsv
FP table -> eval_raw/Sample_A_raw_false_positives.tsv
====================================================================== Sample_B_raw (VCF column: Sample_B)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,422 false positives 23 false negatives 578 precision 0.9948 recall 0.8844 F1 0.9364 recall (het) 0.8375 <- where low depth bites recall (hom) 0.9521
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9156 (4,049/4,422) ref dropout 373 true het called 1/1 (detected, genotyped WRONG) allelic dropout 480 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_raw/Sample_B_raw_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 578 of which HARD sites 59 (11.8% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,920 hom-ref genotypes 366 this sample was 0/0 here missing genotypes 109 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_raw/Sample_B_raw_false_negatives.tsv
FP table -> eval_raw/Sample_B_raw_false_positives.tsv
====================================================================== Sample_C_raw (VCF column: Sample_C)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,871 false positives 2 false negatives 129 precision 0.9996 recall 0.9742 F1 0.9867 recall (het) 0.9729 <- where low depth bites recall (hom) 0.9761
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9984 (4,863/4,871) ref dropout 8 true het called 1/1 (detected, genotyped WRONG) allelic dropout 80 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_raw/Sample_C_raw_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 129 of which HARD sites 4 (0.8% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,920 hom-ref genotypes 35 this sample was 0/0 here missing genotypes 12 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_raw/Sample_C_raw_false_negatives.tsv
FP table -> eval_raw/Sample_C_raw_false_positives.tsv
====================================================================== Sample_D_raw (VCF column: Sample_D)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,859 false positives 15 false negatives 141 precision 0.9969 recall 0.9718 F1 0.9842 recall (het) 0.9695 <- where low depth bites recall (hom) 0.9751
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9975 (4,847/4,859) ref dropout 12 true het called 1/1 (detected, genotyped WRONG) allelic dropout 90 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_raw/Sample_D_raw_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 141 of which HARD sites 5 (1.0% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,920 hom-ref genotypes 39 this sample was 0/0 here missing genotypes 7 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_raw/Sample_D_raw_false_negatives.tsv
FP table -> eval_raw/Sample_D_raw_false_positives.tsv
====================================================================== Summary appended to eval_raw/summary_all_runs.tsv
Report in your wiki, per sample: precision, recall, F1 (detection) recall_het vs recall_hom (where depth actually hurts) gt_concordance (genotyping) ref_dropout, allelic_dropout (the two ways a het goes wrong) indels_this_sample (artifacts -- explain the pattern)
(snps) [alyrcoun@sphinx bcftools_results]$ python3 evaluate_snps_v4.py \
--truth truth_snps.vcf \ --calls filtered_gentle.vcf.gz \ --all-samples \ --tag gentle \ --outdir eval_gentle
====================================================================== Sample_A_gentle (VCF column: Sample_A)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,864 false positives 0 false negatives 136 precision 1.0000 recall 0.9728 F1 0.9862 recall (het) 0.9712 <- where low depth bites recall (hom) 0.9751
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9998 (4,863/4,864) ref dropout 1 true het called 1/1 (detected, genotyped WRONG) allelic dropout 85 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_gentle/Sample_A_gentle_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 136 of which HARD sites 3 (0.6% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,869 hom-ref genotypes 5 this sample was 0/0 here missing genotypes 0 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_gentle/Sample_A_gentle_false_negatives.tsv
FP table -> eval_gentle/Sample_A_gentle_false_positives.tsv
====================================================================== Sample_B_gentle (VCF column: Sample_B)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,419 false positives 0 false negatives 581 precision 1.0000 recall 0.8838 F1 0.9383 recall (het) 0.8365 <- where low depth bites recall (hom) 0.9521
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9158 (4,047/4,419) ref dropout 372 true het called 1/1 (detected, genotyped WRONG) allelic dropout 483 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_gentle/Sample_B_gentle_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 581 of which HARD sites 59 (11.8% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,869 hom-ref genotypes 354 this sample was 0/0 here missing genotypes 96 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_gentle/Sample_B_gentle_false_negatives.tsv
FP table -> eval_gentle/Sample_B_gentle_false_positives.tsv
====================================================================== Sample_C_gentle (VCF column: Sample_C)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,865 false positives 0 false negatives 135 precision 1.0000 recall 0.9730 F1 0.9863 recall (het) 0.9716 <- where low depth bites recall (hom) 0.9751
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9986 (4,858/4,865) ref dropout 7 true het called 1/1 (detected, genotyped WRONG) allelic dropout 84 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_gentle/Sample_C_gentle_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 135 of which HARD sites 5 (1.0% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,869 hom-ref genotypes 3 this sample was 0/0 here missing genotypes 1 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_gentle/Sample_C_gentle_false_negatives.tsv
FP table -> eval_gentle/Sample_C_gentle_false_positives.tsv
====================================================================== Sample_D_gentle (VCF column: Sample_D)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,851 false positives 1 false negatives 149 precision 0.9998 recall 0.9702 F1 0.9848 recall (het) 0.9672 <- where low depth bites recall (hom) 0.9746
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9979 (4,841/4,851) ref dropout 10 true het called 1/1 (detected, genotyped WRONG) allelic dropout 97 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_gentle/Sample_D_gentle_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 149 of which HARD sites 6 (1.2% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,869 hom-ref genotypes 16 this sample was 0/0 here missing genotypes 1 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_gentle/Sample_D_gentle_false_negatives.tsv
FP table -> eval_gentle/Sample_D_gentle_false_positives.tsv
====================================================================== Summary appended to eval_gentle/summary_all_runs.tsv
Report in your wiki, per sample: precision, recall, F1 (detection) recall_het vs recall_hom (where depth actually hurts) gt_concordance (genotyping) ref_dropout, allelic_dropout (the two ways a het goes wrong) indels_this_sample (artifacts -- explain the pattern)
(snps) [alyrcoun@sphinx bcftools_results]$ python3 evaluate_snps_v4.py \
--truth truth_snps.vcf \ --calls filtered_stringent.vcf.gz \ --all-samples \ --tag strict \ --outdir eval_strict
====================================================================== Sample_A_strict (VCF column: Sample_A)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,772 false positives 0 false negatives 228 precision 1.0000 recall 0.9544 F1 0.9767 recall (het) 0.9516 <- where low depth bites recall (hom) 0.9585
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 1.0000 (4,772/4,772) ref dropout 0 true het called 1/1 (detected, genotyped WRONG) allelic dropout 143 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_strict/Sample_A_strict_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 228 of which HARD sites 7 (1.4% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,774 hom-ref genotypes 2 this sample was 0/0 here missing genotypes 0 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_strict/Sample_A_strict_false_negatives.tsv
FP table -> eval_strict/Sample_A_strict_false_positives.tsv
====================================================================== Sample_B_strict (VCF column: Sample_B)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,347 false positives 0 false negatives 653 precision 1.0000 recall 0.8694 F1 0.9301 recall (het) 0.8216 <- where low depth bites recall (hom) 0.9384
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9174 (3,988/4,347) ref dropout 359 true het called 1/1 (detected, genotyped WRONG) allelic dropout 527 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_strict/Sample_B_strict_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 653 of which HARD sites 62 (12.4% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,774 hom-ref genotypes 344 this sample was 0/0 here missing genotypes 83 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_strict/Sample_B_strict_false_negatives.tsv
FP table -> eval_strict/Sample_B_strict_false_positives.tsv
====================================================================== Sample_C_strict (VCF column: Sample_C)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,772 false positives 0 false negatives 228 precision 1.0000 recall 0.9544 F1 0.9767 recall (het) 0.9516 <- where low depth bites recall (hom) 0.9585
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9998 (4,771/4,772) ref dropout 1 true het called 1/1 (detected, genotyped WRONG) allelic dropout 143 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_strict/Sample_C_strict_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 228 of which HARD sites 9 (1.8% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,774 hom-ref genotypes 2 this sample was 0/0 here missing genotypes 0 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_strict/Sample_C_strict_false_negatives.tsv
FP table -> eval_strict/Sample_C_strict_false_positives.tsv
====================================================================== Sample_D_strict (VCF column: Sample_D)
DETECTION (primary metric -- did you find the variant at all?) truth SNPs 5,000 (het 2,954 / hom-alt 2,046) true positives 4,761 false positives 0 false negatives 239 precision 1.0000 recall 0.9522 F1 0.9755 recall (het) 0.9479 <- where low depth bites recall (hom) 0.9585
GENOTYPING (secondary -- of the ones you found, right zygosity?) concordance 0.9981 (4,752/4,761) ref dropout 9 true het called 1/1 (detected, genotyped WRONG) allelic dropout 154 true het not detected (costs recall) hom called het 0 true 1/1 called 0/1 confusion table -> eval_strict/Sample_D_strict_genotype_confusion.tsv
WHY THE MISSES HAPPENED no_call 239 of which HARD sites 10 (2.0% of all 500 HARD truth sites) zero-depth misses 0 median depth at miss NA
BOOKKEEPING records in VCF 4,774 hom-ref genotypes 13 this sample was 0/0 here missing genotypes 0 this sample was ./. here indels carried by this sample 0 EXCLUDED from scoring (the truth set has no indels -- every one of these is an artifact. Compare this number across your samples and explain the pattern.) multiallelic sites 0 removed by thresholds 0
FN table -> eval_strict/Sample_D_strict_false_negatives.tsv
FP table -> eval_strict/Sample_D_strict_false_positives.tsv
====================================================================== Summary appended to eval_strict/summary_all_runs.tsv
Report in your wiki, per sample: precision, recall, F1 (detection) recall_het vs recall_hom (where depth actually hurts) gt_concordance (genotyping) ref_dropout, allelic_dropout (the two ways a het goes wrong) indels_this_sample (artifacts -- explain the pattern)