Step 6: Evaluation Against the Truth Set - rsalani13/EPP622-2026-SNPCalling GitHub Wiki

Checking for accuracy against provided files evaluate_snps_v4.py and truth_snps.vcf

Commands

cd ../
mkdir evaluating_6
cd evaluating_6/
ln -s ../fltering_5/* .

ln -s ../../rawdata/check/* .

nano eval_script.sh

-------------------------------------------------
# Evaluate gently filtered SNPs
python3 evaluate_snps_v4.py \
  --truth truth_snps.vcf \
  --calls variant_gentle.vcf.gz \
  --all-samples \
  --tag gentle \
  --outdir evaluation_gentle

# Evaluate stringently filtered SNPs
python3 evaluate_snps_v4.py \
  --truth truth_snps.vcf \
  --calls variant_stringent.vcf.gz \
  --all-samples \
  --tag stringent \
  --outdir evaluation_stringent
-------------------------------------------------

bash eval_script.sh

My results also need an evaluation for the raw data, which was not included. Here's that part!

nano eval_for_raw.sh

# Evaluate raw SNP calls
python3 evaluate_snps_v4.py \
  --truth truth_snps.vcf \
  --calls variant_script_output.raw.vcf.gz \
  --all-samples \
  --tag raw \
  --outdir evaluation_raw

bash eval_for_raw.sh

Results

Find the precision, recall and F1 for gently filtered and stringently filtered SNP sets.

Sample	Filter	TP	FP	FN	Precision	Recall	F1
A	raw	4875	0	125	1.0000	0.9750	0.9873
A	gentle	4864	0	136	1.0000	0.9728	0.9862
A	stringent	4772	0	228	1.0000	0.9544	0.9767
B	raw	4422	23	578	0.9948	0.8844	0.9364
B	gentle	4419	0	581	1.0000	0.8838	0.9383
B	stringent	4347	0	653	1.0000	0.8694	0.9301
C	raw	4871	2	129	0.9996	0.9742	0.9867
C	gentle	4865	0	135	1.0000	0.9730	0.9863
C	stringent	4772	0	228	1.0000	0.9544	0.9767
D	raw	4859	15	141	0.9969	0.9718	0.9842
D	gentle	4851	1	149	0.9998	0.9702	0.9848
D	stringent	4761	0	239	1.0000	0.9522	0.9755