Week 8: ORA - bcb420-2025/Izumi_Ando GitHub Wiki

⏰ expected - 2 hours & 1 hours : actual - 2.5 hours & 50 mins

Toward a gold standard for benchmarking gene set enrichment analysis

they're trying to make benchmarking gsea methods more fair and reproducible, instead of relying on sketchy custom test cases or biased evaluations

Citation

Geistlinger, L., Csaba, G., Santarelli, M., Ramos, M., Schiffer, L., Turaga, N., ... & Waldron, L. (2021). Toward a gold standard for benchmarking gene set enrichment analysis. Briefings in Bioinformatics, 22(1), 545–556. https://doi.org/10.1093/bib/bbz158

Notes

  • built a benchmark panel of 75 datasets (42 microarray + 33 rnaseq) across 42 diseases
  • used malacards + geneanalytics to get “ground truth” gene sets for each disease
  • this is actually cool: they don’t just say “this method worked” but measure how well each gsea method recovers disease-relevant pathways
  • ORA, GSEA, PADOG, CAMERA, ROAST, GSVA, etc. all tested
  • weird how much variation there is in what gets called “significant”—depends a lot on the null hypothesis used (competitive vs self-contained)
  • runtime + RNA-seq compatibility compared too
  • found that some older microarray methods can still work fine on rnaseq data if you pre-process right (e.g. vst or voom)
  • padog and gsva did well for finding relevant processes
  • they use a relevance score X that weights gene sets based on how high they show up in the EA ranking and how important they are for the disease
  • like that they didn't default to just AUC or cor, but made a metric that actually makes sense for this usecase
  • also tried running all methods on randomized data to estimate type I error (false pos)
  • oh and they checked how sensitive methods are to gene set size by feeding in random sets of various lengths
  • overall, very thorough. i could see myself reusing the compendium or at least the benchmarking logic

Impact of outdated gene annotations on pathway enrichment analysis

this correspondence article highlights how severely outdated annotations in pathway enrichment tools distort results and reduce biological insight in genomics research

Citation

Wadi, L., Meyer, M., Weiser, J., Stein, L. D., & Reimand, J. (2016). Impact of outdated gene annotations on pathway enrichment analysis. Nature Methods, 13(9), 705–706. https://doi.org/10.1038/nmeth.3963

Notes

widespread use of outdated resources

  • surveyed 3,879 papers from 2015 → 67% cited tools using gene annotations >5 yrs old
  • example: DAVID (very commonly used) hadn't been updated since jan 2010 at the time
  • tools with outdated annotations captured only ~26% of current enrichment results

annotations evolve fast

  • between 2009–2016:
    • GO biological processes grew from 6,509 → 14,735
    • Reactome pathways grew from 880 → 1,746
  • deeper annotation trees: more parent terms per GO term (1.73 → 2.09) and longer paths to root
  • median annotations per gene increased from 16 to 29

quality of annotations improving

  • rise in manually curated annotations (Reactome coverage from 15% → 42%)
  • fewer IEA (inferred electronically) terms: 37% → 14%
  • fewer protein-coding genes with no annotations: 12.4% → 4.9%

functional analysis is strongly affected

  • breast cancer RNAi screen: 74% of enriched 2016 terms were missed using 2010 annotations
  • glioblastoma gene analysis:
    • only 20% of 2016 enrichment results recovered using 2010 annotations
    • 96.5% of 2016 results missed when excluding IEA annotations from 2010
    • newer results revealed 13 clinically actionable pathways that old tools completely missed

implications + recommendations

  • major loss of biological signal due to outdated tools → implications for downstream research and clinical insights
  • outdated annotations can’t be fixed post hoc—need to be addressed at time of analysis
  • call for:
    • clear version reporting in papers
    • at least semiannual updates by software developers
    • reproducible access to older versions (g:Profiler does this well)

image
Screenshot of part e of Figure 1, also mentioned in lecture slides this week