Multiple testing - alex-strobel/DPP-LabManual GitHub Wiki
When we test a hypothesis using the frequentist approach, we typically say that the test was significant, if p < .05. Yet, if we test the same hypothesis multiple times, 5% of these tests will produce significant results per chance. As an example: You hypothesise that Neuroticism is negatively correlated with overall well-being. You assess the former using the NEO-PI-R and the later via the WH0-5 self-report questionnaire. If you now correlate the six Neuroticism facts with the WHO-5 total score, you need to correct for multiple testing because you test the same hypothesis six times (for each facet). This following manual page outlines the most commonly used correction methods in our group and will give some general recommendations on multiple testing.