Multiple testing

Deze pagina lezen in het Nederlands.

Author: dr.ir. N van Geloven
Authorship

This page was translated from the Dutch-language original using AI and has not been fully reviewed. Where the two differ, the Dutch version takes precedence.

Contents

• When should I correct for multiple testing?
• How can I correct using the Bonferroni method?
• May I apply a Bonferroni correction if I performed a non-parametric test?
• How can I correct using less strict methods than the Bonferroni adjustment?
• References

The term multiple testing is used for performing multiple statistical tests on the same dataset. The danger of multiple testing is that when you perform many tests, you will more often find a difference purely by chance. You can correct for multiple testing by applying a stricter significance level (<0.05).

When should I correct for multiple testing?

There are no fixed rules for when you should or should not correct. Correcting is wise, because it reduces the chance that you incorrectly reject the null hypothesis and thereby incorrectly identify an effect (type I error). With 20 comparisons between groups, at a significance level of 0.05, you would expect to find 1 significant difference based on chance alone.

Criteria for whether or not to correct can include:

• The phase of the research. It is sometimes argued that more hypothesis-generating research does not need to correct, whereas, for example, phase 3 research certainly must. When the research is still exploratory, incorrectly identifying an effect will lead to further research into it. This is a less serious consequence than bringing to market a drug that has incorrectly been labeled effective.
• The number of tests you perform. When comparing, for example, 3 groups, correction will not be necessary. In genetics, where thousands of possible mutations are tested, it will be.
• To what extent was the observed effect expected in advance? If a clearly hypothesized question is tested, the risk of false positives increases less than when everything available is tested haphazardly.

How can I correct using the Bonferroni method?

For a study on the reliability of endoscopic measurement of a Barrett's esophagus, I have performed multiple tests in which I, for example, compare the absolute agreement between 2 endoscopists for short Barrett's esophagi versus long Barrett's esophagi, expert endoscopists versus non-expert endoscopists, etc. All different comparisons as secondary endpoints. In total, this involves 24 comparisons. I would like to correct for the fact that I am performing multiple tests using the Bonferroni method. Is it correct (for this case) that after the Bonferroni correction I should use an alpha of 0.05/24=0.002? And that I should therefore consider all p-values above 0.002 as not significant?

This is correct.

May I apply a Bonferroni correction if I performed a non-parametric test?

My analysis concerns the comparison of plasma values of healthy volunteers and patients. In my conference abstract I have now worded it as follows: "Plasma concentrations are provided as median (95% CI). Statistical analysis was done by Mann-Whitney U. p < 0.05 was considered significant (p < 0.0029 after Bonferroni correction)."

The Bonferroni correction is not specific to a parametric test. You can therefore use it.

How can I correct using less strict methods than the Bonferroni adjustment?

I have a question about correcting for multiple testing. For this I used the Bonferroni method in an article, dividing the significance level by the number of tests. The reviewer of this article finds that a bit too drastic and suggests Holm's or Hochberg's method (unknown to me). How can I apply these correction methods?

Holm's method is simple to do by hand (or in Excel or similar). It comes down to multiplying the lowest p-value by the number of tests, the second-lowest by the number of tests minus one, the third-lowest p-value by the number of tests minus two, etc. Equivalently, you can also say that you divide the significance level alpha used for the lowest p-value by the number of tests, for the second-lowest p-value by the number of tests minus one, etc. You can also see it explained here.

An example of a more advanced method is the method of Benjamini and Hochberg. This method tries to keep the proportion of false positives under control. These (and many other) adjusted p-values are fairly easy to obtain via the mt.rawp2adjp function in the R package multtest. You give this function the raw p-values and get the corrected ones back. See also the multtest manual.

References

• No adjustments are needed for multiple comparisons. Epidemiology, Rothman, K., Vol. 1, No. 1 (1990), pp. 43-46
• Do multiple outcome measures require p-value adjustment? Feise, R.J., BMC Medical Research Methodology 2002, 2:8
• Controlling the false discovery rate: a practical and powerful approach to multiple testing, Benjamini Y, Hochberg Y, Journal of the Royal Statistical Society, Series B (Methodological), Vol.57, No 1.(1995), 289-300.

---

Finished reading? You can go to the Overview of all topics, or to Choosing a test for help picking a suitable test or analysis. Want to learn more about biostatistics? Then follow the AMC e-learning Practical Biostatistics. Did you find something on these pages that isn't correct? Does a link not work? Or would you like to contribute to the wiki? Then get in touch with us.

This biostatistics wiki is an initiative of the former statistics helpdesk of Amsterdam UMC, location AMC. Amsterdam UMC staff can request statistical support via intranet.