Effect measures

Deze pagina lezen in het Nederlands.

This page was translated from the Dutch-language original using AI and has not been fully reviewed. Where the two differ, the Dutch version takes precedence.

Contents

• Standardized effect measures
  • Why don't our effect sizes match our p-values?
  • How do I interpret my small but statistically significant effect size?
  • References
  • Additional sources

An effect size is a value that expresses the magnitude or strength of an effect or association. That effect can be a correlation, but also an (estimated) difference between two groups or time points. An effect size therefore has no fixed unit, and this sometimes makes interpretation or comparison difficult. There are various ways to standardize effect sizes. Cohen's d is an example of a standardized effect measure.

Standardized effect measures

Why don't our effect sizes match our p-values?

We are working on a manuscript about neuropsychology in treated and control patients. We added Cohen's d effect sizes to the table with the difference scores (3-week follow-up relative to baseline and 8-month follow-up relative to baseline). We calculated the effect sizes as follows: 'We computed effect sizes according to Cohen's d. Effect size is defined as the difference between the mean change scores of both groups divided by the pooled SD of the change scores. An effect of 0.2 reflects a small effect, 0.5 a medium, and 0.8 a large effect.' Our effect sizes are quite large, sometimes even when the p-value is not significant. We find this difficult to interpret. Can you advise us?

The apparent discrepancies between effect size and p-values could be explained statistically as follows: Cohen's d gives a measure of the strength of the effect, relative to the spread (SD). Given that you have a large n, you can estimate the effect and the SD well, and this measure will generally be in line with the p-values. With a small n, however, Cohen's d is not able to account for the uncertainty caused by the small numbers. Nowhere in the calculation is a correction made for, or account taken of, the number of data points that served as input for the effect and the spread. In a test (and the resulting p-value), that aspect is of course taken into account. In other words: if you see a strong effect in a small group, there will be a large Cohen's d and a non-significant p-value. The reverse also holds: in a very large sample, a small effect size (small Cohen's d) can be statistically significant. Perhaps this answers your question?

How do I interpret my small but statistically significant effect size?

I'm not quite sure how to interpret the effect size found in my research into trends in satisfaction. I have a sample size between 7000 and 9000 patients, and I find several significant positive trend lines (in other words, satisfaction has significantly increased). Only in 1 case are the differences between the measurements also relevant (Cohen's d here is > 0.20); for the remaining outcomes, I find effect sizes of 0.07-0.14 for the significant trend lines. What does this actually mean? I think I am able to detect small differences because of the large sample size, but what does that tell me about whether or not they are clinically relevant in this case?

There are always two sides to the story:

• the strength of the effect / effect size indicates whether there is a relevant effect

• statistical significance indicates whether the effect could be due to chance or is expected to be genuinely present

Especially in a large dataset (like yours), it is quite possible for a small, and therefore not very relevant, trend to be found significant. It is then proven that it exists, but on its own it doesn't amount to much. Statistical significance says nothing about whether something is also relevant. Incidentally, I would advise you, when assessing relevance, to also look at the satisfaction scores themselves and not only at standardized effect sizes (such as Cohen's d), because on the original scale it is often easier to judge whether something represents a relevant increase or not.

References

Additional sources

• You may also find our pages on sample size analysis or meta-analysis useful.

---

Finished reading? You can go to the Overview of all topics, or to Choosing a test for help picking a suitable test or analysis. Want to learn more about biostatistics? Then follow the AMC e-learning Practical Biostatistics. Did you find something on these pages that isn't correct? Does a link not work? Or would you like to contribute to the wiki? Then get in touch with us.

This biostatistics wiki is an initiative of the former statistics helpdesk of Amsterdam UMC, location AMC. Amsterdam UMC staff can request statistical support via intranet.