Intraclass correlation coefficient EN - pdmoerland/wiki-biostatistiek-bilingual GitHub Wiki
Deze pagina lezen in het Nederlands.
Author: dr. ir. N. van Geloven
Co-author: Prof.dr. P.M.M. Bossuyt
AuthorshipThis page was translated from the Dutch-language original using AI and has not been fully reviewed. Where the two differ, the Dutch version takes precedence.
- Why is an 'ordinary' correlation coefficient not sufficient for assessing agreement?
- Where do I find the ICC in SPSS?
- Which model should I use to calculate my ICC?
- Which type should I use to calculate my ICC?
- Should I use the 'single measures' or 'average measures' ICC from my output?
- How can I interpret the ICC?
- How can I compare 2 ICCs with each other?
- References
The intraclass correlation coefficient (ICC) is used to quantify the degree of agreement between two (or more) repeatedly measured values on a (semi-)continuous scale. For example, when looking at the extent to which radiologists detect the same number of microbleeds when assessing a brain MRI. The ICC is a generalization of Cohen's kappa (in which agreement is examined for the assessment of a dichotomous or ordinal value).
When analyzing the agreement between two continuous assessments, it is good to start by making a scatterplot. The series of observations from the first assessment is then plotted against the second series of observations. Based on the scatterplot, an initial statement can be made about the degree of association between the two assessments. A correlation coefficient can also be calculated. However, a good correlation coefficient is no guarantee of good agreement. Suppose the first assessment of a series of 10 patients yields the following values: 11, 13, 10, 15, 18, 13, 14, 15, 20, 12 and the second assessment yields the numbers: 21, 23, 20, 25, 28, 23, 24, 25, 30, 22. The two series are perfectly correlated (with a correlation coefficient of 1). But there is no strict agreement here. The second series is systematically 10 points higher than the first. It is therefore wise, after looking at the scatterplot and the 'ordinary' correlation coefficient, to also calculate an ICC. When quantifying agreement between more than two raters, an ordinary correlation coefficient is not suitable at all.
See also the article by Bland and Altman for more examples of 'inappropriate' use of a correlation coefficient when analyzing agreement.
You can calculate the ICC using SPSS. To do this, go to Analyze -> Scale -> Reliability Analysis. Under 'Statistics...', check the Intraclass correlation coefficient. Make sure you choose an appropriate model and type (see below) and that the data are structured correctly. The data structure in a study where 3 raters score different subjects is as follows:
| rater1 | rater2 | rater3 |
|---|---|---|
| score subject 1 | score subject 1 | score subject 1 |
| score subject 2 | score subject 2 | score subject 2 |
| score subject 3 | score subject 3 | score subject 3 |
| etc. | etc. | etc. |
There are 3 models that can be used in SPSS to calculate the ICC: Two Way Mixed, Two Way Random and One Way Random
The Two Way Mixed model assumes that there is a fixed number of raters who assess a sample of all patients. This is the case, for example, when the same samples are read once manually and once by computer. We are interested in the agreement between these two specific techniques and do not want to extrapolate this to a third technique or anything similar.
The Two Way Random model assumes that the raters are not fixed, but are also a random sample from all possible raters. This is the case, for example, when a patient's temperature is measured twice. We are interested in the agreement between the two repeated measurements. But we also want to be able to say something about future repetitions of temperature measurements.
The One Way Random model assumes that the raters are a random sample from all raters, but that now all patients (or targets, in general) are assessed. Because medical research usually works with a sample from the total patient population, this model is not often used in that context.
I have a question about calculating the ICC for assessing the degree of agreement between two observers. 2 observers took temperature measurements of patients. In total, 8 measurements were performed simultaneously, and then 32 measurements were performed separately. We had these first 8 measurements performed jointly to determine whether there is sufficient agreement in measuring. I have now used a one-way random model to determine the ICC. My question is, is this correct?
The 8 measurements can be seen as a sample of all (40) measurements. With a one way random model, you assume that you have measured all measurements (in duplicate). That is not the case here. That brings the choice back to two way random or two way mixed. It depends. If, with this ICC, you purely want to say something about the extent to which these two observers had sufficient agreement, you use two way mixed. If you want to generalize the statement to 2 raters in general (including other raters) then you use two way random.
In SPSS there are two types that can be used when calculating an ICC: Consistency and Absolute Agreement. With 'consistency' you are interested in a strong association between the assessments, in which systematic differences between raters are of less importance. If the interest lies in exact agreement, the 'absolute agreement' option is chosen.
I calculated the ICC for various parameters, and now the output gives me an ICC for single measures and one for average measures. Which ICC should I use?
You are usually interested in the 'single measures' outcome. The average measure is relevant if you would actually want to use the average of the two methods as the assessment.
The ICC is defined as the proportion of true variance relative to the total variance. Differences between measurements can be the result of real differences (between persons, or within persons in the case of repeated measurements) or of noise: differences attributable to imperfections in the measurement instrument or the test itself, or to the rater. With a perfect instrument, all differences are “real”, and the ICC is therefore 1. With a worthless instrument, all differences are noise, and the ICC is 0. In other words: the lower the ICC, the more variation the method of assessment has caused, and thus the worse the agreement between assessments. The ICC is classified in a similar way to the kappa coefficient.
Loosely speaking, a low ICC tells you that a different method of assessment could genuinely classify someone as 'a different patient / a different repetition', while with a high ICC the differences between assessments are small relative to the true differences. Note: because the ICC calculates this relative contribution of patient differences to total differences, the ICC will be sensitive to the range of values that occur in the population. In a heterogeneous population, an ICC will give higher values than in a homogeneous population (see also the next question).
187 patients were seen twice by endoscopists who estimated the length of the Barrett's segment: ICC 0.91. I then created 2 groups based on Barrett's esophagus length: group I (Barrett's 5cm or shorter, n=94) ICC 0.44, group 2 (Barrett's length greater than 5 cm, n=93) ICC 0.83. How can I show that these 2 ICCs are significantly different?
You could say that these two ICCs are independent of each other and therefore compare them using a t-test. I assume you also have the SEs of the two estimates? OpenEpi has a handy little menu to do this test if you know the mean, group size, and SE. Do keep in mind that the ICC strongly depends on the spread that exists between persons in the first place. This also explains why the ICC in the total group is much higher than in the 2 groups separately. In the total group there is much more spread, so the spread between the 2 endoscopists is relatively small compared to the total spread. I can imagine that something similar is happening in the 2 groups, meaning that the spread present in the >5 group is larger than in the <5 group? Think carefully about whether the ICC is actually a suitable measure for your comparison.
- Kennisbasis - intraclass correlation coefficient page. Good explanation of the ICC, including several examples and the formulas for how to calculate the ICC yourself in various situations. (The original link,
http://wynneconsult.com/root/Assoc/Intr341.htm, no longer works.) - Shrout, P. E., and J. L. Fleiss. 1979. Intraclass correlations: Uses in assessing reliability. Psychological Bulletin, 86:, 420-428.
- Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods forpsychology. 2012;8(1):23-34.
- Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment. 1994;6(4):284–290.
Finished reading? You can go to the Overview of all topics, or to Choosing a test for help picking a suitable test or analysis. Want to learn more about biostatistics? Then follow the AMC e-learning Practical Biostatistics. Did you find something on these pages that isn't correct? Does a link not work? Or would you like to contribute to the wiki? Then get in touch with us.
This biostatistics wiki is an initiative of the former statistics helpdesk of Amsterdam UMC, location AMC. Amsterdam UMC staff can request statistical support via intranet.