Deze pagina lezen in het Nederlands.

Author: prof.dr. R.J. de Haan
Co-author: dr. ir. N van Geloven
Authorship

This page was translated from the Dutch-language original using AI and has not been fully reviewed. Where the two differ, the Dutch version takes precedence.

• How is kappa calculated?
  • Example calculation of kappa
• How do I interpret Cohen's Kappa?
• My kappa comes out much lower than I expected, what could be causing this (1)?
• My kappa comes out much lower than I expected, what could be causing this (2)?
• Where do I find kappa in SPSS?
• How can I specify the weights myself for a weighted kappa?
• References

Kappa (also known as Cohen's kappa) is a measure of intra- and inter-observer agreement. The kappa coefficient is a chance-corrected measure of agreement between ratings. A kappa of 0 means that the agreement between ratings is entirely due to chance, while a kappa of 1 is complete agreement.

Cohen's kappa - or kappa for short - is a widely used statistical measure for determining the degree of intra- or inter-rater reliability. Usually it concerns a situation in which a particular clinical phenomenon is scored dichotomously by raters (e.g. ECG abnormality present or absent), but the calculation can easily be extended to three or more rating categories.

Kappa expresses a chance-corrected agreement, and is calculated as follows:

$\displaystyle\kappa=\frac{\%\text{ chance-corrected observed agreement}}{\%\text{ chance-corrected potential agreement}}$

Let's assume that we want to determine whether nurses can reliably assess the pupillary reflexes of patients in the acute phase of a stroke. To investigate this, we ask 2 randomly chosen nurses to independently assess the reflexes in 100 patients.

		rater 1
		absent	present
rater 2	absent	20	15	35
	present	10	55	65
		30	70	100

The observed agreement between the two ratings here is 75 / 100 = 75%. This percentage seems reasonably good, but needs qualification. After all, the nurses may sometimes (or often) also have agreed with each other purely by chance. It is therefore useful, as a next step, to calculate the percentage of agreement that is based on chance. The chance frequency of a cell in a cross-tabulation can be calculated by multiplying the corresponding column total by the corresponding row total, and then dividing the product by the total number of observations.

		rater 1
		absent	present
rater 2	absent	10.5		35
	present		45.5	65
		30	70	100

The chance frequency of cell A is then $\frac{30\times 35}{100}=10.5$ and the chance frequency of cell D is $\frac{70\times 65}{100} = 45.5$. The chance agreement between the raters is therefore $\frac{56}{100}=56\%$. Now we can calculate the chance-corrected observed agreement; in this example that is $75\% - 56\% = 19\%$. As an absolute number, this percentage doesn't say much yet. The 19% only becomes meaningful when set against the potential agreement that is possible (again, of course, chance-corrected). The chance-corrected potential agreement is $100\% - \text{the chance agreement of }56\% = 44\%$. Kappa (% chance-corrected observed agreement / % chance-corrected potential agreement) is therefore: $\frac{19\%}{44\%} = 0.43$ (a 'moderate' agreement).

Kappa values are interpreted (using arbitrary(!) cutoffs) as follows: k < 0 reflects 'poor', 0 to 0.20 'slight', 0.21 to 0.4 'fair', 0.41 to 0.60 'moderate', 0.61 to 0.8 'substantial', and above 0.81 'almost perfect' (see reference Landis and Koch).

I want to calculate the kappa between three radiologists when assessing small pulmonary emboli. What matters to me is how often they see a mini pulmonary embolism (I have given this a value of 2; a larger embolism or no embolism I have given a value of 1). Most values are 1, with a 2 occurring in a few cases (and different ones for the three radiologists). Even though there is discrepancy between the radiologists, you would still expect a reasonably decent kappa: this kind of mini pulmonary embolism doesn't occur that often. Yet I get a very low kappa (0.08). I don't understand this.

It's not possible to say for certain without seeing the data, but a plausible explanation for obtaining such a low kappa in this case is the following. Kappa does not calculate the agreement between the raters as such, but calculates the extent to which the raters agree more than would be expected purely by chance. When the majority of the emboli are scored as 1 by the raters, and the choice of which emboli get a 2 were made purely by chance, there is still a high probability that the raters will both have scored 1 in many cases. It is therefore difficult to add much on top of this chance-based agreement. Apparently the actual agreement did not turn out to be much better.

See also the references to Feinstein, Cicchetti and Lantz at the bottom of this page.

Cohen's Kappa takes on a value of 0.82. However, it is only in 3 of 332 patients that there is discrepancy between rater 1 and rater 2. Can you explain how it can be that with less than 1% of the 'population' having no agreement, this results in a relatively low kappa value? If you compare this, for example, with the agreement for another outcome parameter, where there is discrepancy in 6 of 129 patients (4.6%) and this leads to a Cohen's kappa (nominal values) of 0.91, I don't quite understand the discrepancy. Is this due to the correction for the percentage chance-corrected agreement?

This can indeed be due to the correction for chance agreement. Because in the analysis of 332 patients the same category occurred very often (for both raters), the probability of having agreement based on chance alone is also very high. It is difficult to add much on top of that. See also the question above. In the 2nd analysis there is overall more variation in the categories that occur, so it is less likely, based on chance, to find that much agreement.

See also the references to Feinstein, Cicchetti and Lantz at the bottom of this page.

For two raters, Cohen's Kappa can be calculated in SPSS as follows. Put the variables (the ratings from the 1st and 2nd rater) in columns next to each other. Then go to Analyze->Descriptive statistics->Crosstabs, enter one variable under "rows" and the other under "columns", click on "statistics" and check "Kappa". Then click continue and OK. The output gives you the cross-tabulation and Cohen's Kappa. To calculate intra-observer agreement (two ratings from 1 rater), you can follow the same approach. You then put the first rating and second rating in columns next to each other and enter them under "rows" and "columns".

For multiple raters/ratings (generalized kappa or Fleiss' kappa) there is not yet a standard procedure in SPSS, although an SPSS extension is available (see below). On this website you can find several macros (including in Excel, SPSS and SAS) that can calculate a kappa with multiple observers.

For a weighted kappa, which is used for ordinal outcomes, there is also not yet a standard procedure in SPSS, although an SPSS extension is available (see below). There is also the following option: the weighted kappa (with a quadratic weighting function) is essentially equal to an intraclass correlation coefficient. A clear description of this can be found in "Biostatistics, The Bare Essentials, Norman & Streiner - pp. 220-222".

For both the Fleiss kappa and the weighted kappa, extension bundles are available for SPSS 20. After downloading these, extra menu options appear under Analyze -> Scale. The following steps are needed for this. Before these extensions can be installed, the PythonEssentials plugin must first be downloaded and installed. The correct version for the operating system must be selected here (for most PCs at the AMC this will be Windows 32-bit (win32)). The downloaded installer can then be run. Once the Python plugin has been installed, the extensions for calculating a weighted kappa or Fleiss kappa can be downloaded. These extensions can be activated in SPSS via Utilities > Extension Bundles > Install Extension Bundle. The file location of the downloaded extension bundle can then be selected.

So far I have used a linear weighted kappa via the SPSS extension, but we would now like to manually assign weights to different categories. Manually assigning weights is not possible in the SPSS extension. Do you perhaps know of a small program I could use where this is possible?

In SPSS, instead of using the extension, you can use a piece of syntax and specify the desired weights within it. See the reference to Valiquette on this SPSS website. In Stata you can specify it directly when using the function kapwgt. In R you can also specify the weight matrix directly in the 'kappa2' function of the irr package. Here you do seem to need to provide a symmetric weight matrix. In the cohen.kappa function in package psych you seem to be able to specify the entire matrix.

• Douglas G, Altman, Practical Statistics for Medical Research, p. 404
• Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543-9.
• Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiol. 1990;43(6):551–558.
• Lantz CA, Nebenzahl E. Behavior and interpretation of the kappa statistic: resolution of the two paradoxes. J Clin Epidemiol. 1996 Apr;49(4):431-4.
• Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159–174. DOI: 10.2307/2529310.
• Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology. 2012;8(1):23-34.

---

Finished reading? You can go to the Overview of all topics, or to Choosing a test for help picking a suitable test or analysis. Want to learn more about biostatistics? Then follow the AMC e-learning Practical Biostatistics. Did you find something on these pages that isn't correct? Does a link not work? Or would you like to contribute to the wiki? Then get in touch with us.

This biostatistics wiki is an initiative of the former statistics helpdesk of Amsterdam UMC, location AMC. Amsterdam UMC staff can request statistical support via intranet.