Deze pagina lezen in het Nederlands.

Author: M. Soete
Co-author: dr. ir. N van Geloven
Authorship

This page was translated from the Dutch-language original using AI and has not been fully reviewed. Where the two differ, the Dutch version takes precedence.

• What is diagnostic accuracy?
• What measures exist for expressing diagnostic accuracy?
• What is meant by sensitivity and specificity?
• How do I choose the appropriate measure of diagnostic accuracy?
• Can I use the relative number of correctly classified people as a measure of diagnostic accuracy?
• How can I graphically represent diagnostic accuracy?
• How do I analyze the reliability of my measurement method?
• How do I test the difference in sensitivity and specificity between two diagnostic instruments?
• How can I test the difference in sensitivity and specificity between two unpaired diagnostic instruments in SPSS?
• How can I test the difference between two paired positive or negative predictive values?
• How can I test whether there is a difference between 2 paired AUCs?
• References

When conducting diagnostic research, i.e. investigating measurement instruments, the following three questions are often relevant:

• Is the instrument valid/accurate (does the instrument measure what it is supposed to measure, and can the measurement instrument distinguish between people who are ill and people who are not, or between severe complaints and non-severe complaints)?
• Is it reliable (does the measurement instrument produce the same results when repeated by one or by different people, and across different subjects)?
• Is it responsive (can the measurement instrument demonstrate a change)?

The diagnostic accuracy of a test is the capacity of a test to correctly classify people (or animals or objects) into two classes (for example, people with the disease being investigated by the test and people without the disease) (Šimundić).

Diagnostic accuracy can be expressed in various ways. Frequently used measures include: sensitivity and specificity, likelihood ratios, (positive and negative) predictive values, the diagnostic odds ratio, and the area under the ROC curve (AUC, AUROC). Other, less frequently used measures include Youden's index, Yule's Q, and Yule's Y.

Sensitivity and specificity are used when comparing a new test or measurement method with a "gold standard". Sensitivity relates to people who are classified as positive by the gold standard. Sensitivity is then the percentage of that group that is also classified as positive by the new test. Specificity relates to people who are classified as negative by the gold standard. Specificity is then the percentage of that group that is also classified as negative by the new test. The information on the relative number of 'true positives' and 'true negatives' can be combined using the diagnostic odds ratio.

The choice of measure depends on the type of study, the type of test variable, and the preference of the researcher.

For example, when the prevalence is unknown, as in case-control studies, predictive values (and post-test probabilities of disease, i.e. posterior probabilities) cannot be meaningfully calculated. Indeed, predictive values (and post-test probabilities of disease) strongly depend on the prevalence of the disease in the study population. In a non-representative sample, such as in case-control studies, this prevalence can be chosen at will, in contrast to study designs in which a representative sample of the studied population is taken. Sensitivity and specificity, the odds ratio, and AUCs (among other measures such as Yule's Q and Yule's Y) do not depend on prevalence and can be used in studies where the prevalence can be manipulated.

The diagnostic odds ratio and the AUC are excellent measures for expressing the overall diagnostic quality of tests. These measures can suffice for a test with continuous results. For tests with dichotomous results (high-low, yes-no) and for test results dichotomized using a cut-off point (e.g., >= 5 versus <5), separate measures are needed that express the quality of the test for a positive and for a negative test result. It is indeed possible that a particular test of this kind is good for ruling out or establishing a diagnosis, or for both (respectively a test with high sensitivity, high specificity, or high sensitivity and specificity). To judge the degree of diagnostic quality for a positive and a negative test result, posterior probabilities, predictive values, sensitivity and specificity, and likelihood ratios are the appropriate types of measures.

For example, in the ANA test (anti-nuclear antibody) for diagnosing SLE (systemic lupus erythematosus), the value of a negative result for ruling out a suspected diagnosis of SLE is very high, but the diagnostic value of a positive test result for diagnosing SLE is far too low (T.G. Tape). In contrast to the ANA test just mentioned, if both sensitivity and specificity are very high, as is the case for example with quantitative EEG (QEEG) for diagnosing brain dysfunction typical of organic brain damage after an accident (sensitivity = 95.4% and specificity = 97.4%) (Thornton & Carmody), then one may assume that the diagnostic quality of the test is very high both for establishing the diagnosis and for ruling it out, and thus that the diagnosis of organic traumatic brain damage can be made with a very high degree of certainty in the case of a positive result, and that the diagnosis of organic traumatic brain damage can likewise be ruled out with practical certainty in the case of a negative result (Davidson, 2002).

With dichotomous test results there is thus clearly a need for at least two measures to represent the diagnostic quality of the test (which does not preclude meaningfully adding an overall measure of the test's capacity for this kind of test, if desired). For tests with continuous test results, one can choose to determine a cut-off point, in which case at least two measures will also be needed.

No. Diagnostic efficiency (the sum of true positives and true negatives divided by the total number of observations in the sample) is sometimes wrongly regarded as a measure of diagnostic accuracy. Diagnostic efficiency is not solely a function of diagnostic accuracy but is also strongly influenced by the prevalence and the positivity of the test (number of positives divided by the total number in the sample). For example, when there is no diagnostic accuracy at all and both the prevalence and positivity are 50%, the diagnostic efficiency is 50%; if, however, the prevalence is 90% and the positivity is also 90%, the diagnostic efficiency is 82% even when there is no diagnostic accuracy. This measure is not able to indicate a lack of diagnostic accuracy, let alone reflect its degree.

The diagnostic capacity of a test can be represented graphically by constructing a ROC curve corresponding to a particular odds ratio. This can, of course, also be represented graphically by the area under this curve. This approach assumes that the odds ratios for each test result on a test with continuous results do not differ significantly from one another, and that tests with dichotomous results are conceived as producing continuous test results. This latter assumption allows tests with continuous or dichotomous test results to be compared with each other in terms of their overall degree of diagnostic accuracy (provided it can be assumed and/or postulated that the odds ratio theoretically remains constant at every cut-off point).

More often, empirical ROC curves are constructed for continuous test results. These have a jagged, step-like shape that only roughly approximates a smooth curve, because they represent the sensitivity and specificity at different cut-off points. The area under the curve is also calculated differently than for curves constructed to correspond with the odds ratio.

When looking at the reliability of a measurement instrument (or measurement method), the following analyses are often performed:

• Inter-observer agreement (2 or more raters)
• Intra-observer agreement (multiple assessments by 1 rater) Inter-observer agreement indicates the extent to which different observers arrive at the same judgment when performing the same measurement. It thus indicates how sensitive the measurements are to the person performing them. With intra-observer agreement, one looks at the extent to which an observer arrives at the same judgment when repeatedly performing a measurement.

For these analyses, the intraclass correlation coefficient and Cohen's kappa are often used.

I am comparing 2 imaging strategies in patients suspected of appendicitis. The first strategy is the 'conditional CT strategy': everyone first gets an ultrasound, and if this is negative or inconclusive, a CT follows. The other strategy is the 'direct CT' strategy, in which everyone suspected of appendicitis undergoes a CT directly. I want to compare the diagnostic accuracy of these two strategies. First, I want to compare the sensitivity and specificity of both strategies. I did this using the McNemar test: first selecting patients with appendicitis according to the reference values, and then creating cross tabs for conditional x direct (=comparing sensitivity). The same for the negatives according to the reference values (= specificity). However, I have come across several articles in which the chi-square test is used instead to compare sensitivity and specificity. When should/may you use the McNemar test to compare sens/sens and spec/spec, and when the chi-square test?

The McNemar test belongs to a paired design, the chi-square test to an unpaired design. You have a paired design if, for every patient, you know the result of both the 'conditional CT strategy' and the 'direct CT strategy'. In an unpaired design, you know one result for part of the patients and the other result for a different group of patients. I understand from your dataset that you have a paired design, so the McNemar approach you described is indeed the appropriate way to compare sensitivity and specificity.

I am conducting a retrospective study on the diagnosis of biliary atresia. Two tests are being compared: ERCP and liver biopsy, performed in two different patient populations. I now want to investigate whether there is a significant difference in sensitivity and specificity between the two tests. I understand that this can be tested with the chi-square test, but how do I do this in SPSS?

For this, you need to have the test results of the two tests listed one below the other in a single variable. In addition, there is a variable indicating which test it concerns, and a variable indicating the true result (gold standard). As follows:

patient	test result	type of test	gold standard
1	pos	ERCP	pos
2	neg	biopsy	pos
3	neg	ERCP	neg
4	etc	etc	etc

To compare sensitivity, a selection must then be made of those who are positive according to the gold standard (e.g. via 'Data -> Select Cases'). Within this patient selection, the chi-square test can be performed using the variables 'test result' and 'type of test' (via analyze -> crosstabs). To compare specificity, selection on gold standard = neg is needed instead.

I have a group of 420 patients, of whom 231 patients have a negative test. Despite the negative test, 4 patients have the disease (confirmed with the gold standard). In the same group of 420 patients, I have 248 patients with a similar (but slightly different) test, of whom only one has the disease (confirmed with the gold standard). The tests therefore have a negative predictive value of 227/231 and 247/248, respectively. Now I want to know whether this differs significantly. I expect that it does not, given the overlapping CIs, but I want to know a p-value. How do I perform a test for this?

When comparing such paired negative (or positive) predictive values, the statistically tricky aspect is that part of the data is paired (these patients scored negative on both tests) and part of the data is unpaired (these patients scored negative on only one of the two tests). Both the chi-square test and the McNemar test are therefore unsuitable for such a comparison. This requires more advanced statistical models, for example a multilevel model or GEE. Details can be found in the standard textbook on diagnostic tests by Margaret Pepe) and also in this paper by Wendy Leisenring. These methods are implemented, among others, in the R package DTComPair.

I would like to compare the diagnostic value of two variables in terms of area under the ROC curve. These AUCs come from the same dataset.

AUC1 0.56 (95% CI 0.51 to 0.61) variable 1 vs. outcome

AUC2 0.53 (95% CI 0.48 to 0.58) variable 2 vs. outcome

In that case, the comparison of the AUCs will need to take the pairing into account. The reference at the bottom of this page indicates how this can be done. Unfortunately, there is no direct procedure for this in SPSS. There is, however, a program available on the web (medcalc: http://www.medcalc.org/index.php, see under download) in which the Hanley & McNeil method is implemented. You can try it for free for 15 days before purchasing it. Other packages such as Stata (roccomp) and R also have (more advanced) functions for this.

• Bland J.M., Altman D.G. Statistical methods for assessing agreement between two methods of clinical measurement, Lancet 1986, 1:307-310)
• Obuchowski, N.A., Estimating and Comparing Diagnostic Tests' Accuracy When the Gold Standard Is Not Binary, Academic Radiology, Vol 12, No 9, September 2005
• ROC curve on www.anaesthetist.com A great deal of explanation on all kinds of applications of the ROC curve.
• A. Šimundić. Measures of diagnostic accuracy: basic definitions. Medical and Biological Sciences, 2008, 22/4; 61-65
• T.G. Tape. Interpretating diagnostic tests. Internet text, last accessed: 24.11.2010.
• K.E. Thornton & D.P. Carmody. Traumatic Brain Injury Rehabilitation. Efficacy Review of Computers, Strategies, QEEG Guided Neurofeedback and Medications, Technical Report #1 of the Brain Foundation. Internet text, last accessed: 24.11.2010.
• M. Davidson. The interpretation of diagnostic tests: a primer for physiotherapists. Australian Journal for Physiotherapists, 2002, 48:227-233.
• Hanley JA, McNeil BJ. A Method of Comparing the Areas under Receiver Operating Characteristic Curves Derived from the Same Cases. Radiology 1983. A simple though somewhat rough way to test the difference between two AUCs derived from the same cases.

---

Finished reading? You can go to the Overview of all topics, or to Choosing a test for help picking a suitable test or analysis. Want to learn more about biostatistics? Then follow the AMC e-learning Practical Biostatistics. Did you find something on these pages that isn't correct? Does a link not work? Or would you like to contribute to the wiki? Then get in touch with us.

This biostatistics wiki is an initiative of the former statistics helpdesk of Amsterdam UMC, location AMC. Amsterdam UMC staff can request statistical support via intranet.