Does a number separate two groups? ROC curve and cut-off
In brief
A screening yields scores, and there are two groups (for example with and without a diagnosis). The question: how well does the score separate the groups, and from which score should a result count as positive? In a group comparison with two groups, the collapsible section "Does the number separate the two groups? (ROC, area under the curve)" appears below the result.
What it shows
- AUC (area under the curve) with 95% confidence interval and p: the probability that a randomly drawn person from the "positive" group has a higher value than one from the other group. .50 is chance, 1 is complete separation.
- The ROC curve: for every possible cut-off, sensitivity against 1 − specificity. The further the curve lies from the diagonal towards the top left, the better.
- Best cut-off by Youden with the sensitivity and specificity it achieves. The group with the higher mean counts as "positive", so the cut-off reads "X or more". The Youden index J = sensitivity + specificity − 1 is largest there.
The button "Open curve and cut-offs in Commands" shows the calculation in the commands. The chart can be saved via "Export" as PNG, SVG or PDF or copied as an image. Opening this section also puts the curve into the report as a figure (PDF, Word); the switch "Show the ROC curve in the report" takes it out again.
Caution
- The cut-off is chosen on exactly these data. In new data it usually does worse; without a second sample it is only a suggestion.
- Whether sensitivity or specificity matters more depends on the cost of an error. Missing an illness weighs differently than an unnecessary follow-up test. The Youden index cannot know that.
- With small groups the interval is wide.
Typical example
A short test with 0 to 30 points in 40 people with and 80 without a dementia diagnosis: AUC = .91. Here low scores point to the diagnosis: 24 or less counts as positive (screen-positive), with 85% sensitivity and 88% specificity. Statable takes the group with the higher mean as positive and therefore gives the same cut as "25 or more" for "without diagnosis" - sensitivity and specificity swap roles.
AUC = .91, 95% CI [.86, .96], p < .001; cut-off ≤ 24, sensitivity = .85, specificity = .88, J = .73
If the test already exists as yes/no (positive/negative), the question belongs in the crosstab; see "Does something occur at different rates in different groups?", section on test quality.