Do ratings agree? (kappa, ICC, Bland-Altman)

In brief

Several raters (for example two teachers) or two measurement methods assessed the same cases; the aim is to check how well the values agree. Choose the columns, one per rater, at least two. Statable chooses between Cohen's kappa (two columns of categories), Fleiss' kappa with Krippendorff's alpha (three or more columns of categories; article "Intercoder agreement") and the intraclass correlation (ICC, numbers) and gives a confidence interval; with exactly two number columns a Bland-Altman analysis is added as a collapsible extra. PSPP has no ICC; Statable calculates it itself.

Kappa or ICC

• Cohen's kappa (κ): with exactly two columns of categories (text, nominal columns or numbers with at most 7 distinct values, such as grades). Kappa counts how often both give the same rating and subtracts what chance alone would give. With ordered categories it can be weighted (linear or quadratic).
• ICC: with number columns with many values (for example points) and with more than two raters. Absolute agreement (default) requires identical values. Consistency allows a constant difference between raters (one rater always gives 2 points more) but requires the same distances; single rating or the mean of all raters (more reliable).
• Fleiss' kappa and Krippendorff's alpha: with three or more columns of categories. With exactly two columns Krippendorff's alpha appears below Cohen's kappa; percent agreement and the Brennan-Prediger coefficient are added. Missing codings are allowed (see "Intercoder agreement").
• With number columns the page lets you switch between kappa and ICC.

Bland-Altman: how large are the differences?

The ICC relates the disagreement to the differences between cases; it does not say how large the differences are in the unit of measurement. The plot shows for every case the mean of both values (horizontal) and their difference (vertical, first minus second column). The bias is the mean difference, a constant offset between the methods. About 95% of the differences lie between the limits of agreement (bias ± 1.96 SD). Whether this range is small enough is for the field to decide. If the difference rises or falls with the mean (slope with p < .05), the limits do not hold equally well everywhere.

For exactly two number columns the typical error stands next to it, collapsed: it applies to two measurements of the same people without an intervention in between and shows how far a single measurement is typically off. Its change of the mean is explicitly "second minus first measurement"; Bland-Altman calculates first minus second, where the same number has the opposite sign. More in the article "Measurement precision: typical error".

Method comparison: Passing-Bablok and Deming

With exactly two number columns (two measurement methods) the method comparison stands next to Bland-Altman, collapsed: it shows whether the methods differ by a fixed amount or by a factor, with Passing-Bablok and Deming (λ visible and selectable). More in the article “Comparing two measurement methods”.

Requirements

• All raters assessed the same cases. The ICC and Bland-Altman use complete cases only; the page states how many were left out.
• Kappa: the same categories in all columns. ICC: numbers with meaningful intervals.
• With n < 30 the note "Few cases" appears; Bland and Altman recommend at least 100 for the limits.

Reading the result

• Kappa according to Landis and Koch (1977): up to .20 slight, up to .40 fair, up to .60 moderate, up to .80 substantial, above that almost perfect; below 0 worse than chance.
• ICC according to Koo and Li (2016): below .50 poor, up to .75 moderate, up to .90 good, above that excellent; they recommend considering the whole interval.
• All classifications are rules of thumb.

Kappa: κ (quadratic) = .50, p < .001, 95% CI [.26, .74], n = 30

ICC(2,1) = .74, 95% CI [.45, .91]; F(11, 22) = 9.10, p < .001 (3 raters, 12 cases)

Bland-Altman: bias = −0.33, 95% CI [−0.98, 0.32]; limits of agreement [−2.64, 1.97] (2 instruments, 15 cases)

For specialists

Absolute agreement corresponds to ICC(2,·), consistency to ICC(3,·). Kappa via CROSSTABS /STATISTICS=KAPPA. Bland-Altman: bias with a t interval; the intervals of the limits are exact after Carkeet (2015, noncentral t distribution) and thus somewhat wider and asymmetric compared with the approximation of Bland and Altman (1999); R BlandAltmanLeh uses the coarser approximation s·√(3/n). From n = 234 the 1999 approximation is used.

Typical examples

• Two teachers grade the same 30 essays: kappa (weighted for grades).
• Three raters give points for the same children: ICC.
• Two instruments measure the same people: Bland-Altman.