Intercoder agreement (Fleiss, Krippendorff, Brennan-Prediger)

What it is for

Several people assigned the same units (text passages, open answers) to categories; the aim is to check the reliability of the coding. Choose the columns (one per coder, one row per unit) under "Do ratings agree?". Statable shows Cohen's κ (two coders) or Fleiss' κ (from three), Krippendorff's α, percent agreement and Brennan-Prediger.

Requirements

• Coded independently. From MAXQDA: export the codes per segment as a table.
• Categories as text or numeric codes. Gaps are allowed; units with only one coding are stated, α leaves them out.
• The scale for α follows the level of measurement of the columns (mixed: the lowest; text: nominal).

How to read the result

• κ with a 95% interval; the verbal labels after Landis and Koch (1977) are a convention. With two coders Fleiss' κ would be Scott's π, not Cohen's κ; there Statable shows Cohen's κ.
• α: from .800 reliable, .667 to .800 only tentative conclusions, below that not reliable (Krippendorff, 2004).
• Kappa paradox: agreement from 90%, κ below .40, due to a rare category. Brennan-Prediger (chance agreement 1/q) does not depend on it.

How to report

Fleiss' κ = .52, 95% CI [.41, .63]; α (nominal) = .534, 95% CI [.421, .647]; 40 units, 3 coders

Limits

• κ depends on how often the categories occur. α metric (interval) assumes equal distances between the values; ordinal uses only the order. Intervals after Gwet (2014), without bootstrap.

Sources

Kuckartz and Rädiker (2022), Qualitative Inhaltsanalyse; Krippendorff (2004), Human Communication Research 30(3); Gwet (2014), Handbook of Inter-Rater Reliability; Fleiss (1971), Psychological Bulletin 76(5); Brennan and Prediger (1981), Educational and Psychological Measurement 41(3); Landis and Koch (1977), Biometrics 33(1).