Cohen's kappa (κ)

Kappa shows how well two raters agree after removing the agreement that would occur by guessing alone. κ = 0 means: no better than chance; κ = 1: perfect agreement. With ordered levels (grades) it can be weighted so that a near miss counts less than a large difference. Classification according to Landis and Koch (1977): up to .20 slight, up to .40 fair, up to .60 moderate, up to .80 substantial, above that almost perfect.

Example: Two teachers grade 30 essays: the grades agree for 63%, κ = .50 (95% CI .26 to .74): moderate agreement.

Caution: κ depends on the distribution of the categories: if one category is very rare or very common, κ is low despite high agreement. The Landis and Koch thresholds are rules of thumb, not limits.