Intraclass correlation (ICC)

The ICC shows what proportion of the variance in all ratings reflects true differences between the cases rather than differences between raters or measurement error. 0 means no agreement, 1 perfect agreement. Two distinctions matter: should the values be identical (absolute agreement, ICC(2,·)), or may raters differ by a constant amount (consistency, ICC(3,·))? And does it refer to a single rating ("single") or to the mean of all raters ("mean", more reliable)? Classification according to Koo and Li (2016): below .50 poor, up to .75 moderate, up to .90 good, above that excellent; they recommend considering the whole confidence interval.

Example: Three teachers score 12 essays with points: ICC(2,1) = .74 (95% CI .45 to .91). The point estimate indicates moderate agreement of a single rating, but the interval ranges from poor to excellent - with 12 essays the estimate is still imprecise. The mean of the three is clearly more reliable, ICC(2,3) = .89.

Caution: The ICC is imprecise with few cases (at least 30 recommended) and comes out low when the cases hardly differ - even with good agreement. It requires numbers with meaningful distances.