McNemar test and Cochran's Q (yes/no over time)

In brief

McNemar (two measurements) and Cochran's Q (three or more) compare yes/no answers of the same people.

What for

Has the share of "yes" changed, for example passing before and after a course? Statable chooses the test by itself for yes/no columns.

Requirements

• The same people; each column has exactly two answers, the same in all columns (0 and 1, or "yes" and "no").
• A "yes"-like answer counts as "yes", otherwise the larger number or the alphabetically second text. Statable names it; it can be swapped.

Reading the result

• McNemar only counts people who changed (exact binomial test). The odds ratio with an exact 95% interval shows how many times more frequent the more common direction of change is.
• Cochran's Q tests whether the share of "yes" changes over the time points; with p < .05, pairwise comparisons follow (exact McNemar, Holm).
• McNemar is calculated exactly (binomial), as in PSPP. R mcnemar.test uses χ² with a continuity correction, so p values differ. Statable caps p at 1; with no changes there is no test.

Reporting the result

exact McNemar test, p = .041, OR = 3.00, 95% CI [1.04, 10.55]; Q(2) = 13.45, p = .001

Limits

• Only two answers. Without a comparison group the test does not show the cause; with very few changes the result is uncertain. Cochran's Q has no groups.

Source

Hollander, M., Wolfe, D. A., & Chicken, E. (2014). Nonparametric Statistical Methods (3rd ed.). Agresti, A. (2013). Categorical Data Analysis (3rd ed.). Wiley.