Has something changed? (before/after, paired t test)

In brief

The same people were measured twice (for example a knowledge test before and after a course); the aim is to check whether the values changed on average. Choose the column of the first and that of the second measurement. Statable calculates the paired-samples t test and gives the effect sizes dz and dav with a 95% confidence interval.

Requirements

• Both columns are numbers from the same people, one row per person (for long data, "Reshape" helps).
• The changes show no clear skewness or outliers; the t test then holds even with few people. There is no fixed minimum number for this.
• At least 3 people with both values; the change is not identical for everyone.

Reading the result

• Differences are after − before: positive means increased. The page shows both means, the mean change with 95% confidence interval, t, degrees of freedom and p.
• dz is the mean change divided by the SD of the changes; dav divides by the mean of the two SDs and is comparable with d from two-group studies. That is why the classification (rule of thumb for d: 0.2 small, 0.5 medium, 0.8 large) follows dav. These benchmarks do not fit dz: dz grows the more strongly before and after are correlated; it is the measure for statistical power (Lakens 2013).
• The line chart connects each person's before and after; it shows whether most lines run in the same direction.

Notes on the page

• With n < 20 the note "Small sample" appears: the result is then less certain. Statable recommends a rank test before the calculation if the changes are clearly skewed, have outliers or the measure has few levels (ordinal or at most 7 values). It is chosen below the result with "No normal distribution (rank test)": the Wilcoxon test, which uses only the rank order of the changes (with medians, sentence, effect size r and a line for your paper). It also gives the pseudo-median of the change (Hodges-Lehmann estimate: median of the means of every two changes) with a 95% confidence interval.
• Exact or approximate: with fewer than 50 values per group (Wilcoxon: fewer than 50 pairs with a difference) Statable computes the p value exactly, also with tied values (conditional distribution); above that the normal approximation applies. In Wilcoxon, pairs without a difference are dropped before ranking (as in SPSS). R computes the exact test differently: it ranks pairs without a difference along (Pratt method). So values can differ from R. With small sample sizes a level of exactly 95% cannot be reached; Statable then states the level reached.
• Without a comparison group the measurement shows only that something changed, not why.
• With a comparison group (course vs. no course), analysis of covariance (ANCOVA) is the appropriate route: compare after by group, with before as the covariate. Below the result, "Is there a control group and an intervention group?" leads directly to "Do groups differ?" with the before value as the control variable.

t(13) = 5.10, p < .001, dz = 1.36, 95% CI [0.61, 2.09], dav = 0.77

For specialists

Wilcoxon: Statable calculates it itself (the exact p value for fewer than 50 changes other than 0, also with tied changes, otherwise the normal approximation with tie correction, no continuity correction; checked against R wilcox.test and PSPP NPAR TESTS /WILCOXON).

More options

• Test for equivalence (equivalence test, TOST): whether the change is smaller than a bound set beforehand; see "Equivalence test (TOST)".
• Reliable change for each person (Reliable Change Index): which people changed more than measurement error would lead us to expect; see "Reliable Change Index".
• If both columns are yes/no, Statable calculates the McNemar test (exact) by itself instead of the t test; see "McNemar test and Cochran's Q".

Typical examples

• Anxiety of the same students before and after a course: paired t test.
• Pain score before and after a treatment in 12 people: rank test (Wilcoxon).
• Reaction time without and with caffeine in the same person: paired t test.