Do groups differ? (t test for two independent samples, analysis of variance)

In brief

One number (e.g. points) is to be compared between groups, and each person belongs to exactly one group. For two groups Statable runs Welch's t test, for three to twelve groups Welch's one-way analysis of variance (the classical one is shown next to it).

Are the values in several columns side by side (two or more lists, as in Excel)? Then open the question and choose "My values are in several columns". Statable stacks the lists in a new project and compares their means: with the t test for two independent samples (Welch) for two lists, with the analysis of variance (Welch) for three or more. The lists may differ in length. If the same people were measured twice, "Has something changed?" is the right question instead.

Requirements

• Dependent variable: numbers without clear skewness or outliers within the groups; Welch's test then holds even with small groups.
• Independent observations.
• Welch's version does not require equal spread in both groups.
• The analysis of variance also shows Levene's test for equal spread - for description only: Welch and Games-Howell apply regardless of its result.

Reading the result

The group means are shown side by side. The p value shows how compatible the data are with the assumption that there is truly no difference (a small p value: not very compatible); the effect size Cohen's d (for two groups) says how large it is. d uses the common (pooled) standard deviation of both groups; if a group has fewer than 20 people, Statable also gives Hedges' g, which corrects for the slight overestimation of d in small groups. The confidence interval shows the uncertainty. The analysis of variance only says that at least two groups differ, not which.

t(97.4) = 2.31, p = .023, d = 0.46, 95% CI [0.06, 0.86]

Three or more groups: which ones differ?

If Welch's analysis of variance is clear, Statable shows pairwise comparisons (default Games-Howell; Tukey or Bonferroni can be chosen); see "Which groups differ?".

Does the number separate the groups?

With two groups you can also ask how well the number separates the groups (ROC curve, area under the curve, best cut-off); see "Does a number separate two groups?".

Second grouping or control variable

With a second grouping ("Second grouping") Statable runs the two-way analysis of variance, with a control variable (under "More options") the analysis of covariance. Both at once is not possible. Without these choices everything stays as described here; see "Two-way analysis of variance" and "Analysis of covariance".

Testing for equivalence

With two groups you can also check under "More options" whether the difference is smaller than a bound set beforehand (equivalence test, TOST). A non-significant t test does not show that; see "Equivalence test (TOST)".

Comparing the spread (F test)

With two groups, "Compare the spread of the two groups" is below the t test: the F test as in R (var.test) and on the TI-84 (2-SampFTest), F = variance of the first group divided by that of the second, two-sided, with the 95% range of the ratio. It assumes normally distributed values and is not suitable as a pre-test for the t test; the Welch test allows unequal spread anyway. With three or more groups, Levene's test is below the analysis of variance in "All numbers".

Typical examples

• Reading test by teaching method (two groups): Welch t test.
• Exam performance by field of study (four groups): analysis of variance, then pairwise comparisons.
• Waiting time in days (skewed, 12 people per group): rank test.

No normal distribution (rank test)

Below the result is the choice "No normal distribution (rank test)". Whether a rank test fits better is shown before the calculation in the checklist, with the reason: clear skewness in a group (above 1), very unusual single values (outliers) or a measure with few levels (ordinal or at most 7 distinct values, the same rule as for correlation). A small group alone is no reason: with symmetric data Welch's test still holds, and the rank test tests a slightly different question. The choice belongs before the calculation; Statable never switches by itself. With two groups it runs the Mann-Whitney U test, with three or more groups the Kruskal-Wallis test with Dunn pairwise comparisons (Holm-adjusted). The page shows medians, a sentence, the effect size (r or ε², plus the rank-biserial correlation) and a line for your paper. The sentence and table say whether the p value is exact or approximate.

With two groups the page also gives the Hodges-Lehmann estimate (HL): the median of all differences between one person from one group and one from the other, in the unit of measurement and with a 95% confidence interval. The interval inverts the same test that gives p; if it excludes 0, p < .05 as well. HL is not the difference of the two medians - that can turn out differently.

Exact or approximate: with fewer than 50 values per group (Wilcoxon: fewer than 50 pairs with a difference) Statable computes the p value exactly, also with tied values (conditional distribution); above that the normal approximation applies. In Wilcoxon, pairs without a difference are dropped before ranking (as in SPSS). R computes the exact test differently: it ranks pairs without a difference along (Pratt method). So values can differ from R. With small sample sizes a level of exactly 95% cannot be reached; Statable then states the level reached.

U = 127.5, z = 3.85, p < .001, r = .53, rrb = .62, HL = 32, 95% CI [15, 53]