Understand the test for a proportion: rejection region, actual α, prediction interval

In brief

Under H₀, X is binomially distributed with n and p₀. From the significance level α the tile determines the rejection region K, the numbers of hits at which H₀ is rejected. Example n = 100, p₀ = 0.5, α = 5%, right-sided: K = {59, …, 100}. With at least 59 hits H₀ is rejected.

Rejection region K and actual α

• Right-sided (H₁: p > p₀): K = {g, …, n} with the smallest g for which P(X ≥ g) ≤ α. Left-sided (H₁: p < p₀): K = {0, …, k} with the largest k for which P(X ≤ k) ≤ α. Two-sided (H₁: p ≠ p₀): both sides with α / 2 each.
• The actual α is the probability with which K occurs under H₀. It is never above the chosen α, but often below it, because X takes only whole values (4.43% instead of 5% in the example). Statable always shows it.
• If K is empty, it says: “With this n, H₀ can never be rejected at the level α.” This happens for small n and small α.

p value and decision

One-sided, p = P(X ≥ x) or P(X ≤ x). Two-sided, Statable doubles the smaller one-sided value: p = min(1; 2 · min(P(X ≤ x), P(X ≥ x))). So it always holds that x lies in K exactly when p ≤ α. Decision and p value never contradict each other. R (binom.test) calculates two-sided differently (sum of all probabilities that are at most as large as P(X = x)); for p₀ ≠ 0.5 the p value can be smaller there. Example n = 40, p₀ = 0.4, x = 7: doubling 0.0041, binom.test 0.0033. For p₀ = 0.5 both are the same.

Not rejecting does not mean H₀ is true

If x is not in K, the sentence reads: “The data give too little evidence that p differs from p₀.” It is no proof that H₀ is true. “Significant” only holds together with the level at which it is said.

Type II error and power

Type I error: H₀ is rejected although it is true; its probability is the actual α. Type II error: H₀ is not rejected although a different proportion p₁ holds in reality; its probability is β = P(X ∉ K | p₁). The power 1 − β = P(X ∈ K | p₁) is the probability of finding the difference. Example n = 100, p₀ = 0.5, right-sided, p₁ = 0.6: β = 37.75%, power 62.25%.

Prediction interval or confidence interval?

The prediction interval applies when p is known: it gives the range [a; b] of the number of hits that occurs with a probability of at least 1 − α (forward inference). It is the counterpart of the two-sided K. Example n = 100, p₀ = 0.5, 5%: [40; 60], so proportions from 0.40 to 0.60. The confidence interval applies when x is observed: it gives the range for p (backward inference; tile “Calculate from summary numbers”). The sigma approximation μ ± 1.96 σ is shown next to it for comparison.

For specialists

All numbers are exact binomial sums. K is compared with α using a tolerance of 1e-7 (as R does at equality, for example P = 0.05 exactly). The two-sided p value is the “central” method (doubling); it matches the Clopper-Pearson interval and is available in the R package exactci as binom.exact(…, tsmethod = "central"). Checked against R (qbinom, pbinom, dbinom) and with a loop over all rejection regions, printed examples: teacher training Baden-Württemberg (memory chips, dice, canteen).

Sources

• Fay, M. P. (2010): Two-sided exact tests and matching confidence intervals for discrete data. The R Journal 2(1), 53-58.
• KMK: educational standards in mathematics, general university entrance qualification (2012), guiding idea “data and chance”; Lösungswege 8 (ÖBV, Austrian textbook), chapter 7: one-sided and two-sided hypothesis test with the binomial distribution.
• Teacher training Baden-Württemberg: testing hypotheses (stochastics).