Calculate from summary numbers: interval, test and sample size

In brief

If only the numbers are available (a count x out of n, or mean, standard deviation and n), this tile calculates the confidence interval and the test. It also answers how large the sample must be for an interval of a desired width.

Proportion

• The interval is the Wilson score interval without continuity correction. The test is the score test, z = (p̂ − p₀) / √(p₀ (1 − p₀) / n); interval and test belong together (duality). The interval is always two-sided, the direction of the test (left, right, two) affects only the p value.
• If p̂ is entered instead of x, Statable calculates x = p̂ · n. If the result is at a whole number (0.07 · 100 = 7), it is rounded to it; otherwise x stays and a note appears. Nothing is rounded silently.
• If n · p₀ · (1 − p₀) is at most 9, the normal approximation of the test is rough. The exact test with the binomial distribution is in the tile “Understand the test for a proportion”. With the same numbers (7 of 40, p₀ = 0.4) the two give different p values: 0.0037 for the approximation, 0.0041 for the exact test with doubling.

Why does the textbook differ?

Many textbooks use the approximation formula p̂ ± z · √(p̂ (1 − p̂) / n) (Wald). It is inaccurate for small n or for proportions near 0 and 1 and can even exceed 1. Wilson solves the underlying inequality exactly. Example 7 of 40, 95%: textbook formula 0.0572 to 0.2928, Wilson 0.0875 to 0.3195. The number in the book is therefore not miscalculated but an approximation. (The R function prop.test applies a continuity correction by default and gives 0.0789 to 0.3336 here; Statable calculates without it.)

Mean

Standard error SE = s / √n, t = (x̄ − μ₀) / SE with n − 1 degrees of freedom, interval x̄ ± t · SE and Cohen’s d = (x̄ − μ₀) / s. Because the population standard deviation is unknown, t is used, not z. n below 2 and s = 0 are blocked, because t and d would not be defined.

Sample size for a desired width

• Proportion: n = ⌈z² · p (1 − p) / d²⌉, mean: n = ⌈(z · σ / d)²⌉, with d as half-width (“± d”). Always rounded up. Some books give the whole length L of the interval; then d = L / 2.
• Without prior knowledge leave p = 0.5: that is the worst case, no other p needs more people. Examples at 95%: ± 5 percentage points 385, ± 3 percentage points 1068, ± 2 percentage points 2401, ± 1 percentage point 9604.
• The formula is the simple estimate for n. The interval of the finished sample is then calculated with Wilson. For a mean, t needs a few more people than z.

Rates from events and person-time

• The choice “A rate” calculates from the number of events and the person-time per group the rate (events per person-time) with the exact Poisson interval after Garwood, the rate ratio of each group against a reference group and the rate difference. The person chooses the time unit (days, weeks, months, years); Statable never guesses it. Rates are shown per 1,000 person-years (selectable).
• The rate ratio is exact (conditional binomial distribution as in R poisson.test), the difference a Wald interval (an approximation, uncertain with fewer than about 10 events per group). If the reference group has no event, the ratio is not defined; Statable says so and shows the exact p value, the difference remains. With more than two groups the p values are not adjusted for multiple comparisons.
• A rate is not a risk: it is not the proportion of people who had the event. The calculation assumes Poisson variation and a rate that stays the same over the observation time; if the counts vary more, the intervals are too narrow (use the count regression with a negative binomial distribution then). Person-time is the sum of the actual observation time, not the number of people times an average duration.

For specialists

Checked against R: prop.test(x, n, p, conf.level, correct = FALSE) for interval and p value, t.test for mean, interval and p, qnorm for the sample sizes. For x = 0 the lower limit is exactly 0, for x = n the upper limit exactly 1. At p̂ = 0.5 the Wilson interval is never wider than the Wald interval.

Sources

• Wilson, E. B. (1927): Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22, 209-212.
• Brown, L. D., Cai, T. T. & DasGupta, A. (2001): Interval estimation for a binomial proportion. Statistical Science 16(2), 101-133.
• Lösungswege 8 (ÖBV, Austrian textbook), chapter 7: confidence interval and sample size of a confidence interval.