Learning lab: samples, bootstrap, random experiments and four-field table to try out

In brief

The learning lab plays through chance thousands of times and shows what comes out. It has six tabs and needs no data. Everything shown comes as a picture, a number and a sentence; nothing is saved.

The seed

All random numbers come from the seed at the top (default 1). Same seed, same numbers: this is how a run can be repeated and compared with others. “Again (new seed)” counts up by 1 and runs the open tab again. The seed is stated in every sentence. The random numbers do not match those of R or SPSS; same distribution, different numbers.

The six tabs

• Samples: up to 10,000 samples of size n are drawn from a population (normal, flat, strongly skewed, two-peaked, die or proportion). The picture shows the means (proportions for “proportion”) with the curve the theory predicts. The standard error from the simulation is close to σ/√n, and the distribution becomes narrower and more bell-shaped as n grows (central limit theorem).
• Bootstrap: from up to 200 typed values, new samples are drawn again and again with replacement (same size as the series), noting the mean each time. The percentile interval (quantiles of R type 7) is shown next to the t interval. With fewer than 10 values the percentile interval is often too narrow.
• Randomization: two groups (up to 100 values each) are shuffled again and again, noting the difference of the means each time. The p value is (number of differences at least as large + 1) / (shuffles + 1), so that it is never exactly 0. If there are at most 20,000 possible splits, everything is counted as well (then without the “+1”). The Welch t test is shown next to it for comparison.
• Random experiments: coin, one or two dice (fair or loaded) and urn with or without replacement, up to 100,000 repetitions. The history shows the relative frequency against the probability (law of large numbers); the table puts frequencies and exact probabilities side by side.
• Four-field table and tree: from prevalence, sensitivity and specificity (or from four frequencies) the tile builds the table, a tree with natural frequencies, the reversed tree, the predictive values (PPV, NPV) with the Bayes calculation step by step and an independence check. The reference size is chosen so that all numbers are whole (up to 1,000,000); otherwise decimals are shown, never silently rounded. Example: 1%, 90%, 91% give a PPV of 9.17%: of 981 people who test positive only 90 are ill.
• Confidence intervals: 100 intervals (Wilson for a proportion, t interval for a mean) stacked, the true value as a vertical line. Intervals that cover are thin and solid with a circle, those that do not are thick and dashed with a diamond. “Another 100” keeps counting; the total approaches the level.

What “95%” means

The level belongs to the procedure, not to a single interval: in the long run such a procedure covers the true value about 95 times in 100; a single interval either covers it or not.

Wilson and the textbook formula (Wald)

Statable calculates intervals for proportions with Wilson. Textbooks often give the approximation p̂ ± z·√(p̂(1−p̂)/n) (Wald); it is the simple solution of the inequality whose exact solution is the Wilson interval. This is why the textbook number differs slightly. Example 7 of 40, 95%: Wilson [0.0875, 0.3195], textbook formula [0.0572, 0.2928]. Only in the “Confidence intervals” tab can the textbook formula be switched on for comparison; there it shows that with small n and small or large p it covers clearly less often than the level (for example n = 50, p = 5%: Wald 92.0%, Wilson 96.2%).

Limits

• Sample size 1 to 500, at most 100,000 repetitions (calculated in pieces so that the page stays usable); bootstrap series at most 200 values, randomization at most 100 per group.
• The bootstrap handles only the mean, the randomization only the difference of means. No files of your own, nothing is saved.
• The generator has 32 bits; for very many draws (over 10⁸) it would no longer be sufficient, hence the upper limits.

For specialists

Generator: mulberry32; sub-streams are mixed from (seed, running number), not “+1”. Normal values by Box-Muller, discrete distributions by inverting the distribution function from the mode outwards. Checked by properties with a fixed seed: uniformity of the generator (Kolmogorov-Smirnov), chi-square fit of the transformations to the distributions computed in R, standard error, randomization against full enumeration, coverage against the exact coverage probability (sum over dbinom). Percentile interval: quantile type 7 as in R. Bayes: pure arithmetic, checked against R and against the diagnostic accuracy from data.

Sources

• KMK: educational standards in mathematics, general university entrance qualification (2012), guiding idea “data and chance” (tree diagrams, four-field tables, simulations, inference from samples to the population).
• Gigerenzer, G. & Hoffrage, U. (1995): How to improve Bayesian reasoning without instruction: frequency formats. Psychological Review 102(4), 684-704.
• Brown, L. D., Cai, T. T. & DasGupta, A. (2001): Interval estimation for a binomial proportion. Statistical Science 16(2), 101-133.
• Phipson, B. & Smyth, G. K. (2010): Permutation p-values should never be zero. Statistical Applications in Genetics and Molecular Biology 9(1).
• Hesterberg, T. C. (2015): What teachers should know about the bootstrap. The American Statistician 69(4).
• Lock, R. H. et al.: StatKey (lock5stat.com); Seeing Theory (Brown University): models for sampling distributions, bootstrap and randomization.