Permutation test and bootstrap (no normal distribution)

In brief

Under "More options" as "No normal distribution - with random procedures": Statable shuffles or draws samples again and again and counts what chance alone gives.

What for

• Two groups: permutation test (Welch t, so that it holds with unequal spread) and bootstrap interval of the difference, drawn separately within each group.
• Relationship: permutation test for r, bootstrap interval, Fisher z next to it. Prediction: case bootstrap of the coefficients. Distribution: bootstrap of mean or median.

Requirements

Independent cases. A frequency weight is calculated over the expanded set (up to Σw = 100,000); a survey weight is blocked.

Reading the result

Always visible: number of draws B (9999), seed, type of interval. p = (b + 1)/(B + 1), never 0. "Draw again" shows the random error. With n < 30 the note gives the t interval: bootstrap intervals are then usually too narrow.

Reporting the result

Permutation test, 9999 random assignments, p = .025; 95% percentile bootstrap CI [1.4, 7.6], seed 20261008

Limits

The permutation test checks whether the group assignment matters, not the means alone. No residual bootstrap, no permutation test of single coefficients. The random numbers of Statable are not those of R.

Source

Phipson, B., & Smyth, G. K. (2010). Permutation p-values should never be zero. Statistical Applications in Genetics and Molecular Biology, 9(1), 39. Hesterberg, T. C. (2015). What teachers should know about the bootstrap. The American Statistician, 69(4).