Randomization test

A randomization test evaluates the null hypothesis that the observed data are distributed independently of group labels or treatment assignments. Under the null, the labels are exchangeable: any permutation of them is equally likely. The test statistic is computed for the original data and then for many random permutations of the labels; the proportion of permutations yielding a statistic as extreme as the original provides the p-value.¹² This method is exact because it uses the true permutation distribution rather than an asymptotic approximation. It predates modern computing: R.A. Fisher introduced the idea in 1935 in the context of randomized experiments.³

The procedure has three steps: (1) choose a test statistic (e.g., difference of means, correlation coefficient, or t-statistic); (2) randomly reassign the group labels (or scramble the data) many times, each time recomputing the statistic; (3) compare the observed statistic to the distribution of permuted statistics. The p-value is the fraction of permuted statistics as extreme or more extreme than the observed one. Applications span medicine, ecology, genetics, and machine learning. In genome-wide association studies, permutation tests control for multiple testing. They are also used in A/B testing and nonparametric analysis of variance.

Randomization tests are closely related to bootstrapping, which resamples with replacement to estimate confidence intervals, and to exact tests such as Fisher's exact test for contingency tables. Unlike parametric tests (e.g., t-test or ANOVA), they make no normality assumption, making them robust for small samples or skewed distributions. When the number of possible permutations is large, they are approximated by Monte Carlo sampling rather than full enumeration. The Wilcoxon rank-sum test is equivalent to a randomization test on ranks.⁴ Randomization tests also form the basis of permutation-based p-value adjustment in multiple testing.

Although Fisher's 1935 work is foundational, the formal development of randomization tests is often credited to E.S. Edgington in the 1960s, who systematized the approach and addressed computational challenges.⁵ Early applications were limited by manual computation; only with the rise of computers did the method become widely used. A subtle but crucial point: randomization tests do not require random sampling from a population—only random assignment in an experiment. This makes them ideal for randomized controlled trials where the inference is about the treatment effect in the study sample, not a broader population. Modern implementations use efficient algorithms and parallel computing to handle millions of permutations, enabling applications in neuroimaging and high-dimensional biology.

Exchangeability

Property that the joint distribution of data is invariant under permutation of labels; the key assumption for a randomization test.

Null distribution

The distribution of the test statistic under the null hypothesis, approximated by the permutation distribution in a randomization test.