Multiple comparisons problem

The multiple comparisons problem occurs when a researcher tests many hypotheses simultaneously, and the probability of at least one false positive (Type I error) grows with the number of tests. If each test is conducted at the conventional α=0.05 level, the family-wise error rate (FWER) — the probability of making at least one false discovery — increases rapidly. For m independent tests, the FWER is 1 − (1 − α)^m; with 20 tests it exceeds 64%, and with 100 tests it approaches 99.4%.¹

The problem is not merely theoretical: it underlies the reproducibility crisis in many scientific fields. When researchers perform many comparisons without correction, they are more likely to report findings that are statistical flukes. The issue is distinct from but related to the concept of p-hacking, where researchers selectively report significant results.

The formal study of multiple comparisons began in the mid-20th century. In 1955, John Tukey introduced the term 'multiple comparisons' and developed the Tukey range test for comparing all pairs of means while controlling the FWER.² Earlier, Ronald Fisher had discussed the problem in the context of analysis of variance, but it was Tukey who systematized the approach.

In 1979, Yoav Benjamini and Yosef Hochberg introduced the false discovery rate (FDR), a less conservative criterion that controls the expected proportion of false positives among rejected hypotheses. This method became especially influential in genomics, where thousands of genes are tested simultaneously.³ The Benjamini-Hochberg procedure is now a standard tool in high-throughput biology.

Several methods adjust p-values to account for multiple testing. The Bonferroni correction, the simplest, divides the significance level by the number of tests, controlling the FWER but often being overly conservative. The Holm-Bonferroni method sequentially adjusts p-values and is more powerful while still controlling FWER.

The Benjamini-Hochberg procedure controls the FDR and is less stringent, making it suitable for exploratory studies. Other approaches include the Tukey range test for pairwise comparisons, Dunnett's test for comparisons against a control, and permutation-based methods that resample data to estimate the null distribution. Each method balances the trade-off between false positives and false negatives.

The multiple comparisons problem is pervasive in fields that generate large datasets. In genomics, genome-wide association studies (GWAS) test millions of single-nucleotide polymorphisms, requiring stringent significance thresholds (e.g., p < 5×10⁻⁸) to avoid false associations.⁴ In neuroimaging, researchers analyze thousands of voxels, and corrections like family-wise error or cluster-based permutation tests are standard.

In clinical trials, multiple endpoints or interim analyses raise the risk of false conclusions; regulatory agencies require pre-specified correction strategies. In psychology and social sciences, the problem contributes to replication failures, prompting initiatives like the Open Science Collaboration to encourage pre-registration and correction.

Beyond the standard corrections, there are subtle aspects. The problem also applies to confidence intervals: simultaneous confidence intervals must be wider to maintain a given coverage probability across all parameters. In Bayesian statistics, the issue manifests as the 'Jeffreys-Lindley paradox' and the need for multiplicity adjustments in hierarchical models.

One overlooked historical note: the Bonferroni correction is named after Carlo Emilio Bonferroni, but he did not propose it for multiple testing; it was introduced by Olive Jean Dunn in 1961.⁵ Another edge case: when tests are correlated, the effective number of independent tests is lower, and methods like the Šidák correction or permutation tests can be more appropriate.

In high-dimensional settings, the FDR can be estimated via the 'q-value' approach, and the concept of 'local FDR' provides a posterior probability that a specific test is a false positive.

Family-wise error rate (FWER)

The probability of making at least one false positive among all tests.

False discovery rate (FDR)

The expected proportion of false positives among rejected hypotheses.

Bonferroni correction

A method that divides the significance level by the number of tests to control FWER.

Benjamini-Hochberg procedure

A step-up procedure that controls FDR.

p-hacking

The practice of manipulating analyses to obtain a significant p-value.