Statistical hypothesis testing

The foundation of statistical hypothesis testing was laid by Ronald Fisher in the 1920s, who introduced the null hypothesis and the concept of significance testing.¹ The null hypothesis (H₀) typically represents a default position, such as no effect or no difference, while the alternative hypothesis (H₁) represents the research claim. A test statistic—such as a t-statistic, chi-square, or F-ratio—is computed from the sample data, and its distribution under the null hypothesis is used to calculate a p-value, defined as the probability of observing a result at least as extreme as the one obtained, assuming H₀ is true.² The p-value is then compared to a predetermined significance level (α), often set at 0.05; if p < α, the result is deemed statistically significant and the null hypothesis is rejected. Later, Jerzy Neyman and Egon Pearson formalized an alternative framework that emphasizes controlling Type I and Type II errors and introduced the concept of statistical power.³

The formal steps of a hypothesis test typically follow a five-step process. First, state the null and alternative hypotheses clearly. Second, choose an appropriate test statistic based on the data type and study design. Third, specify the significance level (α) and the decision rule for rejecting H₀. Fourth, collect the sample data and compute the test statistic and associated p-value. Fifth, compare the p-value to α and draw a conclusion—either reject H₀ in favor of H₁ or fail to reject H₀. Tests can be one-tailed (directional alternatives) or two-tailed (non-directional).⁴ The procedure also requires that the underlying assumptions of the test are met, such as independence of observations, normality of residuals (for parametric tests), and homogeneity of variance. Violations of these assumptions can inflate Type I error rates or reduce power.

Despite its widespread use, statistical hypothesis testing is frequently misapplied. A major controversy erupted after the American Statistical Association (ASA) issued a statement in 2016 warning that p-values are often misinterpreted and that statistical significance should not be equated with scientific importance.⁵ Common pitfalls include p-hacking (running many tests until a significant result is found), cherry-picking outcomes, and failing to adjust for multiple comparisons. The replication crisis in many scientific fields has been partly attributed to over-reliance on the dichotomous significant/non-significant threshold.⁶ Additionally, the assumption that the null hypothesis is exactly true is often unrealistic; many tests detect trivial effects with large sample sizes. Researchers are increasingly encouraged to report effect sizes, confidence intervals, and Bayesian alternatives to supplement or replace traditional hypothesis testing.

Beyond the standard textbook narrative, several nuanced aspects of hypothesis testing deserve attention. The Neyman–Pearson lemma, which shows that the likelihood ratio test is the most powerful test for a given significance level, is a foundational result in decision theory but rarely covered in introductory courses.³ Fisher’s exact test, developed for small sample sizes, avoids reliance on asymptotic approximations and remains a critical tool in fields like genetics. Another less-known issue is the problem of optional stopping: collecting data until a significant p-value is obtained inflates the Type I error rate dramatically, a fact that sequential analysis methods address.² The distinction between “statistical significance” and “practical significance” is often overlooked—a very small p-value may arise from a tiny effect that is irrelevant in practice. Similarly, the “null” can be a range of values in equivalence testing (e.g., two one-sided tests, or TOST), which is common in bioequivalence studies. Finally, the role of preregistration and registered reports has emerged as a guard against the flexibility in hypothesis testing that can lead to false positives.

Statistical hypothesis testing is closely tied to several other statistical concepts. Confidence intervals can be viewed as the set of null hypothesis values that would not be rejected at a given significance level, providing a continuous measure of uncertainty. Effect size measures (e.g., Cohen's d, odds ratio) quantify the magnitude of an effect, independent of sample size. Bayesian hypothesis testing incorporates prior probabilities and provides a Bayes factor as a measure of evidence for one hypothesis over another.² Sequential analysis and adaptive designs allow hypothesis testing with interim analyses, maintaining control over error rates. Nonparametric and permutation tests offer alternatives when parametric assumptions are violated. The field of false discovery rate (FDR) control, developed by Benjamini and Hochberg, addresses the problem of multiple testing in high-dimensional data, such as genomics.⁶ These extensions illustrate that hypothesis testing is not a monolithic procedure but a family of methods tailored to different inferential goals.

Null hypothesis (H₀)

A default statement that there is no effect, no difference, or no relationship; the hypothesis that is tested directly.

Alternative hypothesis (H₁)

The research hypothesis that contradicts the null hypothesis; it is supported if the null is rejected.

p-value

The probability of obtaining a test statistic at least as extreme as the observed one, assuming the null hypothesis is true.

Significance level (α)

The threshold below which the p-value is considered statistically significant, typically set at 0.05.

Test statistic

A value computed from sample data that summarizes the evidence against the null hypothesis; its distribution under H₀ is known.

Type I error

Rejecting the null hypothesis when it is actually true (false positive).

Type II error

Failing to reject the null hypothesis when it is actually false (false negative).

Statistical power

The probability of correctly rejecting a false null hypothesis (1 − β, where β is the Type II error rate).