Statistical hypothesis test

The procedure begins with two competing hypotheses: the null hypothesis (H₀), typically a statement of no effect or no difference, and the alternative hypothesis (H₁ or Hₐ), which contradicts H₀. The test computes a test statistic from the sample data, and the p-value is the probability of obtaining a statistic as extreme as the observed one, assuming H₀ is true.² If the p-value is less than a pre-specified significance level α (commonly 0.05), the result is deemed statistically significant, and H₀ is rejected in favor of H₁.

This logic is analogous to a criminal trial: the null hypothesis is “innocent until proven guilty,” and the evidence (data) must be strong enough to convict (reject H₀) beyond a reasonable doubt (α).³

The roots of hypothesis testing lie in the 18th century with John Arbuthnot’s 1710 analysis of sex ratios, but the modern framework emerged in the early 20th century. Ronald Fisher introduced the concept of the null hypothesis and the p-value in the 1920s, advocating for significance testing as a means of inductive inference.⁴ In 1933, Jerzy Neyman and Egon Pearson formalized the decision-theoretic approach, introducing Type I and Type II errors and the concept of power.⁵ The two schools—Fisherian significance testing and Neyman-Pearson hypothesis testing—were later merged into the hybrid procedure commonly taught today, though the philosophical differences remain a subject of debate.

Two types of errors can occur: Type I error (rejecting a true H₀) and Type II error (failing to reject a false H₀). The probability of a Type I error is α, while the probability of a Type II error is β. The power of a test, 1−β, is the probability of correctly rejecting a false H₀. Power depends on the effect size, sample size, significance level, and the variability of the data.⁶ Researchers often conduct power analyses before collecting data to ensure their study is sensitive enough to detect meaningful effects.

Hypothesis tests are ubiquitous in scientific research. Common examples include the t-test for comparing means, the chi-squared test for categorical data, the F-test for variances, and the Mann-Whitney U test for non-parametric comparisons.⁷ In medicine, randomized controlled trials use hypothesis tests to evaluate treatment efficacy; in psychology, they underpin behavioral experiments; in economics, they assess policy impacts. In particle physics, the discovery of the Higgs boson in 2012 required a significance of 5σ, corresponding to a p-value of about 3×10⁻⁷, to avoid false claims.⁸

The p-value is frequently misunderstood. A common error is interpreting it as the probability that H₀ is true, or as the probability that the result occurred by chance alone. In fact, the p-value is conditional on H₀ being true and does not quantify the probability of the hypothesis itself.² The American Statistical Association issued a statement in 2016 warning against the misuse of p-values, and many journals now require effect sizes and confidence intervals alongside p-values.⁹ The “replication crisis” in psychology and other fields has highlighted the problem of p-hacking and publication bias, leading to calls for more robust statistical practices.

Beyond the standard framework, there are several niche but important topics. Sequential analysis, developed by Abraham Wald, allows for interim analyses in clinical trials, stopping early if results are conclusive, which is efficient but requires adjusted significance levels.¹⁰ Bayesian hypothesis testing offers an alternative that incorporates prior beliefs and produces a Bayes factor, which quantifies evidence for H₁ over H₀.¹¹ Randomization tests, also known as permutation tests, make no distributional assumptions and are computationally intensive but exact. In high-energy physics, the CLs method is used to set upper limits on signals, a technique that differs from conventional frequentist tests. Additionally, the Behrens-Fisher problem—comparing means when variances are unequal—has no exact solution, leading to approximations like Welch’s t-test. Finally, the multiple comparisons problem arises when testing many hypotheses simultaneously, requiring corrections such as the Bonferroni or Benjamini-Hochberg procedures to control the family-wise error rate or false discovery rate.¹²

Null hypothesis (H₀)

The default assumption that there is no effect or no difference; the hypothesis to be tested.

Alternative hypothesis (H₁)

The hypothesis that contradicts the null; often what the researcher wants to support.

p-value

The probability of obtaining test results at least as extreme as the observed results, assuming the null hypothesis is true.

Significance level (α)

The threshold for rejecting the null hypothesis, typically set at 0.05.

Type I error

Rejecting a true null hypothesis (false positive).

Type II error

Failing to reject a false null hypothesis (false negative).

Power

The probability of correctly rejecting a false null hypothesis; 1 − β.

Effect size

A quantitative measure of the magnitude of the effect under study, independent of sample size.