A/B testing

A/B testing is a controlled experiment that compares two versions of a variable (e.g., a webpage, email, or feature) to determine which performs better on a predefined metric. Users are randomly assigned to either the control (A) or treatment (B) group, and the difference in outcomes is analyzed for statistical significance.¹ The method relies on the scientific method: hypothesis formulation, experiment design, data collection, and inference. Key principles include random assignment to eliminate selection bias, a single variable change to isolate its effect, and pre-specification of the primary metric to avoid cherry-picking.

Proper A/B testing requires careful planning. Sample size is determined by the desired statistical power, significance level (usually 0.05), and minimum detectable effect. The test statistic is often a difference in means or proportions, analyzed via t-test or chi-squared test.² Sequential testing and Bayesian approaches are alternatives that allow early stopping. Common pitfalls include peeking at results before the planned end, multiple comparisons, and Simpson's paradox, where aggregated data reverses a trend seen in subgroups.

A/B testing is widely used in e-commerce to optimize conversion rates, in marketing to test ad copy and email subject lines, and in product development to evaluate new features. For example, Netflix uses A/B testing to refine its recommendation algorithms and artwork.³ In healthcare, it is used in clinical trials for treatment efficacy, though with stricter ethical and regulatory oversight. The methodology also extends to public policy, where randomized controlled trials inform program design.

The roots of A/B testing trace back to agricultural experiments by Ronald Fisher in the 1920s, who developed the principles of randomization and analysis of variance.⁴ The term 'A/B testing' emerged with the internet era; early web experiments were conducted by companies like Amazon in the late 1990s. The field has evolved with the rise of big data, leading to more sophisticated multi-armed bandit algorithms and continuous testing platforms.

Beyond the basics, A/B testing has niche applications. In web performance, it is used to test the impact of page load times on user engagement. In email marketing, it can test different send times. A lesser-known variant is 'server-side testing,' which allows for more complex experiments without client-side flicker. Another edge case is 'overlapping experiments,' where multiple tests run simultaneously, requiring careful coordination to avoid interference. Additionally, the 'winner's curse' can occur when the best-performing variant is chosen based on limited data, leading to overestimation of its effect.

Control group

The group exposed to the existing version (A) in an A/B test.

Treatment group

The group exposed to the new version (B) in an A/B test.

Statistical significance

The probability that the observed difference is not due to chance, typically p < 0.05.

Power

The probability of detecting a true effect if it exists; typically set to 0.80.

Minimum detectable effect

The smallest effect size that the test is designed to detect.