Conjugate prior

A conjugate prior is defined relative to a likelihood function: if the prior density p(θ) and the likelihood p(x|θ) combine via Bayes' theorem to give a posterior p(θ|x) that has the same functional form as the prior, then the prior is conjugate to that likelihood.¹ For example, a Beta prior combined with a Binomial likelihood yields a Beta posterior; a Gamma prior with a Poisson likelihood yields a Gamma posterior. This property holds for any member of the exponential family of distributions, where a natural conjugate prior exists.² The concept was formalized by Harold Jeffreys in the 1930s, who noted that such priors lead to tractable posteriors.³

Common conjugate pairs include: Beta-Binomial for proportions, Gamma-Poisson for rates, Normal-Normal for means, and Dirichlet-Multinomial for categorical data.⁴ These pairs appear in fields from clinical trials to machine learning. In Bayesian hierarchical models, conjugate priors allow Gibbs sampling to be implemented efficiently, as each conditional posterior is known in closed form.⁵ In online learning, they enable sequential updating without storing all past data, as the posterior becomes the new prior. The use of conjugate priors is also common in empirical Bayes methods, where hyperparameters are estimated from data.

Beyond the standard pairs, conjugate priors exist for less common likelihoods, such as the Pareto distribution with a Pareto prior, and the uniform distribution with a Pareto prior. The concept extends to matrix-variate distributions, e.g., the inverse-Wishart prior for covariance matrices in multivariate normal models. A subtle point is that conjugate priors are not unique; any prior that yields a posterior in the same family qualifies, and mixtures of conjugate priors are also conjugate, though with increased complexity.⁶ Historically, the idea predates Jeffreys: Thomas Bayes himself used a uniform prior for a binomial parameter, which is a special case of a Beta prior. In non-parametric Bayesian methods, conjugate priors appear in Dirichlet process mixtures, where the base distribution is often chosen to be conjugate to the likelihood for computational tractability.⁷

Conjugate priors are not always appropriate; they can be overly restrictive, forcing the prior to have a specific parametric form that may not reflect true prior knowledge.¹ For complex models, such as those with hierarchical or non-linear structures, conjugate priors may not exist or may lead to poor posterior approximations. In such cases, Markov chain Monte Carlo (MCMC) methods or variational inference are used instead. However, conjugate priors remain a cornerstone of Bayesian analysis due to their elegance and computational efficiency, and they are often used as building blocks in more complex models.

Prior distribution

A probability distribution representing uncertainty about a parameter before observing data.

Posterior distribution

The updated distribution of a parameter after incorporating observed data via Bayes' theorem.

Likelihood function

A function that quantifies the probability of the observed data given a set of parameter values.

Exponential family

A class of probability distributions with a specific functional form that includes many common distributions.

Hyperparameter

A parameter of a prior distribution that controls its shape.