Other meanings of Variational Bayesian methods
Bayesian statistics
Variational Bayesian methods are a family of techniques for approximating intractable posterior distributions in Bayesian inference by reformulating inference as an optimization problem. Rather than sampling from the posterior (as in Markov chain Monte Carlo), they seek a simpler distribution that minimizes the Kullback–Leibler divergence to the true posterior, typically by maximizing the evidence lower bound (ELBO). This approach is especially useful for large-scale models and complex data where exact inference is computationally infeasible. Variational methods have become a cornerstone of modern machine learning, enabling scalable inference in deep generative models, topic models, and probabilistic programming.
Variational Bayesian methods recast Bayesian inference as an optimization problem by introducing a family of candidate distributions q(θ) over the latent variables θ and parameters. The goal is to find the member of that family that is closest to the true posterior p(θ|x) in terms of Kullback–Leibler divergence. Because the KL divergence itself is intractable, practitioners maximize the evidence lower bound (ELBO), which is equivalent to minimizing the KL divergence plus a constant. The ELBO decomposes into an expected log-likelihood term minus the KL divergence between q(θ) and the prior p(θ). This formulation turns inference into a tractable optimization that scales well with data size and model complexity.1
A common choice for the variational family is the mean-field approximation, which assumes the posterior factors as q(θ) = ∏i qi(θi). Under this assumption, the ELBO can be optimized using coordinate ascent, updating each factor in turn while keeping others fixed. The resulting update equations often have closed-form solutions when the model uses conjugate exponential families. For non-conjugate models, black-box variational inference uses stochastic gradients of the ELBO, enabling application to arbitrary differentiable models.2 The ELBO itself is a lower bound on the log marginal likelihood, and its maximization automatically considers the trade-off between fitting the data and staying close to the prior.
Variational Bayesian methods have been widely applied in latent Dirichlet allocation, where they replace collapsed Gibbs sampling, and in deep learning through the variational autoencoder (VAE), which uses a neural network to parameterize the variational distribution. Other notable variants include stochastic variational inference (SVI), which uses mini-batches of data to scale to massive datasets, and amortized inference, where the same parameters are shared across data points. The approach has also been extended to structured variational families, such as Gaussian processes and normalizing flows, to capture posterior dependencies more accurately.3 In probabilistic programming, variational methods enable fast inference engines like Pyro and Stan's variational module.
While now mainstream, variational Bayesian methods have roots in statistical physics, where mean-field approximations were used to study spin systems. The first explicit formulation for Bayesian inference appeared in the 1990s, notably in the work of Hinton and van Camp on ensemble learning and in the textbook by Neal. The term "variational Bayes" was popularized by Jordan et al. (1999).4 A lesser-known connection is that variational inference is closely related to expectation propagation (EP), which minimizes the reverse KL divergence instead. Another edge case: the ELBO can be negative, and its absolute value is not a meaningful measure of model fit. Variational methods can also be used for model selection via the Bayesian information criterion approximation, but they tend to underestimate posterior variance, a phenomenon known as "over-confidence".
Variational Bayesian methods are distinct from variational methods in calculus, though both share the idea of optimizing a functional.
Help improve the encyclopedia. Reports go straight to the site manager.