Variational inference

Variational inference approximates a posterior distribution p(z|x) by a simpler distribution q(z) from a family Q, minimizing the Kullback-Leibler divergence KL(q(z)||p(z|x)). Because the true posterior is intractable, VI maximizes the evidence lower bound (ELBO), which equals the log marginal likelihood minus that KL divergence. The ELBO is given by E_q[log p(x,z)] - E_q[log q(z)], and its maximization is equivalent to minimizing the KL divergence.¹

The optimization is typically performed via coordinate ascent (CAVI) or gradient-based methods. For exponential family models, CAVI yields closed-form updates, making VI fast and deterministic, unlike MCMC which relies on stochastic sampling.

The most common variational family is the mean-field approximation, which assumes q(z) factorizes as a product of independent distributions over each latent variable. This simplification reduces the problem to solving a series of one-dimensional optimizations, but it ignores posterior correlations, which can lead to underestimation of uncertainty.²

More expressive families include structured variational approximations that preserve dependencies, and normalizing flows, which apply invertible transformations to a simple base distribution to capture complex correlations. These methods improve accuracy at the cost of increased computational complexity.

VI is a key component in variational autoencoders (VAEs), where it approximates the posterior of latent variables in deep generative models. The VAE, introduced by Kingma and Welling in 2013, uses a neural network to parameterize the variational distribution and optimizes the ELBO via stochastic gradient descent.³

Beyond VAEs, VI is used in Bayesian neural networks for uncertainty quantification, in topic models like latent Dirichlet allocation, and in reinforcement learning for policy optimization. Its scalability to large datasets via stochastic variational inference (SVI) has made it a standard tool in probabilistic programming frameworks such as Stan, Pyro, and TensorFlow Probability.

VI provides a lower bound on the marginal likelihood, which is useful for model comparison. However, it tends to be overconfident, producing posterior approximations that are narrower than the true posterior, especially when the true posterior is multimodal.⁴

Recent theoretical work has analyzed the consistency and asymptotic normality of variational estimators, showing that under certain conditions they converge to the true posterior as sample size grows. Yet, the approximation error can be significant in high-dimensional settings, and there is active research on improving the fidelity of variational approximations.

Variational inference has roots in statistical physics, where the mean-field approximation was used to study spin glasses and other many-body systems. The connection was formalized in the 1980s by physicists like Parisi, who used variational methods for free energy minimization.

An early application of VI was in the analysis of longitudinal data using mixed-effects models, where it provided a faster alternative to numerical integration. Also, VI has been applied to approximate the posterior in Gaussian process models, enabling scalable Bayesian optimization.

A notable edge case is the use of VI for model selection via the ELBO, which can be biased when the variational family is misspecified. Additionally, the 'amortized' VI used in VAEs shares parameters across data points, which can lead to amortization gap—the difference between the optimal per-data-point posterior and the amortized approximation.

ELBO

Evidence lower bound; the objective function maximized in variational inference.

Mean-field approximation

A variational family that assumes independence among latent variables.

Normalizing flows

A class of invertible transformations used to build flexible variational distributions.

Amortized inference

A technique where a neural network predicts variational parameters for each data point, sharing computation.