Evidence lower bound

The ELBO is a lower bound on the log marginal likelihood of observed data. For latent variables z, observations x, a joint model p(x,z), and an approximating distribution q(z), it is defined as L(q) = E_q(z)[log p(x,z) − log q(z)]. The name “evidence” refers to p(x), the probability or density assigned to the observations after integrating out z.

The bound follows from the exact decomposition log p(x) = L(q) + KL(q(z) || p(z|x)).¹ Because the Kullback–Leibler divergence is nonnegative, L(q) cannot exceed log p(x). Equality holds when q(z) equals the true posterior p(z|x) almost everywhere. Thus, maximizing the ELBO simultaneously seeks a good posterior approximation and a high probability for the observed data.

ELBO optimization replaces direct posterior inference with tractable or approximately tractable expectations. In mean-field variational inference, q is restricted to a factorized family, such as q(z) = ∏_jq_j(z_j), and coordinate updates optimize one factor while holding the others fixed. Modern implementations often use gradient methods, automatic differentiation, and stochastic estimates based on minibatches.

The objective can also be written as E_q[log p(x|z)] − KL(q(z)||p(z)). This form separates expected data fit from a complexity penalty determined by the prior. In variational autoencoders, the reparameterization trick expresses samples from q using parameter-free noise, allowing lower-variance gradients through continuous latent variables.² Discrete variables, nonconjugate models, and large data sets may require score-function estimators, control variates, or specialized relaxations.

The ELBO is both an inference criterion and a model-fitting criterion. When q is optimized for each parameter setting, its value approximates the log evidence used in Bayesian model comparison; unlike maximum likelihood, it accounts for uncertainty in latent variables and can discourage unnecessarily flexible explanations.¹

Its behavior depends on the variational family and on how expectations are estimated. A restricted family can make the bound loose even when optimization is exact, while a flexible family can make optimization difficult. Importance-weighted objectives replace one sample-based bound with a tighter multi-sample bound, although tighter does not always mean easier optimization or better learned representations.³ In practice, reported ELBOs are comparable only when likelihood conventions, data scaling, and constant terms are handled consistently.

The ELBO gap measures posterior-approximation error, not merely numerical optimization error. The decomposition distinguishes a poor variational family from failure to reach the best point within that family, a distinction that can be obscured when only the objective value is reported.

Several less visible choices alter the objective’s meaning. Mini-batch estimates usually rescale the data-fit term to approximate the full-data ELBO, whereas annealed or beta-weighted objectives deliberately change the balance between reconstruction and regularization and are therefore not ordinary ELBOs. In hierarchical models, local latent variables may be integrated out or optimized variationally, producing different bounds. The bound is also sensitive to parameterization: two distributions representing the same density can yield different gradient variance even though their exact ELBOs agree. These details matter in variational autoencoders, topic models, Gaussian processes, and Bayesian neural networks.⁴

Marginal likelihood

The probability or density of observed data after integrating or summing over latent variables and parameters; it is also called model evidence.

Variational distribution

A tractable distribution q chosen to approximate an otherwise difficult posterior distribution.

Kullback–Leibler divergence

A nonnegative measure of the discrepancy between two probability distributions, equal to zero when they agree almost everywhere.

Reparameterization trick

A transformation that represents a random variable as a deterministic function of parameters and parameter-free noise, enabling pathwise gradient estimates.