Other meanings of Variational autoencoder
Generative model
A variational autoencoder (VAE) is a generative neural-network model, introduced by Diederik P. Kingma and Max Welling in 20131, that learns a probabilistic mapping between a high-dimensional data space and a lower-dimensional latent space. VAEs combine variational inference with deep learning, enabling the generation of new data samples by sampling from the learned latent distribution.
The VAE consists of two neural networks: an encoder that maps an input data point 𝔹 to parameters of a latent variable distribution 𝔹 (typically a Gaussian), and a decoder that reconstructs the data from a sample of the latent variable1. The encoder outputs a mean һ and variance 𝜎², which define the approximate posterior ⽅(𝔹|𝔹). The decoder defines the likelihood ⽅(𝔹|𝔹). Unlike traditional autoencoders, the VAE imposes a prior distribution ⽅(𝔹) (usually a standard normal) on the latent space, forcing the learned representations to be smooth and continuous2. This structure enables the model to generate new data by sampling from the prior and decoding, a process that is not possible with deterministic autoencoders.
VAEs are trained by maximizing the evidence lower bound (ELBO) on the log-likelihood of the data. The ELBO consists of two terms: the reconstruction loss (expected log-likelihood of the data given the latent variable) and the Kullback–Leibler (KL) divergence between the approximate posterior and the prior3. The KL divergence acts as a regularizer, encouraging the encoder to produce latent distributions close to the prior. A key technical innovation is the reparameterization trick, which makes the sampling operation differentiable by expressing the latent sample as 𝔹 = һ + 𝜎 * 𝜀, where 𝜀 is a random noise variable1. This allows gradient-based optimization via backpropagation through the encoder and decoder. The ELBO can be optimized using stochastic gradient descent, making VAEs scalable to large datasets.
VAEs have been applied to image generation, data compression, anomaly detection, and semi-supervised learning4. Notable variants include the conditional VAE (CVAE), which conditions the generation on auxiliary information, and the β-VAE, which increases the weight of the KL term to enforce stronger disentanglement of latent factors5. Other extensions incorporate autoregressive decoders (e.g., VQ-VAE) or hierarchical latent structures. VAEs are also used in drug discovery to generate molecular structures, and in reinforcement learning for planning and exploration6. Despite their versatility, VAEs often produce blurrier samples than generative adversarial networks (GANs), a trade-off arising from the ELBO objective.
One lesser-known challenge is posterior collapse, where the decoder learns to ignore the latent variable, causing the KL divergence to vanish and the model to degenerate into a vanilla autoencoder7. This issue is common with strong decoders like autoregressive models. Another subtlety is the VAE's relationship to the Helmholtz machine and the wake-sleep algorithm, which the reparameterization trick largely superseded1. The choice of prior and posterior family (e.g., Gaussian vs. flow-based) significantly affects the quality of the learned representation. Additionally, the standard VAE objective can be interpreted as a form of variational Bayesian inference on a latent variable model, and extensions like the VAE-GAN hybrid combine the adversarial loss with the ELBO to improve sample quality4. The VAE's theoretical foundation in variational inference makes it a principled tool for learning probability distributions, but practical implementation details such as annealing the KL term are often critical for successful training.
The reparameterization trick is crucial for training VAEs with backpropagation.
Help improve the encyclopedia. Reports go straight to the site manager.