Normalizing flows

Normalizing flows construct a complex probability distribution by transforming a simple base distribution through an invertible map. If z follows a tractable density, such as a standard Gaussian, and x = f(z), the change-of-variables rule gives the density of x from the base density and the determinant of the Jacobian of f.¹

The defining trade-off is between expressiveness and tractability: the transformation must be rich enough to model complicated data, but its inverse and Jacobian determinant must remain computationally manageable. This makes flows useful for both generation and density estimation. Sampling runs the transformation forward from latent noise, while inference maps observations back into latent space through the inverse.

Unlike many implicit generative models, flows can provide a normalized probability density rather than only a procedure for producing samples.

Flow architectures make the Jacobian determinant easy to compute by organizing transformations into structured layers. Coupling layers leave part of a vector unchanged while transforming the remainder using parameters predicted from the unchanged part; their triangular Jacobians make determinant calculation efficient.²

Autoregressive flows use conditional transformations whose parameters depend on preceding variables. Masked autoregressive flow favors fast density evaluation, whereas inverse autoregressive flow favors fast sampling, so the direction of computational efficiency matters in practice.⁴ Real NVP and Glow expanded the approach with affine coupling, multiscale designs, invertible reshaping, and learned channel-mixing operations.²³

Models are commonly trained by maximizing the log-likelihood of observed data, combining the base log-density with the summed log-determinants of the transformations.

Normalizing flows are used for density estimation, sample generation, variational inference, anomaly detection, and probabilistic modeling in areas such as images, audio, physical simulation, and scientific inference.¹ Their invertible latent representation also supports exact reconstruction and likelihood-based comparison between observations.

The method has important constraints. An exactly invertible map ordinarily preserves dimensionality, so modeling data on a lower-dimensional manifold requires dequantization, augmented variables, specialized priors, or another workaround. Architectural restrictions imposed to keep Jacobians tractable can also reduce efficiency or expressive power. High-dimensional image models may require substantial memory, and likelihood scores do not always align with perceptual quality or semantic usefulness.

These trade-offs have encouraged hybrid systems that combine flows with autoregressive networks, diffusion methods, latent-variable models, or continuous-time dynamics.

Continuous-time normalizing flows replace a finite sequence of layers with an ordinary differential equation whose vector field transports probability mass. The instantaneous change-of-variables formula uses the divergence of that field, allowing flexible transformations but requiring numerical integration and often repeated trace estimation.⁵

Earlier flow research includes non-volume-preserving transformations and independent component analysis, while later work formalized a broad taxonomy covering coupling, autoregressive, residual, and continuous flows.^{16 Residual flows can use invertible residual blocks, but guaranteeing invertibility may require Lipschitz constraints that affect optimization and computational cost.}

Flows are also valuable as proposal distributions: a learned transformation can convert a difficult posterior or target density into a simpler latent representation for importance sampling, Bayesian inference, or simulation-based scientific analysis.

Base distribution

A simple, tractable latent distribution, commonly a standard Gaussian, that is transformed into the modeled data distribution.

Change of variables

The probability rule that combines a base density with the absolute Jacobian determinant of an invertible transformation.

Coupling layer

An invertible layer that keeps one subset of variables fixed while transforming another subset conditionally.

Jacobian determinant

A measure of local volume change under a transformation; its logarithm contributes to the flow's log-likelihood.

Autoregressive flow

A flow whose transformation parameters for each variable depend on earlier variables under a chosen ordering.

Continuous normalizing flow

A flow defined by a continuous-time differential equation rather than a finite stack of discrete transformations.