Empirical process

The empirical process is built from the empirical distribution function, which assigns mass 1/n to each observation. For a sample X₁, …, Xₙ from a distribution F, the empirical distribution function is Fₙ(t) = (1/n) Σᵢ 1{Xᵢ ≤ t}. The empirical process Gₙ(t) = √n (Fₙ(t) − F(t)) measures the pointwise fluctuation of Fₙ around F, scaled by √n to yield a non-degenerate limit. At each fixed t, the central limit theorem implies that Gₙ(t) converges in distribution to a normal random variable with mean 0 and variance F(t)(1−F(t)). However, the process as a whole exhibits much richer behavior, and its study requires functional analysis and stochastic calculus.

The Glivenko–Cantelli theorem, proved in 1933, states that the supremum of |Fₙ(t) − F(t)| converges almost surely to 0, establishing uniform consistency of the empirical distribution function. This result is a cornerstone of nonparametric statistics, justifying the use of empirical distributions as estimates of the true distribution. In 1952, Monroe D. Donsker proved a functional central limit theorem, showing that the empirical process converges weakly to a Brownian bridge in the space of càdlàg functions. This result, known as Donsker's theorem, extends the finite-dimensional convergence to the entire process, enabling the derivation of asymptotic distributions for statistics such as the Kolmogorov–Smirnov statistic and the Cramér–von Mises statistic.

The empirical process underpins many statistical methods, including goodness-of-fit tests, bootstrap procedures, and empirical likelihood. It also generalizes to more complex settings, such as empirical processes indexed by classes of functions, which are central to modern machine learning theory and empirical process theory. The concept of Vapnik–Chervonenkis (VC) dimension arises from the need to control the supremum of empirical processes over function classes, leading to uniform laws of large numbers and uniform central limit theorems. These tools are essential for analyzing the behavior of estimators in high-dimensional and nonparametric models, and they have applications in fields ranging from econometrics to biostatistics.

Beyond the classical results, the empirical process has several subtle and less-known facets. For instance, the empirical process converges to a Brownian bridge only under the assumption of a continuous distribution; for discrete distributions, the limit is a tied-down process with jumps. The rate of convergence in the Glivenko–Cantelli theorem is governed by the law of the iterated logarithm, which gives a precise almost-sure bound of order √(log log n / n). Another niche area is the study of empirical processes for dependent data, such as time series, where the limit is a Gaussian process with a covariance structure that depends on the dependence. Additionally, the empirical process has been used to construct confidence bands for distribution functions, and its multivariate extensions are used in copula modeling.

Empirical distribution function

The cumulative distribution function that puts mass 1/n at each observed data point.

Brownian bridge

A Gaussian process with covariance function s(1−t) for s ≤ t, which arises as the limit of the empirical process.

Donsker's theorem

A functional central limit theorem stating that the empirical process converges weakly to a Brownian bridge.

Glivenko–Cantelli theorem

A theorem stating that the empirical distribution function converges uniformly to the true distribution function almost surely.

VC dimension

A measure of the capacity of a class of functions, used to control uniform convergence of empirical processes.