Autoregressive model

An autoregressive model of order p (AR(p)) is defined as X_t = c + φ₁X_t−1 + φ₂X_t−2 + … + φ_pX_t−p + ε_t, where c is a constant, φ_i are coefficients, and ε_t is white noise.¹ The model assumes stationarity, meaning the statistical properties (mean, variance) do not change over time. Stationarity imposes constraints on the φ coefficients: the roots of the characteristic polynomial must lie outside the unit circle.² The partial autocorrelation function (PACF) is used to identify the appropriate order p: for an AR(p) process, the PACF cuts off after lag p.³

The most common method for estimating AR coefficients is the Yule–Walker estimator, which solves a system of linear equations derived from the sample autocovariance function.⁴ Alternatively, ordinary least squares (OLS) regression can be applied to the lagged values. Model selection criteria such as the Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC) balance goodness-of-fit against model complexity, penalizing higher orders to avoid overfitting.¹ Residual diagnostics, including tests for autocorrelation (e.g., Ljung–Box test), verify that the residuals resemble white noise. For non-stationary series, differencing is applied before fitting an AR model, leading to the ARIMA framework.

Autoregressive models are used in economics (e.g., GDP growth forecast), climatology (e.g., temperature series), and neuroscience (e.g., EEG analysis).² Extensions include the vector autoregressive (VAR) model for multiple time series, and the threshold autoregressive (TAR) model that allows regime-switching behavior.⁵ In modern machine learning, autoregressive neural networks (e.g., WaveNet, Transformer-based language models) adapt the core idea of conditioning on past outputs, though they are not linear.¹

The term “autoregressive” was coined by Udny Yule in 1927, who applied the model to study sunspot numbers.⁵ The Yule–Walker equations were independently derived by George Udny Yule and Gilbert Walker (1910) in the context of atmospheric pressure.⁴ An edge case: an AR(1) model with coefficient φ=1 (a random walk) is non-stationary, yet it is often used as a starting point for unit root tests. A less known fact: the AR model can be reinterpreted as a linear stochastic difference equation, and its spectral density can be derived from the coefficients, revealing periodicities in the data. In fields like ecology, AR models are used to model population dynamics with lagged density dependence.

AR(p)

Autoregressive model of order p; uses p past values to predict the current value.

Yule–Walker equations

A set of linear equations that relate the autocovariances of a stationary AR process to its coefficients; used for estimation.

Stationarity

A property of a time series where its mean, variance, and autocovariance are constant over time.

Partial autocorrelation function (PACF)

The correlation between X_t and X_{t-k} after removing the effects of intermediate lags; used to identify AR order.