Logistic regression

Logistic regression models the probability that a binary outcome belongs to a designated class. If p is that probability and x is a vector of predictors, the model assumes that log(p/(1−p)) is a linear function of the predictors. The inverse transformation is the logistic or sigmoid function, which keeps fitted probabilities within the interval from 0 to 1.¹

The method serves both classification and explanation. A threshold, often 0.5 but not necessarily so, converts estimated probabilities into class labels; the probabilities themselves can support risk prediction, ranking, or decision analysis. Predictors may be continuous, categorical, or represented by interactions and nonlinear terms. The model is commonly used in epidemiology, medicine, social science, ecology, and machine learning, where the outcome may be disease versus no disease, approval versus rejection, or presence versus absence.

Logistic regression coefficients are usually estimated by maximum likelihood rather than ordinary least squares. The likelihood measures how well the observed binary outcomes agree with the probabilities implied by the proposed coefficients; numerical optimization then finds the parameter values that maximize it.¹

A coefficient describes a change in log-odds while other modeled predictors are held constant. Exponentiating a coefficient gives an odds ratio: a one-unit increase in a predictor multiplies the odds by that value. An odds ratio above one indicates higher odds, while a value below one indicates lower odds; it is not generally the same as a probability ratio or risk ratio. Confidence intervals, likelihood-ratio tests, and predicted probabilities provide complementary accounts of uncertainty and practical effect size. Categorical predictors require a reference category, and centering or scaling continuous predictors can make coefficients easier to interpret without changing fitted predictions.

Reliable logistic regression requires an appropriate binary outcome, independent observations or an explicitly modeled dependence structure, and a sensible relationship between continuous predictors and the log-odds. It does not require normally distributed predictors or equal variances, but influential observations, missing data, omitted interactions, and poorly chosen functional forms can distort estimates.

Model assessment should separate discrimination from calibration. A confusion matrix, sensitivity, specificity, and the area under a ROC curve describe classification performance across thresholds, whereas calibration compares predicted probabilities with observed frequencies. Apparent performance can be optimistic when the same data are used for fitting and evaluation, so cross-validation, bootstrap validation, or a genuinely separate test set is preferable. Multicollinearity can inflate coefficient uncertainty, and a large sample does not automatically correct systematic bias from confounding or measurement error.

Separation is a distinctive edge case: if predictors perfectly or nearly perfectly divide the outcome classes, maximum-likelihood coefficients can diverge and ordinary estimates become unstable. Firth's penalized likelihood provides a bias-reduction approach that can produce finite estimates in many such cases.⁴ Rare outcomes create another problem: conventional estimates may be biased when the event rate is very low, particularly in small samples; specialized corrections and careful sampling designs are available.³

Regularization adds a penalty to discourage overly large coefficients. L1 penalties can set some coefficients exactly to zero, supporting sparse prediction, while L2 penalties generally shrink correlated coefficients together. These approaches improve prediction in high-dimensional settings but alter the interpretation of coefficients. Logistic regression also has extensions for clustered observations, repeated measurements, ordinal outcomes, and more than two classes, but those models should not be confused with the standard independent-observation binary form. The method originated in the broader development of binary response models, with a foundational formulation published by David Cox in 1958.²

Logit

The logarithm of the odds, log(p/(1−p)); logistic regression models the logit as a linear predictor.

Odds ratio

The multiplicative change in odds associated with a one-unit change in a predictor, holding other modeled variables constant.

Separation

A condition in which predictor values perfectly or nearly perfectly distinguish outcome classes, causing ordinary coefficient estimates to become unstable or unbounded.

Calibration

The agreement between predicted probabilities and the observed frequency of outcomes.

Regularization

A penalty added during model fitting to shrink coefficients and reduce overfitting or instability.