Naive Bayes classifier

Naive Bayes classifies an observation by comparing posterior probabilities for its possible classes. Bayes’ theorem combines a prior probability for each class with the likelihood of the observed features under that class: P(class|features) is proportional to P(class) multiplied by P(features|class).¹ The “naive” qualification refers to the simplifying assumption that features are conditionally independent once the class is known. Under that assumption, the joint likelihood becomes a product of individual feature likelihoods, making estimation especially efficient. The classifier is usually trained by counting feature occurrences or estimating simple distribution parameters, rather than optimizing a large set of interacting weights. It is therefore a generative classifier: it models class-conditional data distributions and uses them to make predictions.

Naive Bayes variants differ mainly in how they represent feature values. Gaussian Naive Bayes models continuous features with a class-specific normal distribution; multinomial Naive Bayes is commonly used for count data such as word frequencies; and Bernoulli Naive Bayes represents binary feature presence or absence.² Categorical implementations model discrete outcomes directly, while complementary Naive Bayes modifies multinomial estimation for some imbalanced text problems. Training generally requires a single pass through the data to calculate class priors and feature statistics, giving the method low computational and memory costs. Laplace or other additive smoothing prevents unseen feature values from forcing an entire class likelihood to zero. Logarithms are normally used during prediction so that many small probabilities can be added instead of multiplied.

Naive Bayes is strongest when fast, interpretable baseline performance matters more than modeling complex feature interactions. It has long been applied to spam filtering, document and news classification, language identification, sentiment analysis, and other high-dimensional text tasks.³ The independence assumption is often unrealistic: words, measurements, or behavioral signals can be correlated even within the same class. Nevertheless, classification accuracy can remain good when the resulting decision boundaries are useful, because accurate posterior probabilities are not always necessary for accurate labels. Probability estimates themselves may be poorly calibrated, especially when features are dependent, so calibration should be assessed when probabilities drive decisions.⁴ Naive Bayes also cannot natively capture sequential structure, rich interactions, or latent relationships without feature engineering or a different model.

Naive Bayes can remain effective in regimes where the number of features greatly exceeds the number of training examples. Its compact sufficient statistics make incremental or online updates practical, provided the data-generating assumptions and class definitions remain stable. In text classification, multinomial and Bernoulli formulations are not interchangeable: one uses counts, while the other emphasizes whether a term appears at all.² Class priors can materially affect predictions, particularly with rare categories, and should reflect the training design or the intended deployment population rather than being accepted uncritically. Missing values, correlated duplicate features, distribution shift, and extreme class imbalance can each distort the estimated likelihoods. Despite its simplicity, the method is also useful diagnostically: its feature statistics can reveal which observations or terms distinguish one class from another.

Bayes’ theorem

A rule for updating the probability of a hypothesis using prior probability and observed evidence.

Conditional independence

An assumption that features are independent of one another after conditioning on the class.

Prior probability

The estimated probability of a class before the current features are observed.

Smoothing

A parameter-estimation technique that assigns nonzero probability to previously unseen feature values.

Calibration

The degree to which predicted probabilities correspond to observed outcome frequencies.