Entropy (information theory)

Entropy measures the average uncertainty of a random variable. For a discrete variable X with outcomes x and probabilities p(x), Shannon entropy is H(X) = −Σ p(x) log₂ p(x).¹ The logarithm base determines the unit: base 2 gives bits, base e gives nats, and base 10 gives hartleys. A certain event has zero entropy, while a uniform distribution over n equally possible outcomes has the greatest entropy, log₂ n.

The quantity can be understood operationally: it is the minimum average number of binary questions needed to identify outcomes in an efficiently coded long sequence, subject to the usual coding assumptions. Rare outcomes carry more information when they occur, because their individual surprise is −log₂ p(x); entropy is the probability-weighted average of that surprise.

Entropy is additive for independent variables and obeys a chain rule for joint distributions. If X and Y are independent, H(X,Y) = H(X) + H(Y); more generally, H(X,Y) = H(X) + H(Y|X), where conditional entropy measures the remaining uncertainty about Y after observing X.²

The difference between uncertainty before and after observation is captured by mutual information, I(X;Y) = H(X) − H(X|Y). Mutual information is symmetric and cannot be negative. A related comparison between two distributions is Kullback–Leibler divergence, which measures the inefficiency or discrepancy incurred when one distribution is represented using another; unlike entropy, it is not generally symmetric and is not a distance in the mathematical sense.

Entropy sets a fundamental limit on lossless data compression. For a source with entropy H(X), no uniquely decodable code can have an average length below the entropy, while suitable codes approach it for long sequences.¹ Huffman coding approaches this bound for many finite-alphabet sources, and arithmetic coding can approach it more closely by representing an entire sequence rather than assigning an integral-length codeword to every symbol.

In communication, source entropy concerns uncertainty in the message, whereas channel capacity concerns the maximum reliable information rate through a noisy channel. Shannon’s noisy-channel theorem separates these roles: rates below capacity can, in principle, be protected by coding with arbitrarily small error, while rates above capacity cannot.² Entropy therefore underlies both compression and error-correcting communication.

Entropy depends on the modeled distribution, not solely on the observed alphabet. The same possible outcomes can have very different entropies when their probabilities differ, and estimating entropy from a small sample is subject to bias, especially when many outcomes are rare.³

Several extensions serve different purposes. Rényi entropies form a family indexed by an order parameter and include Shannon entropy as a limiting case.⁴ Differential entropy applies to continuous variables, but it can be negative and changes under coordinate transformations, so it is not directly analogous to discrete entropy; relative entropy and mutual information retain more robust interpretive properties in continuous settings. Entropy also appears in statistical inference through maximum-entropy methods, where the least-committal distribution is selected subject to specified constraints.⁵

Information-theoretic entropy should not be confused with thermodynamic entropy, although the two share mathematical structures and historical connections. Shannon introduced the term in his 1948 theory of communication after John von Neumann reportedly suggested it, partly because the quantity had useful formal parallels with statistical mechanics.¹

The word “entropy” also appears in algorithmic information theory, where complexity concerns the length of the shortest description of an individual object rather than the average uncertainty of a probability distribution. In this article, entropy means the Shannon measure and its standard probabilistic generalizations used in information theory.

Shannon entropy

The expected self-information of a random variable, usually measured in bits when logarithms are base 2.

Self-information

The surprise of an outcome with probability p, defined as −log p.

Conditional entropy

The average uncertainty remaining about one variable after another variable is known.

Mutual information

The reduction in uncertainty about one variable obtained from observing another.

Relative entropy

Another name for Kullback–Leibler divergence, a directed comparison of probability distributions.

Differential entropy

The continuous-variable analogue of an entropy formula, with transformation-dependent behavior.