Evolutionarily stable strategy

An evolutionarily stable strategy is defined in the context of a symmetric game where individuals interact pairwise. A strategy I is an ESS if, for any alternative strategy J, either E(I,I) > E(J,I) or E(I,I) = E(J,I) and E(I,J) > E(J,J), where E(A,B) is the payoff to an individual playing A against an individual playing B. The first condition ensures that I is a strict best response to itself; the second ensures that if J does equally well against I, it does worse against itself, preventing invasion. This definition implies that an ESS is a Nash equilibrium, but the converse is not true: some Nash equilibria are not evolutionarily stable because they can be invaded by neutral mutants. The concept applies to both pure and mixed strategies, where a mixed strategy is a probability distribution over pure actions.

The Hawk-Dove game is the canonical example. In this model, individuals contest a resource of value V; Hawks fight and risk injury (cost C), while Doves display but retreat. If V > C, pure Hawk is an ESS; if V < C, a mixed strategy with probability V/C of playing Hawk is evolutionarily stable. This explains the maintenance of ritualized aggression in many species. Another classic is the war of attrition, where individuals compete by persisting in a display; the ESS is a mixed strategy with an exponential distribution of persistence times. The sex ratio is also an ESS: in a large panmictic population, a 1:1 ratio is stable because any deviation is invaded by mutants producing more of the rare sex. These models have been applied to real organisms, such as dung flies and side-blotched lizards, where alternative male mating strategies coexist in stable frequencies.

ESS theory has been extended beyond pairwise interactions to include multiplayer games, spatial structure, and finite populations. In finite populations, the concept of an ESS is replaced by an evolutionarily stable state, which accounts for stochastic effects and the probability of fixation. The theory has been applied to the evolution of cooperation, where strategies like Tit-for-Tat can be ESS in repeated games, and to signaling theory, where costly signals are stable because they are honest. In behavioral ecology, ESS models explain the evolution of alternative reproductive tactics, such as sneaker males in fish, and the timing of dispersal in plants. The concept has also influenced economics and social science, where it provides a dynamic foundation for equilibrium selection in games.

Beyond the standard examples, ESS theory has subtle implications. For instance, an ESS can be a mixed strategy that is not a Nash equilibrium of the one-shot game but arises in a dynamic context. The concept of a continuously stable strategy (CSS) extends ESS to continuous traits, where the strategy set is a continuum and stability requires convergence in trait space. ESS theory has been applied to the evolution of language, where the emergence of communication systems can be modeled as an ESS. Also, the idea of an ESS has been used to explain the maintenance of genetic polymorphism, as in the case of the side-blotched lizard, where three male color morphs form a rock-paper-scissors cycle. The original paper by Maynard Smith and Price was inspired by the observation of conventional fighting in animals, which seemed to contradict the idea of relentless competition.

Nash equilibrium

A set of strategies where no player can improve by unilaterally changing their strategy.

Hawk-Dove game

A game theory model of conflict where individuals choose between aggressive and peaceful tactics.

Mixed strategy

A strategy that involves randomizing over pure actions according to a probability distribution.

Continuously stable strategy

An extension of ESS to continuous traits, requiring convergence stability.