Adam (optimization algorithm)

Adam maintains two moving averages for each parameter: the first moment (mean) and the second moment (uncentered variance) of the gradients. At each step, it computes the gradient, updates these averages with exponential decay rates β1 and β2, and then applies bias correction to counteract the initial bias toward zero. The parameter update is then scaled by the corrected first moment divided by the square root of the corrected second moment plus a small constant ε to avoid division by zero.

This combination yields per-parameter adaptive learning rates that are robust to noisy gradients and sparse features. The algorithm's computational cost is modest, requiring only first-order gradients and a constant amount of memory per parameter.

Adam was introduced in a 2015 paper by Diederik P. Kingma and Jimmy Ba, building on earlier work on adaptive methods such as AdaGrad and RMSProp. Its name derives from 'adaptive moment estimation,' reflecting its use of gradient moments. The paper quickly became one of the most cited in machine learning, and the algorithm is now a standard optimizer in frameworks like TensorFlow and PyTorch.

Adam's popularity stems from its strong empirical performance across a wide range of tasks, including image classification, natural language processing, and reinforcement learning. It often converges faster than plain stochastic gradient descent and requires less tuning of hyperparameters.

Several variants have been proposed to address Adam's known issues, such as convergence to suboptimal solutions in some settings. AMSGrad, introduced by Sashank J. Reddi and colleagues, modifies the second moment update to ensure that the learning rate does not increase. AdamW, proposed by Ilya Loshchilov and Frank Hutter, decouples weight decay from the gradient update, improving generalization in many tasks.

Other notable variants include AdaBound, which clips learning rates to a range, and RAdam, which rectifies the variance of the adaptive learning rate during early training. These refinements aim to combine Adam's speed with the stability of classical methods.

Adam's default hyperparameters (β1=0.9, β2=0.999, ε=1e-8) are often used without tuning, but the choice of ε can significantly affect performance, especially in low-precision settings. The algorithm's bias correction is crucial for early steps; without it, the first few updates are too small.

Adam has also been applied beyond neural networks, such as in online learning and non-convex optimization problems. Its convergence properties have been studied extensively, with theoretical guarantees under certain conditions. Notably, the original paper's title was 'Adam: A Method for Stochastic Optimization,' and it was presented at ICLR 2015.

First moment

The mean of the gradients, used to estimate the direction of the update.

Second moment

The uncentered variance of the gradients, used to scale the learning rate per parameter.

Bias correction

A correction applied to the moment estimates to account for their initialization at zero.