If you train a neural network today, there is a very good chance you use Adam or one of its descendants. Introduced by Diederik Kingma and Jimmy Ba in a paper presented at ICLR 2015, Adam became one of the most cited papers in all of computer science. It is the workhorse behind the training of language models, image generators and much more.

The problem: training is optimization

Training a neural network means finding parameter values that make the model’s error — the loss — as small as possible. The standard tool is gradient descent: compute how the loss changes with respect to each parameter (the gradient) and nudge every parameter a little in the direction that reduces the loss.

In practice, models use stochastic gradient descent (SGD): gradients are estimated on small random batches of data rather than the whole dataset. This is efficient but noisy, and plain SGD has two well-known weaknesses.

  1. One learning rate for everything. Some parameters need big steps, others tiny ones. A single global step size is a compromise that suits few of them.
  2. Noisy, zig-zagging updates. In ravine-shaped regions of the loss landscape, gradients point back and forth across the ravine, slowing progress along it.

Adam’s two ideas

Adam — short for adaptive moment estimation — combines two earlier techniques.

1. Momentum (the first moment)

Instead of using only the current gradient, Adam keeps a running average of recent gradients. Like a ball rolling downhill, this average smooths out noise and keeps moving in consistent directions, while back-and-forth oscillations cancel out.

2. Adaptive step sizes (the second moment)

Adam also keeps a running average of the squared gradients for each parameter. Parameters whose gradients are consistently large get smaller effective steps; parameters with small or rare gradients get relatively larger steps. This idea comes from earlier methods such as AdaGrad and RMSProp.

Putting it together

At each step, Adam updates each parameter in proportion to its momentum divided by the square root of its squared-gradient average. Because the running averages start at zero, the paper adds a bias correction so early steps are not artificially small.

Adam has a handful of settings: a learning rate and two decay rates for the running averages, with defaults (0.9 and 0.999 for the decay rates) that work reasonably well across many problems. That robustness is a large part of its success.

Why it won

  • It works out of the box. Adam often trains well with little tuning, which matters when every experiment is expensive.
  • It handles sparse and uneven gradients, common in language models where many words appear rarely.
  • It is cheap: it only stores two extra values per parameter.
  • It suits large, complex models with very different parameter groups.

Known pitfalls and fixes

Generalization. In some computer vision tasks, carefully tuned SGD with momentum was found to generalize slightly better than Adam. Practitioners debated the trade-off for years.

Weight decay done wrong. Weight decay is a technique that discourages large parameter values to reduce overfitting. In 2017, Ilya Loshchilov and Frank Hutter showed that the common way of combining it with Adam was flawed, and proposed AdamW, which decouples weight decay from the gradient update. AdamW became the standard optimizer for training transformers, including large language models.

Memory at scale. Storing two extra values per parameter is cheap for small models but significant for models with hundreds of billions of parameters. This has motivated memory-efficient variants and optimizer-state sharding across GPUs.

Instability. Very large training runs sometimes suffer loss spikes, which teams manage with learning-rate warm-up, gradient clipping and careful tuning.

The bigger picture

Adam is a good example of how a relatively simple, well-engineered idea can shape an entire field. It did not make models smarter by itself; it made training reliable enough that researchers could focus on architectures, data and scale. Newer optimizers continue to appear, but Adam and AdamW remain the baseline every one of them must beat.

Frequently asked questions

What is the difference between Adam and AdamW?

Both use momentum and adaptive per-parameter step sizes. AdamW applies weight decay directly to the parameters, separately from the gradient-based update, instead of mixing it into the gradient. This “decoupled” version regularizes more predictably and is the standard choice for training transformers.

Do I need to tune Adam’s settings?

The default decay rates usually work, but the learning rate almost always needs tuning, together with a schedule such as warm-up followed by gradual decay. For large models, warm-up and gradient clipping are important for stability.

Why does Adam use more memory than SGD?

It stores two extra values for every parameter: the running average of gradients and the running average of squared gradients. For a model with billions of parameters, those optimizer states can take more memory than the model weights themselves.

Is Adam always the best optimizer?

No single optimizer wins everywhere. Well-tuned SGD with momentum remains competitive for some vision tasks, and researchers continue to propose new optimizers. Adam’s strength is that it works reliably across many problems with little tuning.