Backpropagation

The algorithm that computes how much each weight in a neural network contributed to the error, so the weights can be adjusted to reduce it.

After a forward pass produces a prediction and a loss, backpropagation applies the chain rule of calculus backwards through the layers to obtain the gradient of the loss with respect to every weight. An optimizer such as gradient descent or Adam then nudges each weight in the direction that lowers the loss.

The method was popularized for multi-layer networks in a 1986 Nature paper by Rumelhart, Hinton and Williams and remains how virtually all neural networks are trained.

#Glossary #Training

Stories about Backpropagation

Paper of the DayResearch

Paper of the Day: Can the brain build its own backpropagation-like learning from simple local rules?

A PNAS study shows networks of randomly wired units can self-assemble motifs that approximate gradient descent and match backpropagation-level performance.

thelatentdaily.com Editorial TeamOct 194 min read