AI Glossary
An optimization method that repeatedly adjusts model parameters in small steps in the direction that most reduces the error.
Imagine walking downhill in fog by feeling the slope underfoot: each step goes the steepest way down. The learning rate sets the step size — too large overshoots, too small is slow. Stochastic gradient descent estimates the slope on small batches of data, and optimizers such as Adam add momentum and per-parameter step sizes.
Try the simple gradient descent example.
A PNAS study shows networks of randomly wired units can self-assemble motifs that approximate gradient descent and match backpropagation-level performance.
thelatentdaily.com Editorial Team4 min read
The algorithm that trains almost every AI model, explained with a hiker in the fog and a few lines of Python: gradients, learning rates and why training can fail.
thelatentdaily.com Editorial Team3 min read