Gradient descent: how a network walks downhill

Millions of dials, one score

A network's behaviour is set by its weights: plain numbers, from a few dozen in a toy model to hundreds of billions in a large language model. A loss function turns the network's mistakes on the training data into a single number. Training means finding weights that make that number small.

Searching by trial and error is hopeless. Try just ten values for each of a million weights and there are 10^1,000,000 combinations. Gradient descent never looks at the whole landscape. It only asks which way is downhill from where it stands right now.

Picture standing on a hillside in thick fog. You cannot see the valley, but you can feel which way the ground slopes under your feet. Take a step that way, feel again, step again. The height is the loss; your position is the weights.

Slope, derivative, gradient

With a single weight $w$ , the slope is the derivative: how much the loss changes for a tiny nudge in $w$ . You can measure it by nudging both ways:

\frac{d L}{d w} \approx \frac{L (w + h) - L (w - h)}{2 h}

This is how the tests for this guide check every derivative the pages use.

If the derivative is positive, increasing $w$ increases the loss, so you should decrease $w$ . If it is negative, increase it. Either way, you move against the sign of the slope.

With many weights, collect one slope per weight into a vector. That vector is the gradient, written $\nabla L$ :

\nabla L = (\frac{\partial L}{\partial w_{1}}, \frac{\partial L}{\partial w_{2}}, \dots, \frac{\partial L}{\partial w_{n}})

The gradient points in the direction in which the loss rises fastest, so its negative points straight downhill. Working out all those slopes cheaply, even for billions of weights, is the job of backpropagation, the next chapter.

The update rule

Gradient descent repeats one line:

w_{t + 1} = w_{t} - η \nabla L (w_{t})

$η$ (eta) is the learning rate: how big a step to take. It is the most important number in training, and the easiest to get wrong. Too small and training crawls. Too large and each step overshoots the valley floor, lands higher on the other side, and the next step overshoots further.

How big can a step be?

For a loss shaped like a simple bowl you can work out the exact limit. Take $L (w) = \frac{1}{2} λ w^{2}$ , where $λ$ measures how sharply the bowl curves. The gradient is $λ w$ , so one step gives:

w_{t + 1} = w_{t} - η λ w_{t} = (1 - η λ) w_{t}

Every step multiplies the weight by the same factor, $1 - η λ$ . The weight shrinks towards the minimum at 0 only while that factor lies between −1 and 1:

| 1 - η λ | < 1 ⟺ 0 < η < \frac{2}{λ}

Between $\frac{1}{λ}$ and $\frac{2}{λ}$ the factor is negative, so the weight flips sign on every step: it zig-zags but still converges. Past $\frac{2}{λ}$ each zig-zag is bigger than the last, and the weights blow up.

Real losses curve differently in different directions, and the sharpest direction sets the limit. The first landscape below curves ten times more steeply in $y$ ( $λ = 10$ ) than in $x$ ( $λ = 1$ ), so plain gradient descent only survives for $η < 0.2$ . Press Stability to run η = 0.19 and η = 0.21 side by side.

Even at the safe 0.19, the gentle $x$ direction shrinks by only 0.81 per step. Steep directions force small steps, and then flat directions take forever. Almost every improvement in the rest of this chapter is a way round that.

Race six optimisers across a loss landscape

Race all six runs every optimiser from the same start, each with a learning rate chosen for that landscape. Run one uses your own settings. Click or tap the map to move the start and run again.

Landscape Optimiser Learning rate η = 0.15 Momentum μ = 0.9 Steps = 300

Start x Start y

start (−2.50, 2.50)

Darker is lower loss. White ring: the minimum. Black-ringed dot: the start.

Loss above the true minimum against step, on a log scale. A line that drops off the bottom has reached the minimum.

Optimiser	η	Loss above minimum	Ends at	Outcome

What the race shows

No single learning rate suits every optimiser, so each gets its own, picked by a grid search on each landscape for a 300-step run. Momentum's rates come out about ten times smaller than SGD's: with μ = 0.9 its velocity builds up to about η / (1 − μ), ten times a single step.

Elongated bowl. All six reach the minimum. SGD's rate, 0.15, sits between 1/λ and 2/λ for the steep direction, so its path zig-zags across the bowl on the way down.

Rosenbrock valley. The minimum at (1, 1) lies at the bottom of a long, curved valley. SGD and momentum end closest to it. Adam and AdamW end on the valley floor but still short of the minimum. RMSProp ends further away, and AdaGrad makes little progress: it divides each step by the running total of every squared gradient it has seen, so a few steep gradients early on shrink all its later steps. Near (1, 1) the valley curves about 2,500 times more sharply across than along, which puts SGD's 0.002 almost exactly on its stability limit. At 0.005 it blows up.

Tilted double well. The start is on the slope of a shallow well; a deeper one lies on the far side of a hump. Only momentum carries enough speed over the hump to reach the deeper well. Every other optimiser ends in the shallow one, where the gradient is zero, so nothing pushes it further.

These are two-dimensional toys, chosen so you can see a path. The learning rates that work here say nothing about the right ones for a real network: see what these pictures leave out.

Where it came from

1847Augustin-Louis Cauchy describes the method of steepest descent, to solve the systems of equations that arise in astronomy.
1944Haskell Curry analyses steepest descent for nonlinear problems.
1951Herbert Robbins and Sutton Monro prove that noisy steps still converge if the step size shrinks at the right rate: the theory under stochastic gradient descent.
1960Bernard Widrow and Ted Hoff train ADALINE with the least-mean-squares rule, stochastic gradient descent built in hardware.
1964Boris Polyak adds momentum: the “heavy ball” method.
1983Yurii Nesterov's accelerated gradient reaches the best possible rate for smooth convex problems.
1986Rumelhart, Hinton and Williams train hidden layers with backpropagation and gradient descent with momentum.
2011Duchi, Hazan and Singer's AdaGrad gives each weight its own step size.
2012Geoffrey Hinton presents RMSProp in a Coursera lecture. It was never written up as a paper.
2014Diederik Kingma and Jimmy Ba publish Adam.
2017Ilya Loshchilov and Frank Hutter separate weight decay from Adam's update: AdamW.

Stochastic gradient descent

The true gradient averages over every example in the training set. With millions of examples, computing it for every step is far too slow. Stochastic gradient descent estimates it from a small random batch and steps on that estimate. Each step is noisy, but right on average.

Robbins and Monro showed in 1951 that this still converges, provided the step sizes shrink neither too fast nor too slowly:

\sum_{t = 1}^{\infty} η_{t} = \infty \sum_{t = 1}^{\infty} η_{t}^{2} < \infty

The first condition lets the weights travel as far as they need to; the second makes the noise average out. Modern schedules rarely meet these conditions exactly, but they keep the lesson: lower the learning rate as training goes on. The noise may also help. Keskar and colleagues found in 2017 that very large batches, which are less noisy, tend to land in sharper minima that generalise worse, though that explanation is still debated.

The race above uses exact gradients, so its “SGD” is plain full-batch gradient descent.

Momentum

Plain gradient descent has no memory. In a narrow valley it bounces from wall to wall instead of running along the floor. Polyak's 1964 fix gives the weights a velocity, like a heavy ball rolling downhill:

v_{t + 1} = μ v_{t} - η \nabla L (w_{t})

w_{t + 1} = w_{t} + v_{t + 1}

With μ = 0.9, pushes that keep pointing the same way add up to about ten times a single step, while pushes that alternate across the valley cancel out. The built-up speed also carries the ball over small bumps, which is how momentum reaches the deeper well in the race. Nesterov's 1983 variant measures the gradient at the point the velocity is about to carry you to, and reaches the best possible convergence rate for smooth convex problems.

A step size for every weight

Some weights see large gradients on every step; others see small or rare ones. AdaGrad (2011) gives each weight its own step size by dividing by the square root of the sum of all its past squared gradients $g_{t}$ :

s_{t} = s_{t - 1} + g_{t}^{2}

w_{t + 1} = w_{t} - \frac{η}{\sqrt{s_{t}} + ε} g_{t}

That helps on sparse problems such as text, where rare words deserve big updates. But the sum only grows, so every step shrinks forever, which is why AdaGrad stalls in the Rosenbrock valley. RMSProp (Hinton, 2012) replaces the sum with a decaying average that forgets old gradients:

s_{t} = ρ s_{t - 1} + (1 - ρ) g_{t}^{2}

Adam (Kingma and Ba, 2014) keeps two decaying averages: one of the gradient, as momentum does, and one of its square, as RMSProp does. It corrects both for starting at zero. (Here $v$ is the average of squared gradients, not momentum's velocity.)

m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}

v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}

w_{t + 1} = w_{t} - η \frac{m_{t} / (1 - β_{1}^{t})}{\sqrt{v_{t} / (1 - β_{2}^{t})} + ε}

Its defaults (β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸) work across a very wide range of problems, which is a large part of why Adam became the default choice.

AdamW: weight decay done properly

Weight decay shrinks every weight a little on each step, which discourages a network from leaning on a few huge weights. With plain SGD, adding an L2 penalty $\frac{λ}{2} {‖ w ‖}^{2}$ to the loss and shrinking the weights directly are the same thing. Loshchilov and Hutter pointed out that inside Adam they are not: the penalty's gradient is divided by Adam's per-weight scaling too, so weights with large gradients barely decay. AdamW applies the decay outside the adaptive step:

w_{t + 1} = w_{t} - η \cdot (Adam step) - η λ w_{t}

Here λ is the decay rate, not a curvature.

In the race, Adam and AdamW trace almost the same paths. The difference shows when the gradient is zero: Adam stops, while AdamW keeps shrinking the weights by a factor of $1 - η λ$ per step. This guide's tests check exactly that.

What these pictures leave out

Real networks have millions of dimensions, not two. Dauphin and colleagues argued in 2014 that most flat points there are saddle points, which slow training down, rather than bad local minima that trap it.
With mini-batches, the landscape itself changes slightly on every step.
The learning rates in the race were tuned for three toy surfaces and 300 steps. A real network needs its own, usually found with short trial runs.

What people use now

Transformers are usually trained with AdamW, a short warm-up during which the learning rate rises from near zero (the original Transformer used 4,000 warm-up steps), and then a slow decay, often along a cosine curve. SGD with momentum is still competitive for many image models. New optimisers keep arriving. Lion (2023), found by an automated search, uses only the sign of a momentum-like update. Muon (2024) orthogonalises the update for each weight matrix and has been used to train large language models.

Try it in the workbench

The workbench's MLP trainer trains a small network on 2-D data with the optimiser you choose, and redraws its decision boundary as it learns.

Sources

A.-L. Cauchy (1847). Méthode générale pour la résolution des systèmes d'équations simultanées. Comptes rendus de l'Académie des sciences 25, 536–538.
C. Lemaréchal (2012). Cauchy and the gradient method. Documenta Mathematica, Extra Volume ISMP, 251–254.
H. Robbins and S. Monro (1951). A stochastic approximation method. Annals of Mathematical Statistics 22(3).
B. T. Polyak (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4(5).
Y. Nesterov (1983). A method for solving the convex programming problem with convergence rate O(1/k²). Soviet Mathematics Doklady 27.
D. E. Rumelhart, G. E. Hinton and R. J. Williams (1986). Learning representations by back-propagating errors. Nature 323.
J. Duchi, E. Hazan and Y. Singer (2011). Adaptive subgradient methods for online learning and stochastic optimization. JMLR 12.
G. Hinton, N. Srivastava and K. Swersky (2012). Neural networks for machine learning, lecture 6 (RMSProp).
D. P. Kingma and J. Ba (2014). Adam: a method for stochastic optimization.
I. Loshchilov and F. Hutter (2017). Decoupled weight decay regularization.
Y. Dauphin et al. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.
N. S. Keskar et al. (2017). On large-batch training for deep learning: generalization gap and sharp minima.
A. Vaswani et al. (2017). Attention is all you need.
X. Chen et al. (2023). Symbolic discovery of optimization algorithms (Lion).
H. H. Rosenbrock (1960). An automatic method for finding the greatest or least value of a function. The Computer Journal 3(3).