On the Learning Rate — Prime Atlas

i. Intuition

Imagine standing on a foggy hillside, trying to reach the valley floor. You cannot see far, but you can feel the slope beneath your feet. Gradient descent does exactly this: feel the slope, take a step downhill, repeat. The learning rate is the length of your stride — too short and you will be walking all day; too long and you leap clean over the valley and land on the other side.

The learning rate η scales every update, w ← w − η∇L, and it is usually the first hyperparameter worth tuning. Too low and training stalls; too high and the loss oscillates or diverges. Modern training rarely holds it fixed: a short warm-up, then a cosine or step decay, paired with an optimizer such as Adam that adapts the step for each parameter.

For a quadratic loss with Hessian H, gradient descent converges exactly when 0 < η < 2/λmax(H), at a rate set by the condition number κ = λmax/λmin. Neural networks are not quadratic, and full-batch training behaves strikingly: sharpness rises until λmax ≈ 2/η, then hovers there while the loss keeps falling, non-monotonically — the so-called edge of stability.

ii. Play

Figure 2 Ten steps, one dimension

Learning rate η

Converging steadily.distance × 0.70 each step · loss after 10: 0.010

Fig. 2. — Ten steps of gradient descent on L(w) = (w − 3)². Find the value of η that lands in a single step.

iii. Derive

Why does η = 0.5 land in one step, and why does η = 1 bounce forever? Four lines are enough.

wₜ₊₁ = wₜ − η L′(wₜ) (1) the update
L(w) = (w − w*)² ⇒ L′(w) = 2(w − w*) (2) our loss
wₜ₊₁ − w* = (1 − 2η)(wₜ − w*) (3) substitute
converges ⇔ |1 − 2η| < 1 ⇔ 0 < η < 1 (4) the condition

Every step multiplies the distance to the minimum by the same factor, 1 − 2η. At η = 0.5 that factor is zero: one step and you are there. At η = 1 it is −1, and the weight flips sides for ever.

iv. Code

PyTorch train.py

model = torch.nn.Linear(1, 1)
opt = torch.optim.SGD(model.parameters(), lr=0.5)

for x, y in data:
  opt.zero_grad()
  loss = ((model(x) - y) ** 2).mean()
  loss.backward()      # compute ∇L
  opt.step()           # w ← w − η∇L

The lr argument is η. In scikit-learn’s SGDRegressor the same idea is called eta0.

v. Check

On L(w) = (w − 3)², starting from w = 5 with η = 0.25, where is w after one step?

vi. Source

Where the idea comes from Augustin-Louis Cauchy, 1847 Méthode générale pour la résolution des systèmes d’équations simultanées

The idea: To make a function smaller, step against its gradient — the first written account of gradient descent.
What came next: Robbins & Monro (1951) showed it still works with noisy gradients — the root of stochastic gradient descent.