Answers · The mathematics underneath

How does momentum stop the steps from cancelling each other out?

In shortMomentum keeps a running total of past gradients, so directions that keep agreeing build up while directions that keep flipping sign cancel out.

Momentum stops the steps cancelling each other out by not taking each gradient as a separate step. It keeps a running total, so a component that points the same way at every step is added to its own past and grows, while a component that flips sign is subtracted by its own past and shrinks. The parameter update uses that total, not the current gradient alone. The velocity carries the agreement; the gradient only nudges it.

v_{k+1} = \beta v_k + g_k, \qquad x_{k+1} = x_k - \alpha v_{k+1}

Unroll the recursion and the running total is a weighted list of every gradient so far: $v_k = g_{k-1} + \beta g_{k-2} + \beta^2 g_{k-3} + \cdots$ . The weight on a gradient from $j$ steps ago is $\beta^j$ . With $\beta=0.9$ a gradient ten steps old still carries $0.9^{10}$ , which is about $0.349$ , and one fifty steps old carries $0.9^{50}$ , about $0.00515$ .

Take a direction where the gradient is $1$ at every step. Starting from zero velocity, the first four values are $v_1=1$ , $v_2=0.9(1)+1=1.9$ , $v_3=0.9(1.9)+1=2.71$ , and $v_4=0.9(2.71)+1=3.439$ . Every term has the same sign, so every term adds. The sum is $1+0.9+0.81+\cdots$ , which is $1/(1-0.9)=10$ . The step length is therefore $10\alpha$ , not $\alpha$ .

Now take the same coefficient and a direction where the gradient is $1,-1,1,-1$ , the pattern produced by overshooting a narrow valley. Starting from zero velocity, $v_1=1$ , $v_2=0.9(1)-1=-0.1$ , $v_3=0.9(-0.1)+1=0.91$ , and $v_4=0.9(0.91)-1=-0.181$ . The positive history is being subtracted by the next negative gradient, so the total never builds. The alternating sum is $1-0.9+0.81-0.729+\cdots$ , which is $1/(1+0.9)=1/1.9$ , about $0.526$ .

FIG 1

step	gradient (steady)	velocity (steady)	gradient (alternating)	velocity (alternating)
1	1	1.0	1	1.0
2	1	1.9	-1	-0.1
3	1	2.71	1	0.91
4	1	3.439	-1	-0.181

1+\beta+\beta^2+\cdots=\frac{1}{1-\beta}, \qquad 1-\beta+\beta^2-\beta^3+\cdots=\frac{1}{1+\beta}

The first sum has all plus signs and the second has alternating signs. That is the whole difference between a direction that agrees with itself and one that does not. At $\beta=0.9$ the first sum is $10$ and the second is $0.526$ , so the same running total treats the two directions differently by a factor of $19$ .

Momentum is therefore not smoothing in the sense of averaging away noise. It is a weighted sum in which age is discounted and sign is preserved. A past gradient is never forgotten outright; it is multiplied by a power of $\beta$ and added.

The natural guess

The natural guess is that momentum simply adds speed, so both directions should get a larger step. That guess fails because the running total is not a multiplier applied equally to every component. It is a sum whose signs matter. A direction that alternates meets its own past with the opposite sign, so the old contribution cancels the new one. A direction that agrees meets its own past with the same sign, so the old contribution reinforces the new one. The cancellation has not been removed; it has been pushed into the history, where it hits the zigzag and leaves the valley direction alone.

What the coefficient decides

The ratio between the two limits is $(1+\beta)/(1-\beta)$ . At $\beta=0.9$ that is $1.9/0.1=19$ : the steady direction is nineteen times larger than the alternating one. At $\beta=0.99$ the steady multiplier is $1/(1-0.99)=100$ and the alternating one is $1/(1+0.99)=1/1.99$ , about $0.503$ , so the ratio is about $199$ . Raising the coefficient does not make both directions faster by the same amount. It widens the gap, which is why a high coefficient straightens a zigzag but also makes the stable step length much larger.

Where it stops

This picture assumes the gradient repeats, either steadily or with a clean sign flip. When the gradient changes irregularly, the geometric sum is still a filter, but it is filtering a moving signal rather than a fixed one. A high coefficient then keeps stale directions alive for many steps, and a noisy or zero-mean gradient can be accumulated as easily as a real slope. The other limit is the step length. The learning rate really in force is $\alpha/(1-\beta)$ . With $\alpha=0.01$ and $\beta=0.9$ that is $0.01/0.1=0.1$ ; with $\beta=0.99$ it is $0.01/0.01=1.0$ . The learning rate has not changed, but the step has grown by a factor of ten, and a run that was stable can diverge.

Worth keeping

With beta = 0.9 a steady gradient builds a velocity ten times the gradient, so the real learning rate is ten times the number set.
With the same beta an alternating gradient builds to only about 0.526 of a single gradient, a ratio of 19 to the steady case.
Raising beta from 0.9 to 0.99 raises the steady multiplier from 10 to 100, so a stable learning rate can become unstable without any change to the learning rate.

Where this goes next

This question is one step of The Mathematics of Gradient Descent, a course that works through its subject from the beginning. The page that teaches this part of it is Momentum, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course