Basis Lab

Answers · The mathematics underneath

Why is the negative gradient the direction of steepest descent?

In shortA unit step goes downhill fastest when it points directly opposite the gradient, because the dot product with the gradient is then as negative as it can be.

The negative gradient is the steepest descent direction because, among all unit steps, it gives the largest first-order decrease in the function. That fact is a dot-product inequality, not an extra geometric intuition: it follows from writing the change per unit step as the gradient dotted with the step direction and then choosing the direction that makes that dot product as negative as possible.

f(x+εu)=f(x)+ε∇f(x)⋅u+O(ε2),∥u∥=1.f(x+\varepsilon u)=f(x)+\varepsilon \nabla f(x)\cdot u+O(\varepsilon^2), \qquad \|u\|=1.

Subtracting f(x)f(x), dividing by ε\varepsilon, and letting ε\varepsilon shrink leaves ∇f(x)⋅u\nabla f(x)\cdot u. That number is the rate of change of ff per unit distance travelled along uu. The gradient is fixed once xx is fixed; the only thing you choose is uu, and the only thing uu affects is that one dot product. If you double the step, the change roughly doubles; if you halve it, the change roughly halves. That linearity is what makes direction and distance separable: direction chooses the dot product, distance chooses the multiplier.

Put the angle into the dot product. For any two vectors, ∇f⋅u=∥∇f∥ ∥u∥cos⁡θ\nabla f\cdot u=\|\nabla f\|\,\|u\|\cos\theta, and for a unit vector uu this is ∥∇f∥cos⁡θ\|\nabla f\|\cos\theta. The gradient's length is whatever the function is doing at xx; the cosine is the part you control. The cosine is smallest at θ=π\theta=\pi, where it equals −1-1, so the largest decrease per unit step is ∥∇f∥\|\nabla f\|, reached by u=−∇f/∥∇f∥u=-\nabla f/\|\nabla f\|. No other direction reaches it, because cos⁡θ=−1\cos\theta=-1 at that angle alone.

The angle version also explains a familiar fact: directions at right angles to the gradient have cos⁡θ=0\cos\theta=0, so they change ff not at all to first order. Those are the tangent directions to the contour through xx. The negative gradient is not just downhill; it is the unique direction that gets the full first-order decrease, not merely a share of it.

FIG 1The bound and its equality case
how much a unit step in the direction u can change the function
the ceiling on that change. The second factor is one for a unit direction, so the ceiling is set by the gradient alone, and it is unreachable by any direction that is not parallel to it
The equality condition is the useful half: only a direction parallel to the gradient reaches the bound.

What steepest secretly assumes

The natural guess is that the negative gradient is an intrinsic fact about the function, independent of how steps are measured. The proof does not support that: it used the ordinary Euclidean length ∥u∥2=∑iui2\|u\|^2=\sum_i u_i^2 when it set ∥u∥=1\|u\|=1. That is a choice. The coordinates of a step are model parameters, and a change of 0.010.01 in a first-layer weight is not automatically the same size as a change of 0.010.01 in a final bias. If length is measured by a positive definite matrix PP, so that a unit step means u⊤Pu=1u^\top P u=1, the same argument gives a different winner. The mathematical statement is that steepest is relative to a norm, and the gradient is the steepest direction only for the Euclidean norm.

u⊤Pu=1⟹usteepest∝−P−1∇f(x).u^\top P u = 1 \quad\Longrightarrow\quad u_{\text{steepest}} \propto -P^{-1}\nabla f(x).

The gradient still carries the function's first-order information, but PP decides what counts as far. Take PP to be the Hessian and the rule becomes Newton's method. Take PP diagonal, built from a running average of squared gradients, and the rule is very close to Adam. These are not escapes from steepest descent. They are steepest descent under a different ruler, which is why the same proof keeps reappearing with a new matrix in place of the identity.

Downhill is not the way home

Even with the Euclidean ruler, the theorem is about an infinitesimal step, not about where the minimum lives. Take f(x,y)=12(x2+100y2)f(x,y)=\tfrac12(x^2+100y^2), whose minimum is at the origin and whose contours are ellipses stretched ten to one. At the point (0.71,0.071)(0.71,0.071), the gradient is (0.71,7.07)(0.71,7.07). The straight-line direction to the minimum is −(0.71,0.071)-(0.71,0.071).

∇f(0.71,0.071)=(0.71,7.07),−(0.71,0.071) points to the minimum.\nabla f(0.71,0.071)=(0.71,7.07),\qquad -(0.71,0.071)\text{ points to the minimum}.

The negative gradient there is (−0.71,−7.07)(-0.71,-7.07), of length 0.5+50=7.11\sqrt{0.5+50}=7.11. The homeward direction is (−0.71,−0.071)(-0.71,-0.071), of length 0.5+0.005=0.711\sqrt{0.5+0.005}=0.711. Their dot product is 0.5+0.5=1.000.5+0.5=1.00, so the cosine of the angle between them is 1/(7.11×0.711)=1/5.05=0.1981/(7.11\times0.711)=1/5.05=0.198, and the angle is about 7979 degrees. The steepest direction is nearly perpendicular to the direction that would actually reach the minimum.

If the step is finite, the discarded O(ε2)O(\varepsilon^2) term can overtake the first-order term, so a step along the negative gradient need not decrease ff at all. The theorem is airtight for a Euclidean unit step in the limit of an infinitesimal step; relax either condition and the negative gradient is no longer the whole answer.

Worth keeping

  • At a point, the rate of change per unit distance in a chosen direction is the gradient dotted with that unit direction.
  • The negative gradient is the unique best first-order direction because the dot product is most negative when the step is antiparallel to the gradient.
  • Steepest descent depends on the ruler used to measure step length; under a metric given by a positive definite matrix, the steepest direction is proportional to minus the inverse metric times the gradient.

Where this goes next

This question is one step of The Mathematics of Gradient Descent, a course that works through its subject from the beginning. The page that teaches this part of it is The Steepest Direction, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. How does momentum stop the steps from cancelling each other out?Momentum keeps a running total of past gradients, so directions that keep agreeing build up while directions that keep flipping sign cancel out.
  2. Why does computing the gradient cost about the same no matter how many parameters the network has?Because one backward sweep shares intermediate derivatives across all parameters, so the whole gradient costs a few forward passes in total rather than one per parameter.
  3. Why does Adam need bias correction at the start of training?Adam needs bias correction because its two running averages start at zero, and without correction the early steps are several times larger than the learning rate asks for.
  4. Why does gradient descent still work when each step uses only one minibatch's gradient?A minibatch gradient is wrong at every step but correct on average, so the errors cancel over many steps while the true downhill pull keeps adding up.
  5. What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
  6. How do you make a vague claim precise enough to tell whether it's true?Fix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.

Read alongside