Why the Steepest Direction Is the Negative Gradient

Every course on training says the gradient points uphill, so go the other way.

It is true, and it is one of the few claims in the subject with a two-line

proof. The proof is worth doing, not for rigour but because it exposes an

assumption that later optimisers spend their whole existence attacking.

Start with a function $f$ from $\mathbb{R}^n$ to $\mathbb{R}$, a point $x$, and

a direction $u$ with $\|u\| = 1$. Step a small distance $\varepsilon$ along

$u$ and ask what happens to $f$.

Divide by $\varepsilon$ and let it shrink. What is left is the rate of change

of $f$ per unit distance travelled along $u$, and it equals $\nabla f(x) \cdot u$

and nothing else. This number has a name, the directional derivative, but the

name is less useful than the observation: a question about a curved surface in

$n$ dimensions has become a dot product between two vectors.

That reduction is the entire content of first-order optimisation. The gradient

is fixed once you have chosen where you are standing. The only thing left to

choose is $u$, and the only thing $u$ affects is one dot product.

Write the dot product with the angle in it. For any two vectors,

$\nabla f \cdot u = \|\nabla f\| \, \|u\| \cos\theta$, and since $u$ is a unit

vector this is just $\|\nabla f\| \cos\theta$.

So the rate of change along $u$ is the gradient's length times the cosine of the

angle between $u$ and the gradient. The gradient's length is not something you

control; it is whatever the function is doing at $x$. The cosine is entirely

yours.

The cosine is smallest at $\theta = \pi$, where it equals $-1$. That is the

direction exactly opposite the gradient. So the largest possible decrease per

unit step is $\|\nabla f\|$, achieved by

$u = -\nabla f / \|\nabla f\|$, and by no other direction.

The same statement without the angle, which is the version that generalises:

Two lines, and the claim is settled. No direction beats the gradient's length,

and only one direction attains it.

Read the derivation again and look for where it says what $\|u\| = 1$ means.

It does not say. The proof assumed the ordinary Euclidean length,

$\|u\|^2 = \sum_i u_i^2$, because that is what the dot product's angle formula

refers to. That was a choice, and it is a strange one to make without comment,

because the coordinates of $u$ are parameters of a model, and they are not

comparable quantities. One coordinate might be a weight in the first layer and

another a bias in the last. Declaring that a step of 0.01 in one is the same

size as a step of 0.01 in the other is a statement about units, not about the

function.

Suppose instead that length is measured by a positive definite matrix $P$, so

that a unit step means $u^\top P u = 1$. Running the same argument gives a

different answer: the steepest direction becomes proportional to

$-P^{-1} \nabla f$. The gradient is still what carries the information about

the function, but the matrix decides what counts as far.

This single loophole accounts for most of what comes later. Choose $P$ to be

the Hessian and you get Newton's method. Choose $P$ diagonal, built from a

running average of squared gradients, and you get something very close to Adam.

Neither is a different idea from steepest descent. Both are steepest descent

under a different ruler.

One more thing the derivation does not promise. It says the negative gradient

is the best direction for an infinitesimal step. It says nothing whatever about

the location of the minimum.

Take the bowl $f(x, y) = \tfrac{1}{2}\left(x^2 + 100 y^2\right)$, whose minimum

sits at the origin. Its contours are ellipses stretched ten to one.

At that marked point the gradient is $(x, 100y) = (0.71, 7.07)$. The direction

that would take you straight to the minimum is $-(0.71, 0.071)$. The angle

between the negative gradient and the straight line home is about 79 degrees.

The steepest direction is nearly perpendicular to the direction you actually

want.

Nothing has gone wrong. Both statements are true at once: no other direction

lowers $f$ faster over a short step, and the direction is nearly useless for

getting to the bottom. The disagreement grows with the stretch of the bowl, and

the lesson on conditioning turns that observation into a number that predicts

how many steps a run will take.

The rate of change along a direction is a dot product. Cauchy-Schwarz caps that

dot product at the gradient's length and reaches the cap only along the

gradient itself. That is the theorem, and it is airtight.

The two soft spots are both in what it does not say. It does not say how to

measure a step, and the answer changes if you measure differently. And it is a

statement about an infinitesimal step, which is not a statement about where the

minimum is.

The next lesson leaves geometry aside and asks a mechanical question: given a

model with a hundred billion parameters, how is the gradient obtained at all,

and why does getting all hundred billion partial derivatives cost about the

same as evaluating the function once.