Why does gradient descent still work when each step uses only one minibatch's gradient?

Because the minibatch gradient is an unbiased estimate of the gradient of the full loss. Each individual step points somewhere slightly wrong, but the wrongness has no preferred direction, so across many steps those errors cancel while the true downhill pull keeps adding up.

The full loss is an average over the training set, $F(\theta) = \frac{1}{N}\sum_{i=1}^{N} f_i(\theta)$, so its gradient is the average of the per-example gradients. A minibatch gradient replaces the population average with an average over a random batch $B$ of size much smaller than $N$.

The expectation is over which examples happen to land in the batch. It does not say the batch gradient is close to the full one: on thirty-two examples it can differ noticeably in both length and direction. It says the difference $\varepsilon_B = g_B - \nabla F(\theta)$ has mean zero. That is a claim about the centre of the scatter, not about its width.

Unroll the run. After $k$ steps the parameters are the starting point minus the sum of the step directions taken, and each direction splits into the true gradient plus its error.

The first sum is ordinary descent and moves the parameters a distance of roughly $k\alpha|\nabla F|$ after $k$ steps. The second sum is a sum of independent zero-mean terms, so it does not grow like $k$; it grows like a random walk, with standard deviation $\alpha\sigma\sqrt{k}/\sqrt{|B|}$ where $\sigma$ is the per-example spread. Drift grows linearly in $k$, noise grows as the square root, so the ratio of noise to progress falls as $1/\sqrt{k}$. Nothing accumulates in a wrong direction: it spreads. Put numbers in: take $\sigma = 1$, $|\nabla F| = 0.5$ and a batch of 16, so each step has error standard deviation $1/\sqrt{16} = 0.25$. After 100 steps the drift is about $100 \times 0.5\alpha = 50\alpha$ and the accumulated error has standard deviation $0.25 \times \sqrt{100}\,\alpha = 2.5\alpha$. The signal outweighs the accumulated noise twenty to one.

The natural reading is that a noisy gradient must give a wrong answer, that unless every step uses the exact gradient the run cannot settle at the true minimum. The first half is true and the second does not follow. Noise in the step is not noise in the function being minimised: the average step still points downhill, and where the run comes to rest is decided by where that average step vanishes, not by how widely the individual steps scatter around it. A bias behaves completely differently. If the errors had a mean of just $0.01|\nabla F|$ in some fixed direction, the steps would stop where $\nabla F$ equals minus that bias, and the run would settle permanently beside the minimum. More steps would not help, because more steps make a bias more decisive, not less. This is why drawing the batch uniformly matters: a batch that over-represents easy examples gives a biased gradient, and a biased gradient minimises a different function from the one that was written down.

The cancelling argument works on the way down, while the true gradient is large. Near a minimum the true gradient shrinks toward zero and the noise does not, because the disagreement between examples is still there when their average is zero. With a fixed step size the run then settles into a region around the minimum rather than at it, and the size of that region is proportional to the step size.

Halving the step size halves that floor, which is why a curve that has flattened for a long time often drops immediately when the learning rate is cut, and why the drop is abrupt rather than gradual. It also stops being true at very large batches: doubling the batch halves the variance of each step but also halves the number of steps per pass over the data, and the two effects roughly cancel if the step size is doubled at the same time, keeping $\alpha/|B|$ fixed. That rule holds over a wide range and then fails, because the step size it demands runs into the ceiling set by curvature, which knows nothing about batch size. Warmup schedules exist to manage that collision.