Answers · How models work

Why does a bigger batch make the gradient less noisy?

In shortEach step's gradient is an average, and an average's error shrinks with the square root of the number of examples, so four times the batch only halves the noise.

A bigger batch makes the gradient less noisy because the gradient a step uses is an average, one gradient per example in the batch, and the typical error in an average falls as the square root of the number of terms in it. Quadrupling the batch divides that error by two, not by four. The averaging is the entire mechanism: nothing about the model or the optimiser changes, only how many per-example gradients get averaged before the weights move.

\varepsilon = \frac{\sigma}{\sqrt{B}}

Here $\sigma$ is how much the per-example gradients disagree with each other. It belongs to the data and the model, not to the batch, and no batch size changes it. $B$ is the batch size, and it appears under a square root. Take $\sigma = 1$ for arithmetic's sake and measure every error relative to a batch of 16, whose error is $\sqrt{16}/\sqrt{16} = 1$ : the error at any other batch is $4/\sqrt{B}$ , because 4 is $\sqrt{16}$ .

FIG 1

batch	sqrt(B)	error, relative to B=16	arithmetic per step
16	4	1.00	1x
64	8	0.50	4x
256	16	0.25	16x
1024	32	0.125	64x
4096	64	0.0625	256x

Read the last two columns as a price list. Going from 256 to 1024 examples per step costs four times the arithmetic and buys a gradient that sits half as far from the true gradient of the whole dataset. Going from 256 to 4096 costs sixteen times the arithmetic and buys a quarter of the noise. The returns do diminish, but on a known schedule rather than an arbitrary one, and the schedule is the only reason batch size is a hyperparameter worth thinking about rather than a dial you turn to fill the GPU.

The step size has to move with it

The natural guess is that batch size is purely a throughput knob: make it bigger, keep everything else, finish sooner. Do that and the run gets measurably worse, and the reason is a counting argument rather than a statistical one. The dataset is fixed, so four times the batch means a quarter of the steps per epoch. Each step still moves the weights a distance set by the learning rate. The run therefore makes a quarter as many moves of the same length, and covers a quarter of the distance through weight space on the same data. The gradients were better, but there were far fewer of them, and the noise reduction never got spent.

\eta_{\text{new}} = \eta_{\text{old}} \cdot \frac{B_{\text{new}}}{B_{\text{old}}}

Four times the batch, four times the learning rate, so that distance travelled per example seen is unchanged. The justification is loose, since it says that a step four times more reliable can be taken four times further, but it holds well across one or two orders of magnitude. What it does not survive is its own success: quadrupling the rate gives a value the run cannot survive at step one and can survive at step two thousand, which is the whole reason large-batch runs pair the scaling rule with a warmup that ramps the rate up over the first few thousand steps.

Where the noise stops being the limit

The noise is not the only thing holding the step size down. The curvature of the loss is the other: a step past a certain length overshoots the valley it is descending and diverges no matter how accurately its direction was measured. While noise is well above that threshold it binds first, so cutting it lets the step grow and the run shortens. Once noise is well below the threshold, halving it again changes nothing, because the step was never being held back by noise. The step count flattens. Going from a batch of 256 to 16384, sixty-four times the data per step, takes 40000 steps down to 3000, a factor of about thirteen rather than sixty-four, while the examples consumed rise from $256 \times 40000 = 10{,}240{,}000$ to $16384 \times 3000 = 49{,}152{,}000$ , roughly four and a half times the bill for the thirteen-fold saving in steps. The batch size where that knee sits depends on the task and on how far through training you are, not on the hardware.

Two things are routinely blamed on batch size and are not fixed by it. Averaging reduces variance and does nothing to bias: if the inputs and targets are misaligned by one position, or the loss is averaged over the wrong axis, every per-example gradient is wrong in the same direction and the mean of a million of them is exactly as wrong as any one of them. A fault that a larger batch leaves untouched is a bias, which makes batch size a decent diagnostic for telling the two apart. The other is memory. Activations for a batch of 4096 will not fit where a batch of 256 did, and the usual answer is gradient accumulation: run sixteen batches of 256, add the gradients, then take one step. That matches the statistics of the large batch exactly and matches none of its speed, because the arithmetic is identical and only the number of updates has fallen.

Worth keeping

The error in a batch gradient is the per-example disagreement divided by the square root of the batch size, so four times the examples buys half the noise.
A larger batch with an unchanged learning rate travels less far per epoch, because the number of steps falls in proportion to the batch.
Averaging over more examples reduces variance and never reduces bias, so a fault that survives a bigger batch is a fault in the data or the loss.

Where this goes next

This question is one step of Training a Model From Scratch, a course that works through its subject from the beginning. The page that teaches this part of it is How Much to Look at First, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course

Why does a bigger batch make the gradient less noisy?

The step size has to move with it

Where the noise stops being the limit

Worth keeping

Where this goes next

Read alongside