Answers · The systems around the model

Should the gradients be summed or averaged across GPUs?

In shortSum the gradients when each device already divides its loss by the global batch size; average them when each device uses its own local mean. Doing both makes the gradient too small by the number of devices.

Sum them if each device's loss has already been divided by the global batch size. Average them if each device's loss is a mean over its own local batch. The reduction operation is what decides, not the loss: a sum reduction and a mean reduction differ by a factor of the number of devices, so asking for the wrong one is exactly how you double-count.

B = n b, \quad S_g = \sum_{j=1}^{b} \ell_{g,j}, \quad L = \frac{1}{B}\sum_{g=1}^{n} S_g = \frac{1}{n b}\sum_{g=1}^{n} S_g

Let eight devices each hold 32 examples, so the global batch is $8 \times 32 = 256$ . Device $g$ computes $S_g$ , the sum of its 32 losses. If every device divides its loss by 256, its gradient is $(1/256)\nabla S_g$ . Summing those eight gradients gives $(1/256)\nabla(S_1 + \cdots + S_8)$ , which is the gradient of the global mean loss. That is the correct sum. If the eight gradients are then averaged, the result is $\nabla L / 8$ , because the averaging divides by eight a second time. The error is not subtle: the gradient is eight times too small, and every parameter update inherits that factor.

FIG 1

Loss on each device	Reduction to use	Gradient after reduction
divided by global batch	sum
mean over local batch	mean, or sum then divide by
divided by global batch	mean
mean over local batch	sum

The natural guess is that all-reduce always averages, so a global-batch division must be double-counting. It is not the all-reduce that carries a hidden normalisation. A ring exchange that sums chunks sums them; if you request an average, the library performs the sum and then divides by $n$ . Check the op. With a sum op, your global-batch division is the only normalisation and it is correct. With a mean op, the global-batch division is one normalisation too many. The factor is $n$ , not a rounding difference: at eight devices the gradient is eight times too small, at $n$ devices it is $n$ times too small.

\text{sum of global-batch-scaled gradients} = \sum_g \frac{1}{B}\nabla S_g = \nabla L, \qquad \text{mean of the same} = \frac{1}{n}\sum_g \frac{1}{B}\nabla S_g = \frac{1}{n}\nabla L

Where the rule changes

Two assumptions are doing work. The first is equal local batch sizes. If device $g$ has $b_g$ examples and $B = \sum_g b_g$ , the correct combination is a weighted sum, $\sum_g (1/B)\nabla S_g$ , or equivalently a weighted average of local means with weights $b_g/B$ . A plain mean over devices is wrong when $b_g$ differ, because it gives a device with few examples the same say as one with many. The second is what the loss is divided by. If the normaliser is a token count rather than a batch count, use the token count: the same cancellation has to happen in tokens, not in sequences. Gradient accumulation has the same shape: accumulate sums over microbatches and divide once by the total number of examples or tokens, not once per microbatch and again at the end.

Production wrappers often settle this for you. A distributed data parallel wrapper commonly computes a local mean and averages gradients across workers, which matches the local-mean row in the table. If you are using one of those and you also divide the loss by the global batch size, you have applied two normalisations. Either let the wrapper average local-mean gradients, or bypass its averaging and use a sum reduction with a globally normalised loss. Do not mix the two and hope the optimiser absorbs it: the gradient scale is wrong by $n$ , and a learning rate tuned under one convention will be wrong under the other.

A quick numerical check is to run one step on a single device with the whole global batch and compare the update with the multi-device update after the chosen reduction. They should agree to floating-point tolerance. If the multi-device update is smaller by eight in the eight-device example, you averaged when you should have summed. If it is larger by eight, you summed when you should have averaged. The test costs one step and catches a factor of $n$ before it reaches a learning-rate sweep.

The ring itself is not the issue. Its two phases move chunks and add them; the finished chunk is a sum across devices. An average is that sum with a division by $n$ at the end. The communication cost is unchanged: the bytes each device sends are $2(n-1)s/n$ , which at eight devices is $2 \times 7/8 = 1.75$ times the gradient size. One scalar division separates the two conventions, but that division is either correct or wrong by a factor of eight.

So the safe reading of a loss divided by global batch size is: this gradient is already globally normalised, and the exchange should sum. The safe reading of a local mean loss is: this gradient is locally normalised, and the exchange should average, or sum and then divide by $n$ . The all-reduce is a mechanical operation. The normalisation is a choice you make before it, and the choice has to match the loss.

Worth keeping

A sum all-reduce and a mean all-reduce differ by a factor of the number of devices.
Dividing the loss by global batch size and then averaging gradients makes the gradient too small by the number of devices.
The ring exchange computes a sum chunk by chunk; an average is that sum followed by a division by the number of devices.

Where this goes next

This question is one step of Training Across Many Machines, a course that works through its subject from the beginning. The page that teaches this part of it is Adding Up the Gradients, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course

Should the gradients be summed or averaged across GPUs?

Where the rule changes

Worth keeping

Where this goes next

Read alongside