Answers · How models work

Why do so many tokens end up routed to the same few experts?

In shortBecause picking and training reinforce each other: an expert chosen more often gets trained more, which gets it chosen more often again, so the traffic collapses onto a few.

Because choosing and training reinforce each other. Whichever expert is selected receives gradient and gets better at what it sees, which makes the chooser select it more often still, and the experts that were never selected produce no output at all, so nothing ever tells the chooser what they could have done. In an eight expert layer that starts with an even eighth of the traffic each, the busiest expert typically climbs to holding the large majority of tokens within a couple of thousand training steps.

The loop, one step at a time

Start from an accidental imbalance so small you would not notice it. One expert is picked for slightly more tokens than its siblings in the first few batches. More tokens means more training signal, so it becomes genuinely better at the kind of input it happened to see. The chooser now scores it higher, and it is right to: it is the better expert now, and the scoring is doing exactly what it was designed to do. So it receives more tokens again, which closes the loop. Four steps, each sensible on its own, and the layer ends up with one useful expert and seven inert ones. The irreversibility comes from the last step. An expert that is never picked emits nothing, so the chooser never observes whether it would have handled a token well, and there is no signal pushing back towards it.

Take one real batch of roughly 28,000 tokens spread over eight experts. An even split would be 3,500 tokens each. Here is a collapsed batch beside a balanced one, in thousands of tokens per expert.

FIG 1Tokens per expert in one batch, thousands

	c1	c2	c3	c4	c5	c6	c7	c8
collapsed	1.1	0.4	0.6	24.2	0.3	0.8	0.5	0.1
balanced	3.6	3.4	3.7	4.1	3.3	3.5	3.6	2.8

Both rows total 28,000 tokens. In the collapsed row one expert takes 24,200 of them, which is 86 per cent of the batch, while the other seven share 3,800 between them.

Two bills follow, and they are separate. The first is capacity: 24,200 tokens buy one expert, and the 3,800 remaining are spread so thinly across seven that none of them learns very much. The parameters were paid for in memory and are carried on every machine for the life of the model, so you get the deployment difficulty of a large model with the useful capacity of a small one. The second is time. A step finishes when its slowest expert finishes, and the busiest expert here has 24,200 tokens against 4,100 in the balanced row, so the step takes 24,200 divided by 4,100, about 5.9 times as long. The other seven experts sit idle for most of it.

Why a trained router does not spread the load by itself

The natural guess is that a router trained alongside the experts will learn to distribute work, or that a badly distributed router is a broken one. Neither holds. The gradient reaching an expert flows only through the tokens that expert was selected for, so an expert selected for nothing receives nothing to learn from and stays exactly as it was initialised. The chooser is not making an error when it concentrates traffic; it is correctly preferring the expert that has seen the most data, and that expert is better precisely because it saw the most data. There is no mistaken step in the loop to correct, which is why balancing has to be imposed from outside the loop rather than trained into it.

The traditional remedy adds a penalty to the loss. With $n$ experts, $f_i$ the share of the batch actually sent to expert $i$ , and $P_i$ the chooser's average score for expert $i$ over the batch whether or not it was selected:

L_{b} = \alpha\, n \sum_{i=1}^{n} f_{i} P_{i}

The product matters, and each half alone fails. The share $f_i$ comes from a discrete selection, so on its own it gives nothing to differentiate; the mean score $P_i$ can be made flat by a chooser that spreads its scores while still sending everything to one expert. Multiplied together they are smallest when both are flat, and the continuous half supplies the gradient. Check it on two experts with $\alpha = 1$ . Perfectly even, $f = (0.5, 0.5)$ and $P = (0.5, 0.5)$ , gives $2 \times (0.25 + 0.25) = 1$ . Collapsed, $f = (0.9, 0.1)$ and $P = (0.9, 0.1)$ , gives $2 \times (0.81 + 0.01) = 1.64$ . The penalty is 1.64 times larger on the collapsed routing, which is the pressure that pushes back.

The coefficient $\alpha$ is a dial between two failures rather than a setting with a right answer. Too small and the main objective overwhelms it and collapse happens anyway; too large and the chooser sends tokens to experts that handle them worse purely to even the numbers, costing quality. In practice it lands near a hundredth and has to be found again whenever the layer shape changes. The alternative avoids the dial entirely: keep one extra offset per expert, added to that expert's score for selection only and never in the blend weights, so it does not affect the output or the loss. After each step, nudge the offset down for an overused expert and up for a starved one, by a fixed small amount. Across a few steps the offsets might read $-0.02$ , then $-0.06$ , settling at $-0.07$ on the overloaded expert and $+0.06$ on the starved one, at which point traffic is even and they stop moving. Nothing has been added to the objective, but the offsets are a control loop rather than a guarantee, and they can lag a shifting distribution or oscillate if adjusted too aggressively.

Two things limit all of this. Balance within a batch is not balance within a sequence: an expert can receive its fair share of a batch while receiving nothing from most of the sequences in it, and where sequences arrive one at a time it is the second that decides how well the expert is trained. And with a very large number of experts and few tokens per batch, the shares are coarse, so the correction is noisy. Neither the penalty nor the offsets removes the feedback loop; they only hold it in check while the training distribution stays roughly where it was.

Worth keeping

An expert is only trained on the tokens it was selected for, so selecting it less is the same as training it less, which justifies selecting it less.
Collapse wastes capacity twice: parameters that are held in memory but never trained, and hardware that waits on one overloaded expert while the rest idle.
A balancing penalty trades traffic evenness against model quality through a coefficient that has to be retuned whenever the layer shape changes; per-expert score offsets avoid that trade but behave as a control loop rather than a guarantee.

Where this goes next

This question is one step of Many Specialists, One Model, a course that works through its subject from the beginning. The page that teaches this part of it is When Everything Goes to One Place, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course

Why do so many tokens end up routed to the same few experts?

The loop, one step at a time

Why a trained router does not spread the load by itself

Worth keeping

Where this goes next

Read alongside