Answers · How models work

What happens to the model if most of the training data comes from one source?

In shortThe model becomes specialised to that source and loses ground on everything else, because a training mixture is a set of shares that must sum to one.

If most of the training data comes from one source, the model becomes a specialist in that source and a poor generalist everywhere else. The shares in a mixture sum to one, so raising the dominant source does not add data to the model: it subtracts from every other source at once, and it subtracts in proportion to what each source had before.

p_i = \frac{w_i}{\sum_j w_j}, \qquad \sum_i p_i = 1.

Take the simplest case. Source A has 90 per cent and source B has 10 per cent. Raise A to 95 per cent: B must fall to 5 per cent. It has not lost five points because somebody chose to cut B; it has lost them because the total has to stay at 100. Every other source is multiplied by the same factor, $(1 - 0.95)/(1 - 0.90) = 0.5$ . A source that was 2 per cent becomes 1 per cent; a source that was 0.5 per cent becomes 0.25 per cent. The dominant source now has $95/5 = 19$ times as much share as all the rest combined, where before it had $90/10 = 9$ times.

There is no adding

With more than two sources the same factor applies to every source that was not raised. If code goes from 13 per cent to 20 per cent, the other weights are multiplied by $(1 - 0.20)/(1 - 0.13) = 0.80/0.87 = 0.9195$ . Web at 67 per cent becomes $67 \times 0.9195 = 61.6$ per cent. Books at 8 per cent becomes 7.4 per cent. Papers at 4 per cent becomes 3.7 per cent. Each is down about 8 per cent of itself, not 8 percentage points. The largest absolute loss falls on the largest remaining source, which is why the headline overall loss looks small.

FIG 1Raising code's share from 13 to 20 per cent

step	source	share before	share after	change, per cent of itself	what happened
1	1	13	20	54	Code, raised deliberately from 13 to 20 per cent.
2	2	67	61.6	-8	Web, reduced by eight per cent of itself; the largest absolute change.
3	3	8	7.4	-8	Books, down by the same proportion.
4	4	4	3.7	-8	Papers, down the same; at this size the loss can matter.

4 steps

The factor for every source that was not raised is , so each falls by about 8 per cent of itself. The unlisted remainder falls by the same factor, which is what makes the new shares sum to one.

The natural guess is that a source is used because there is a lot of it, and that giving it most of the mixture will simply make the model better at that source. It will. It will also make the model worse at whatever the other sources were carrying, and the model has no separate dial that says 'keep general ability fixed'. The sources at risk are the small ones. A source at 4 per cent that falls to 3.7 per cent has lost only 0.3 percentage points, but a capability that was only just represented enough to be learned can be rounded away entirely. This is why setting shares by availability is not a neutral choice: it makes the largest pile the default and treats every other capability as the residue.

What the model learns

If one source is most of the data, the model's default behaviour follows that source. A web-dominated mixture produces a model that writes like web text and handles web-shaped tasks; code, mathematics, formal text and smaller languages are not absent, but they are learned from a smaller and possibly repeated sample. At 95 per cent, all other sources together have 5 per cent, so any capability that needs a distinct syntax or vocabulary is competing for a twentieth of the training signal. At 99 per cent it is competing for a hundredth. The model does not fail at those capabilities in a clean way. It becomes uneven: good on the dominant source, brittle on the rest.

Repeating a small source is the standard way to raise its share without changing the large source's weight. A source that is 0.5 per cent of the mixture, repeated twelve times, becomes $0.5 \times 12 = 6$ per cent. That is a deliberate and documented choice, but the repeat count is also the memorisation dial. Twelve passes over the same text is the regime in which a model starts to reproduce passages verbatim. If the text is public and stable, that may be acceptable; if it contains personal or sensitive material, the repeat count should be capped in the low single digits. Repeating the already dominant source is rarely the right move, because it raises a share that is already large and increases memorisation of the same pile.

When a dominant source is correct

A dominant source is not automatically a fault. If the task is one source, a specialist trained mostly on that source is the right model, and the loss on other capabilities is the price of the specialism. The fault appears when the dominance was set by what was available rather than chosen, because then the trade was never priced. It also depends on how broad the dominant source is: a large, diverse web corpus covers many kinds of text, while a large single-site or single-author corpus covers one style. Finally, short experiments can change sign with scale, so a mixture that looks good at small scale should be checked per capability before it is trusted at full scale. Compare code, mathematics, general knowledge, long-form writing and non-English separately; a single overall loss is dominated by the largest source and will not show you the small source that disappeared.

Worth keeping

A mixture's shares sum to one, so raising one source's share lowers every other source's share.
If one source takes 95 per cent, all other sources together take 5 per cent, a ratio of 19 to 1.
Repeating a small source to raise its share also raises the model's tendency to reproduce that text verbatim.

Where this goes next

This question is one step of Where Training Data Comes From, a course that works through its subject from the beginning. The page that teaches this part of it is How Much of Each, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course