Can a better model make up for a bad training set?

Write the measured loss as $L = H + R$. $H$ is irreducible uncertainty in the text itself: the next token genuinely cannot be predicted better from the context. $R$ is reducible structure the model has not yet learned. Parameters, architecture and training time buy down $R$. None of them buy down $H$, because $H$ is a property of the corpus. The ceiling is the best loss the data permits, and every improvement a team can purchase is the distance from where it is to that ceiling.

Take three runs with identical architecture. Run 1 is trained on a general web mixture and reaches held-out loss 2.31, but is weak at arithmetic and nobody notices at first. Run 2 adds mathematics at three per cent of the mixture. In a 1.2 trillion token mixture that is $1.2\times10^{12} \times 0.03 = 3.6\times10^{10}$, or 36 billion tokens. Its loss is 2.34, which is $2.34 - 2.31 = 0.03$ worse, yet its arithmetic is much better. Run 3 removes mathematics and increases web text. It has the best loss of the three, 2.29, which is $2.31 - 2.29 = 0.02$ better than run 1 and $2.34 - 2.29 = 0.05$ better than run 2, and it is the worst model for the intended use. The loss ranked the runs in the wrong order.

The natural guess is that a stronger model would have learned the arithmetic anyway. It would not, if the arithmetic was not in its training text. The held-out loss is computed on data drawn from the same mixture, so a corpus missing mathematics is judged by an evaluation that is also missing mathematics. A training run cannot report on text that never entered it. This is why absence is the expensive fault: it is silent in every number the run prints, and it surfaces later when somebody asks the model to do the thing in front of a customer. More parameters do not create information; they only fit the information that reached the weights.

The mixture is the specification. In a 1.2 trillion token corpus, 13 per cent code is $1.2\times10^{12} \times 0.13 = 1.56\times10^{11}$ tokens, or 156 billion. Two per cent conversation is $1.2\times10^{12}\times0.02 = 2.4\times10^{10}$, or 24 billion. A language that makes up a fraction of a per cent has that fraction of the tokens, and the model's competence in it follows the proportion. A better model can be more efficient with those tokens, but it cannot turn 24 billion tokens of conversation into the same thing as 804 billion tokens of filtered web pages, since $1.2\times10^{12}\times0.67 = 8.04\times10^{11}$. The forecast is available before training from the mixture chart.

If the skill is present but rare, a better model can help. It may reach a lower $R$ on the same tokens, so it can partly offset a thin slice of the mixture. That is a real effect and it is why scaling sometimes looks like a data fix. But it is bounded by the text that exists. If a language is absent, the count is zero, and no amount of compute changes zero. If the mixture is wrong in proportion, a stronger model can squeeze more from the underweighted slice, but it cannot make that slice the dominant source of competence. The ceiling moves down only when the corpus changes.

Where it stops being true is the boundary of the word bad. If bad means the corpus is missing a capability, the answer is no. If bad means the model has not yet extracted structure that is present, a better model or longer training can make up ground, because that is $R$ and it is for sale. If bad means the answers are the wrong shape, the repair is alignment, not a new corpus. If bad means the service is slow, the repair is serving. The question is not whether a better model can compensate in general; it is which term of $L = H + R$ the fault sits in, and whether the fault is in the corpus at all.

Before training, write the mixture down and list what the model is supposed to do. For each item, find the text that demonstrates it. That check takes a day. If an item has no text, no later choice will create it. If it has text, a better model may learn it more efficiently, and that is the only sense in which a better model makes up for a bad training set.