If it's all just English text, why does where it came from matter?

The words are the same English either way; what differs is who was in a position to write them. A source is a selection made before anyone applies a filter, and it fixes which speakers, registers and subjects are present in the corpus at all, so two corpora of equal size and equal average quality can teach a model quite different things. Provenance matters because the lean sits in the text itself, not in how good the text is.

A quality filter asks one question about a document: is it well written, coherent and informative? That question is silent about which well-written, coherent, informative documents exist to be filtered. Delete every badly written page from a web crawl and you still have a web crawl. The size of the effect is easiest to see by counting what each source can actually supply, before any mixture is chosen.

The crawl offers 2400 billion tokens and 55 per cent of them survive cleaning, so 1320 billion usable. Books offer 30 billion at 90 per cent, so 27 billion usable. That is a factor of 49 between the two largest and one of the smallest. The pool adds up as $1320 + 153 + 48 + 27 + 19 = 1567$ billion tokens, and if you built the mixture out of whatever happened to be lying around you would get 84 per cent crawl, 10 per cent code, 3 per cent papers, 2 per cent books and 1 per cent reference works. Nothing in that procedure looks at content at all.

Real mixtures do not do this. A 1.2 trillion token corpus with books at 8 per cent is holding 96 billion book tokens drawn from a pool of 27 billion, so the books have gone round three and a half times.

The natural guess is that if everything is English prose of decent quality, the mix washes out. It does not, because quality is not the axis along which the sources differ. Take a product page written in 2019 for a software company and an essay written in 1974. Both are fluent, both are grammatical, both pass any reasonable quality filter with the same score. The crawl holds a very large number of the first and very few of the second, not because the essay is worse but because the essay is old, probably unlinked and quite possibly not served fast enough to be worth fetching. The filter keeps both when it finds them; it never goes looking for what was never there. A quality score is a property of a document. The lean is a property of the population of documents, and no per-document score can see it.

Code shows the same mechanism with a crisp edge. Its usable share is 85 per cent, the second highest in the table, and what survives is open-source libraries, tools and infrastructure. The business logic that most working programmers spend their days on is almost absent, because it lives in private repositories. A model trained on that mixture writes confident calls into well-known libraries and struggles to lay out an application of its own. No filter asking whether the code is correct can add the missing material, because the missing material was never public.

A lean is only a fault when it points away from what you intend to do with the model. If the task is completing calls to a well-known library, a code-heavy mixture tilted towards open source is close to ideal, and the missing private business logic costs nothing. The size of the share sets the size of the effect: a source at 1 per cent of the mixture cannot move the model much in either direction, while the same source repeated three and a half times is doing real work. The arithmetic also assumes the permission factor is 1. In practice a fetched byte yields usable tokens at roughly $0.15 \times 0.5 \times 0.9 = 0.0675$, about one token in fifteen, and that last factor moves further between organisations than any other.

So the answer to whether provenance can be ignored is that it cannot, and the practical consequence is dull and durable: record where each document came from, at the level of the document rather than the source. A corpus that cannot say which document came from where cannot be re-mixed later, and re-mixing is the only lever the sums above give you.