Why is 'hello' a different token from ' hello' with a space in front?

The space is not padding. It is a byte, and the tokenizer merges it onto the front of the following word, so ' hello' and 'hello' are different symbols with no structural link between them.

The merge algorithm has no idea what a word is. It counts adjacent byte pairs in a corpus. A space sits immediately before almost every word, so the pair of a space and a common first letter is one of the commonest pairs anywhere. That pair merges in the first few rounds, and later merges extend it until the whole word, space included, is one entry. The tokenizer is not deciding that the space belongs to the word. It is following the count.

Nothing in the model connects these entries. There is no shared stem and no mechanism saying that the bare 'hello' and the space-prefixed ' hello' are the same word. Everything the model knows about one form it learned separately from the contexts that form appeared in. In practice the vectors are pulled together because the forms turn up in similar contexts, but that costs data and capacity. The space-prefixed form appears constantly; the bare form far less; the capitalised form less still; the all-capitals form rarely enough that it may be assembled from fragments.

The natural guess is that whitespace is invisible padding and should be ignored. It is not ignored, because the tokenizer does not remove it before counting. If you strip the leading space from your input, you do not get the same token with the space deleted; you get a different vocabulary entry. The model has to learn the relation between the two entries from data, and one of them has much less data behind it.

Take a 50,257-entry vocabulary split roughly as 17,400 space-prefixed words, 13,900 bare forms, 6,100 capitalised forms, 2,600 all-capitals or other variants, and 10,257 fragments, punctuation, digits and byte fallbacks. The first four figures add to 17,400 + 13,900 + 6,100 + 2,600 = 40,000 entries. The duplicate forms alone, the bare, capitalised and all-capitals entries, are 13,900 + 6,100 + 2,600 = 22,600, which is 22,600 ÷ 50,257 ≈ 0.45, or about 45 entries in every 100. That is not pure waste, since each form needs a representation, but the usable distinct vocabulary is smaller than the headline number, and the rarer forms are under-trained.

The same structure explains two familiar complaints. Asking how many times a letter appears in a word is hard because the word is one opaque symbol whose spelling the model never sees. A word at the start of a line behaving differently from the same word mid-sentence is not superstition: the two are genuinely different inputs, and the start-of-line form had less training behind it. Neither is a reasoning failure in the usual sense. Both follow from what the vocabulary made cheap and what it made rare.

This is a property of the tokenizer and its pre-splitting rule, not of language. If a tokenizer strips leading whitespace, or marks word boundaries with a sentinel such as ▁ instead of a raw space byte, the same word may still be split from its bare form, but the mechanism and the symbols differ. At the very start of a document the bare form is the normal one, because there is no preceding space to attach. The diagnostic that matters is the other end of the input: a prompt that ends in a space leaves the model continuing a token rather than starting a word. That split almost never occurs in natural text, so the model is predicting a rare bare-form entry after a stranded space. The output degrades, and the cause is invisible on screen. Delete the space and the prompt asks for the ordinary space-prefixed form again.