The Space Belongs to the Word That Follows It
Last timeHow Big the Vocabulary Should Be
Almost every vocabulary attaches the leading space to the word, so the same word is a different symbol at the start of a line than in the middle. Several famous failures start here.
Why the space gets attached
The merge algorithm has no idea what a word is. It counts adjacent bytes. A
space is a byte like any other, and it sits immediately before almost every
word in the corpus, which makes the pair of a space and a common first letter
one of the commonest pairs in the entire count.
So the space merges early, and it merges forwards. Not because anybody decided
the space belongs to the following word, but because the pair of space and the
letter after it is far more frequent than the pair of the final letter and the
space before it. English words begin with a smaller set of letters than they end
with, and the leading pair is more predictable, so it wins the count.
This is also why writers of tokenizer code talk about the space-prefixed form as
the normal form and the bare form as the special case. In running text, a word
almost always has a space before it. The bare form occurs only at the start of a
document, after a line break, after an opening bracket or quotation mark, and
after a hyphen.
One word, several symbols
The consequence is that a single word in the language is several unrelated
entries in the vocabulary.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents