Answers · How models work

Why can't you just add new tokens to a tokenizer after training?

In shortBecause each token id is tied to weights the model has already trained, so changing the vocabulary makes old numbers name different pieces while the model keeps the old meanings.

You cannot just add them later because a token id is not a label sitting outside the model. Each id owns a row of the embedding table and a row of the output table, and training has already written a meaning into both rows; changing the tokenizer changes which piece each id names while the model keeps the old meaning.

E\in\mathbb{R}^{V\times d},\qquad U\in\mathbb{R}^{V\times d},\qquad x\mapsto T(x)\in\{1,\dots,V\}.

What is tied to an id

Take a vocabulary of 50,000 pieces and a model width of 4,096. The embedding table is 50,000 by 4,096, which is 50,000 times 4,096, or 204,800,000 learned numbers. If the output table is untied it is another 204,800,000. Row k is not a name for piece k in the abstract; it is the only place the model stores what piece k means, fitted from every context in which id k appeared.

50{,}000\times4{,}096=204{,}800{,}000,\qquad 1{,}000\times4{,}096=4{,}096{,}000.

Appending 1,000 new ids at the end therefore costs 4,096,000 new embedding numbers plus 4,096,000 new output numbers, or 8,192,000 numbers in total if the two tables are separate. None of those numbers has been fitted. The old rows are still correct for the old ids, but the new ids start as noise, and the output row for a new id starts as noise too, so the model has no reason to prefer it and no learned meaning to attach to it.

The guess that fails

The natural guess is that the vocabulary is a setting, like a list of allowed words that can be edited and saved. It fails because the integers are arbitrary names and the model has no separate dictionary to consult. A refitted tokenizer still returns integers in the old range, so nothing errors: id k arrives exactly as before, the model applies row k, and the text that produced id k has quietly changed. The model is not updated, it is misinstructed.

FIG 1One lookup, two fitted rows

: the output rows, also fitted, one per id the tokenizer could produce
: everything the trained weights do in between
: the embedding row the piece id selects, fitted during training

The same integer selects both rows. A tokenizer edit changes T while E and U stay where training left them.

Appending at the end is the mild case, not a free one. If you also freeze the old segmentation so that every string that used to produce old ids still produces them, no old row is invalidated: the new rows are simply cold. But a tokenizer that has been given a new piece will usually use it when it is the better merge, so some text that used to arrive as two or three old ids now arrives as one new id. The model sees a row it never trained in place of a sequence it did train, and the rest of the weights were fitted to the old sequence lengths and boundary statistics.

Why retraining a corner does not fix it

You can replace or extend the embedding table, but every weight above it was fitted to the old statistics: which pieces follow which, how long a word takes, where the boundaries fall inside a number. Those weights do not know that the input language changed. Brief continued training recovers a lot, because the new rows and the top weights both move, but it recovers by rewriting part of a model that was fitted to another segmentation. It is a repair with a residue, not an update.

Where this stops being catastrophic is worth stating exactly. If the old ids keep their old meanings and the new ids are only ever emitted for text that never appeared in training, the model is merely ignorant of those ids, not confused by them; fine-tuning on text containing them can teach the rows. If the new ids are emitted for text the model already knew, the old meaning is not lost from the table, but it is attached to the wrong piece, and no amount of using the old rows will make that sequence read correctly. That is why adding a token and changing what an existing token means are different failures with the same cause.

T_{\text{old}}(\text{cat})=[a,b],\qquad T_{\text{new}}(\text{cat})=[c],\qquad E_c\ \text{untrained}.

The practical asymmetry is extreme: a day of measurement before training, or a partial recovery afterwards that never quite closes the gap. If the vocabulary is going to serve prose, code, numbers, tables and transcripts, the pieces per unit of meaning have to be counted on text that resembles what will actually arrive. That is not a modelling step. It is the step that decides what the modelling step is allowed to learn.

Worth keeping

A token id is an index into learned rows, not a label stored beside the text.
Appending a new token adds an untrained row, while changing an old token's meaning makes the model apply a trained row to the wrong piece.
The damage is silent because the new ids are still in range, so the model answers confidently in a language it was not trained on.

Where this goes next

This question is one step of Cutting Text Into Pieces, a course that works through its subject from the beginning. The page that teaches this part of it is Why It Cannot Be Changed Later, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course