Answers · How models work

If you ban certain tokens, does that change what the model would have said?

In shortBanning tokens leaves the model's raw scores alone but changes the probabilities you sample from, so the text can change.

Yes, in the sense that matters for the text you get: the model's raw next-token probabilities are unchanged, but the distribution you sample from is changed, so the output can be different. Zeroing the banned tokens and dividing by the mass that remains makes every allowed token more likely than it was, and it can make a token the model barely considered the most likely choice.

Masking is done before a token is chosen, at every step. Let $p_i$ be the probability the model gives token $i$ , and let $m_i$ be one if that token is allowed here and zero if it is banned. The masked probability is not $p_i$ with the banned entries simply deleted: the survivors have to be rescaled so that they sum to one.

p'_i = \frac{p_i \, m_i}{\sum_j p_j \, m_j}

The divisor is the total probability the model put on allowed tokens, $\sum_j p_j m_j$ . Every survivor is divided by that same number, so the ratio between any two allowed tokens is exactly what the model gave them. What changes is the amount of mass each one holds. A token with probability 0.10 next to a token with probability 0.30 is still a three-to-one underdog, but its absolute chance can go up or down depending on how much illegal mass was removed.

What the numbers do

Take a five-way split in which three labels are legal with probabilities 0.30, 0.10 and 0.05, an opening quotation mark is banned at 0.40, and all other tokens are banned at 0.15. The allowed mass is $0.30+0.10+0.05=0.45$ . Dividing each survivor by 0.45 gives the masked distribution.

FIG 1

token	raw probability	allowed?	masked probability
label A	0.30	yes	0.667
label B	0.10	yes	0.222
label C	0.05	yes	0.111
opening quote	0.40	no	0.000
all other tokens	0.15	no	0.000

The ratios survive: $0.30/0.10=3$ and $0.667/0.222=3$ ; $0.10/0.05=2$ and $0.222/0.111=2$ . The absolute numbers do not. The first label rises from 0.30 to 0.667. The quotation mark, which was the single most likely token before masking, is now impossible. Nothing about the model's scores changed; the conditioning changed which scores were allowed to count.

The case that exposes the mistake

A common guess is that masking only removes the banned tokens and leaves the rest as they were. That cannot be right, because the remaining probabilities would no longer sum to one. The extreme case shows how far the correction can go. Suppose two tokens are allowed with probabilities 0.010 and 0.005, and the other 0.985 is banned. The allowed mass is $0.010+0.005=0.015$ . The masked probabilities are $0.010/0.015=0.667$ and $0.005/0.015=0.333$ . A token that had one percent of the model's mass now has two thirds of the distribution. That is not a filter after the fact; it is conditioning on a set the model did not favour, and the conditioning creates a choice the unconstrained model would almost never have made.

The effect is not confined to one step. Because the mask is applied to the text so far at every step, later probabilities are computed for a prefix the model would not have produced without the mask. The model may have been about to start a sentence, and the mask removes that option and forces the start of a legal structure. So even though every local ratio among allowed tokens is preserved, the sequence of tokens can be one the unconstrained model would not have said. That is why two systems with identical weights and identical prompts can produce different text when one has a format constraint and the other does not.

What a ban cannot do

Within a step the mask chooses what is possible and the model still chooses among those possibilities. Across steps, choosing the most probable allowed token is only a local rule. The most probable valid output is a global property, and a constraint can walk the text into a legal prefix from which every valid continuation is bad, then compel one of them. The other cost is quieter: the model can no longer hedge, refuse, or write a sentence instead. Uncertainty is routed into a confident, perfectly valid answer, and the format check passes on everything, so it stops being evidence that the answer is any good. It cannot make the model believe the answer either. The formula also assumes some allowed mass remains; if no token is legal in a state, there is no distribution left to sample from. And the allowed sets must be built against the actual token vocabulary, because a single token can span a quote mark, a colon and the start of a word.

Worth keeping

Zeroing banned tokens and renormalising preserves the ratios among allowed tokens but changes their absolute probabilities.
A token that had one percent of the mass can become two thirds of the distribution when the other ninety-nine percent is banned.
A guaranteed format is not a guaranteed correct answer.

Where this goes next

This question is one step of How a Model Picks Its Next Word, a course that works through its subject from the beginning. The page that teaches this part of it is Forcing It Into a Format, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course