Answers · How models work

Why does a model cost the same to run no matter what you ask it?

In shortIn an ordinary model every weight runs on every token, so the arithmetic per token is fixed by the model size, not by how hard the question is.

In an ordinary model every weight is applied to every token, so the arithmetic per token is fixed by the size of the model rather than by the content of the question. A hard prompt and an easy prompt cost the same per token because both are pushed through the same complete set of weights.

The arithmetic

A token arrives as a vector. Each layer multiplies it by matrices whose entries are the parameters, and in a dense model none of those multiplications is skipped. If the model holds $p$ parameters, a token touches $p$ parameters. Difficulty does not enter that count. The input determines the values in the vector, not which weights are read. A question asking for a capital city and a question asking for a proof both produce one vector at a time, and each vector meets the same matrices.

This is unusual. A reference book does not take longer to consult because it has more entries, because an index tells you where to look. A dense model has no index: it reads every weight on every pass. The absence of an index is why the size of the model and the cost of one token are the same number.

FIG 1What a token runs when copies exist

: active parameters: what one token touches, which sets the time per token
: the shared parts, run by every token regardless of routing
: how many copies each token is sent to, usually one or two
: how many copies exist
: the parameters held across all the copies together

In the dense case there is nothing to choose between: , so cost and capacity are the same number. The fraction is the only thing that can separate them.

The per-token price is constant, not the total bill. A model that emits twenty tokens does twenty times the per-token arithmetic of a model that emits one, because each token is a separate pass. That is why a hard question can cost more in practice: it often produces a longer answer or a longer chain of reasoning. The extra cost comes from the number of tokens, not from extra work inside a token. If two prompts produce the same number of input and output tokens, a dense model charges them the same.

The natural guess

The natural guess is that a harder question should make the model work harder, in the way a person pauses and thinks. That would require a mechanism that reads the question, decides that more computation is needed, and runs extra weights for that token. An ordinary model has no such mechanism. It has one path, and the path is the whole model. The router in a sparse model does make a choice, but it chooses which copies to visit, not how many: $k$ is fixed, so a difficult token still visits $k$ copies. The constancy is a structural consequence of applying every parameter to every input, not a law that models must obey.

Replacing the largest block with several copies and sending each token to a few of them separates what is held from what is run. Let the shared parts hold 1 billion parameters, let the copies hold 8 billion between them across 8 copies, and let each token visit 1 copy. Then the active count is $1 + (1/8)\times 8 = 2$ billion, while the held count is $1 + 8 = 9$ billion. The ratio is $9/2 = 4.5$ . A harder token still visits one copy, so its active count stays at 2 billion. More knowledge is bought by adding copies, which adds to the held count and adds nothing to the per-token arithmetic.

p_a = 1 + \frac{1}{8}\cdot 8 = 2,\quad p_{\text{held}} = 1 + 8 = 9,\quad \frac{p_{\text{held}}}{p_a} = \frac{9}{2} = 4.5

The chooser is a small matrix that scores the copies, so it adds a little arithmetic and no large amount. The routing decision can differ from token to token, and the set of visited copies can differ, but the number visited is fixed by design. That is why two tokens with very different content can have the same active parameter count.

The arithmetic is saved; the memory is not. The 9 billion parameters in the example all have to be stored, even though a token touches only 2 billion of them. A model that holds many copies therefore needs memory for all the copies, and moving tokens to the copy on another machine adds communication that no parameter count shows. That is why the arrangement is a trade rather than a free increase in knowledge. It is worth it when arithmetic is the bottleneck and memory is spare; it is a poor deal when memory is the constraint.

Where it stops

The constancy stops if the architecture itself changes the amount of computation per token. A model that loops until it is confident, or one that varies $k$ with the input, can spend more arithmetic on a harder token. Standard dense models and standard sparse mixtures do not do that: dense runs all, sparse runs a fixed number of copies. Batching and hardware can also change the wall-clock time you observe, but they do not change the per-token parameter count. The question's difficulty can change the number of tokens, and that is usually what people are measuring when they say a hard question cost more.

Worth keeping

Cost per token in a dense model is proportional to the number of parameters, because every parameter runs on every token.
A longer answer costs more because it contains more tokens, not because any one token is harder.
Sparse copies separate held parameters from active parameters, but they move the cost into memory rather than removing it.

Where this goes next

This question is one step of Many Specialists, One Model, a course that works through its subject from the beginning. The page that teaches this part of it is Capacity You Do Not Have to Run, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course