Answers · How models work

When should I fine-tune a model instead of just prompting it better?

In shortFine-tune only when the model has been told clearly what you want and still will not comply reliably; missing facts need retrieval, and unstated instructions need a better prompt.

Fine-tune when the gap is behaviour, not knowledge or instruction. If the answer depends on a document or a price or a customer record, fetch it and put it in the context. If the model was never told what shape the output should take, write the instruction and show two examples. Train a small correction only when the model has been asked clearly, shown examples, and still does not comply reliably.

Sort the gap before choosing a tool

Three failures look identical from outside. The model has never seen your internal document; the model has never been told the output shape; the model has been told and will not hold to it. The first is knowledge and wants retrieval, because fetched text can be cited, corrected when it changes, and removed when a customer leaves. The second is instruction and wants a written instruction plus examples, which is an afternoon rather than a training run. The third is behaviour, a shape or tone or judgement that the model will not hold consistently even when asked clearly and shown examples. That is the only one of the three where training is the first good answer.

FIG 1The same gap closed four ways

step	setup cost in engineer days	extra tokens per request	score on an output shape task	score on a company facts task	what happened
1	1	900	0.79	0.44	An instruction and three examples in the prompt. Cheapest to build, and it carries nine hundred extra tokens on every single call.
2	8	1400	0.8	0.91	Retrieval added. The facts score jumps because the facts are now in front of the model, and the shape score barely moves.
3	14	60	0.93	0.45	A trained correction instead. The shape score jumps, the facts score does not move at all, and the per-request tokens collapse.
4	21	1400	0.94	0.92	A correction and retrieval together, which is the arrangement most serious systems end up with.

4 steps

Read the last two columns as a pair. No single approach lifts both, because they are closing different gaps, and the final row is cheaper per request than the second despite scoring higher on both.

Read the last two columns as a pair. Adding retrieval to the prompt takes the facts score from 0.44 to 0.91, a rise of 0.47, and moves the shape score from 0.79 to 0.80, a rise of 0.01. Swapping the long prompt for a trained correction takes the shape score from 0.80 to 0.93, a rise of 0.13, and moves the facts score from 0.44 to 0.45, a rise of 0.01. The final row adds both: 0.94 on shape and 0.92 on facts, while using 1,400 tokens per request rather than the 60 of the correction alone. No single approach lifts both columns because they are closing different gaps.

FIG 2Where the money goes in each approach

: the cost attributable to one request
: the number of requests served over the life of the system
: the extra tokens per request, from a long instruction and from retrieved passages
: the cost of one token
: the one-off cost of training a correction, spread across every request

Prompting and retrieval live in the first term and never go away. A correction lives in the second, which shrinks as traffic grows. At low volume the first term is cheaper and at high volume it is not, and the crossing point is usually lower than people assume.

Cost has a different shape

The trace gives the arithmetic. Retrieval carries 1,400 extra tokens per request; a correction carries 60. The difference is 1,340 tokens on every call. At 1,000,000 requests that is 1,340,000,000 extra tokens. At a token price of $0.50 per million, that costs $670: 1,340,000,000 divided by 1,000,000 is 1,340, and 1,340 times $0.50 is $670. The correction costs 14 engineer days against 8 for retrieval, so 6 extra days. At $800 per engineer day that is $4,800. The extra tokens cost $0.00067 per request, because 1,340 times $0.50 divided by 1,000,000 is $0.00067, so the 6 days break even at about 7,164,000 requests: $4,800 divided by $0.00067 is 7,164,179. Below that traffic the prompt and retrieval are cheaper; above it the correction is. Substitute your own token price and engineer day, because the shape is what matters, not these particular prices.

The quality crossing

On a behaviour gap with labelled examples, a small correction beats the prompt, but not below roughly fifty examples. The crossing is worth measuring. Take a correction whose score is 0.88 minus 0.26 times e to the minus t over 120, against a prompt fixed at 0.79. At 50 examples the correction scores 0.709, because e to the minus 50 over 120 is 0.659 and 0.26 times 0.659 is 0.171, so 0.88 minus 0.171 is 0.709. The prompt is ahead. At 200 examples the correction scores 0.831, because e to the minus 200 over 120 is 0.188, 0.26 times 0.188 is 0.049, and 0.88 minus 0.049 is 0.831. The correction is ahead.

0.88 - 0.26 e^{-t/120} = 0.79 \quad\Longrightarrow\quad e^{-t/120} = \frac{0.09}{0.26} = 0.3462 \quad\Longrightarrow\quad t = -120 \ln 0.3462 = 127.3

So the curves cross at about 127 labelled examples, not at thousands. Below about fifty, a correction trained on that little data mostly learns noise and the prompt wins. The prompt is not a weak baseline; a correction that fails to beat it has told you something about your data rather than about the method.

The natural guess is that training is the stronger version of prompting. It is not a stronger version of the same thing; it is a different intervention. Training on facts buries them in the weights, where they cannot be cited, corrected when they change, or removed when a customer leaves. Training before writing the instruction down is worse than that: the model cannot learn a rule nobody stated. And below roughly fifty labelled examples a correction mostly learns noise, so the prompt is not the weaker baseline. The order that fails is train first and ask what gap you had later.

FIG 3Which approach closes which gap

	company facts	output shape	house judgement	a new language
prompt only	0.44	0.79	0.61	0.52
prompt and retrieval	0.91	0.80	0.64	0.55
a trained correction	0.45	0.93	0.88	0.71
a full update	0.47	0.94	0.90	0.89

The marked cell is the one that decides most real deployments: a correction closes the behaviour gap at a small fraction of the cost of the row below it, and the row below only pulls ahead in the final column.

A full update is right in a narrow set of cases: very large in-domain text where a restricted correction runs out of capacity, a language the base model barely saw, or a change to the shape of the model that no correction added on top can express. The grid makes the narrowness visible. A correction scores 0.93 on output shape against 0.94 for a full update, and 0.88 on house judgement against 0.90, while a full update only pulls clearly ahead on a new language, 0.89 against 0.71. If none of the three gaps fits, the task is probably underspecified and the measurement needs attention rather than the weights.

Worth keeping

Missing facts belong in retrieval, not in the weights, because fetched text can be cited, corrected and removed.
A small trained correction is for unreliable compliance with an instruction the model has already been given clearly.
A full update is justified by abundant in-domain data or a structural change, not by a gap that a prompt or a lookup could close.

Where this goes next

This question is one step of Adapting a Model You Did Not Train, a course that works through its subject from the beginning. The page that teaches this part of it is Deciding What Actually Needs Changing, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course