Showing Instead of Explaining

Last timeInstructions That Survive Contact

Three worked examples usually beat three paragraphs of description, and the examples have to cover the awkward cases, because everything in them is copied exactly.

Showing rather than explaining

Consider trying to describe, in words, what a good reply to a support message

looks like. Length, tone, whether to greet, whether to apologise, how much

detail about the policy, whether to offer a next step, how to handle a question

that has no answer, what to do with the customer name.

That is a long document and it will still be incomplete, because most of what

makes a reply good was never consciously decided by anybody.

Now show three real replies. Everything above is settled at once, including the

dozen properties nobody would have thought to write down, and it takes a

quarter of the room.

FIG 1What a description fixes against what an example fixes

	fixed by a description	fixed by one example	tokens to state in words
the format	1	1	40
the typical length	1	1	15
the tone	1	1	60
whether to greet	1	1	20
how much policy detail	1	1	80
what to do with an unans	1	1	90
the level of hedging	0	1	0

The last row is the interesting one. The amount of hedging in a reply is something people notice immediately and essentially never specify, because putting it into words is hard and nobody thinks of it. An example settles it without anybody having to articulate it, and that silent transfer of a dozen unstated properties is the real argument for examples over description.

FIG 2How many examples a room budget affords

: tokens set aside for examples
: the length of one example
: examples that fit

Two thousand four hundred tokens at a hundred and eighty tokens an example is thirteen, which is almost certainly more than the useful number. The returns fall off sharply: the first example settles the format, the second and third settle the edges, and the tenth is usually paying rent. The expression is worth having because it makes the trade visible, since the same room could hold three more retrieved passages, and that comparison is the subject of the fifth lesson.

Examples at the edges

Given that only a few examples fit, which few? The instinct is to pick typical

cases, and it is wrong.

A typical case teaches nothing that was in doubt. The model already handles the

customer who asks a clear question about a recent order. The room is better

spent on the cases where the right behaviour is not obvious: the message that

asks three things at once, the one where the answer is that nobody knows, the

one that is angry, the one that is empty, the one where the correct output is a

refusal.

FIG 3Two sets of three examples, same room

step	the set	what the examples show	ordinary cases handled	awkward cases handled
1	three typical messages	the format and the tone	96 per cent	48 per cent
2	one typical, one with no answer, one with several questions	the format, the tone, and two edges	95 per cent	79 per cent
3	three awkward messages and no typical one	the edges, and a format slightly skewed towards them	88 per cent	84 per cent
4	one typical, two awkward, with an explicit note on each	everything, with the reasoning visible	96 per cent	86 per cent

4 steps

Illustrative numbers, and the shape is what matters. The first row is what most systems ship. The third row shows that going all the way to edges costs something on the ordinary case, because the examples also teach what a typical answer looks like and all three were atypical. The last row is the usual best arrangement: one ordinary case to anchor the format, the rest spent on edges, and a short note on each saying why it was answered that way.

FIG 4Where examples pay for themselves

	format errors per hundre	format errors per hundre	examples needed
a fixed reply structure	9	2	3
a classification into si	14	3	6
a short structured recor	11	1	2
a free text summary	6	4	3

The second row needs more examples than the others because there are six categories and the set has to show the space of labels rather than just the format, which is one of the more robust findings in this area. The last row shows the weakest case: a free text summary has no format to get wrong, so examples are doing less work and three of them may not be worth the room.

Examples are copied exactly

The property that makes examples powerful also makes them hazardous: everything

in them is a signal, including the parts nobody intended.

If all three examples are short, the replies will be short. If two of them begin

with the word certainly, that word will appear constantly. If one contains a

typographic quotation mark because it was pasted from a document, that mark will

show up in output. If every example happens to be about a refund, questions

about delivery will get answers shaped like refund answers.

The most costly version of this is an example containing a mistake. An example

that formats a date as the third of April where the convention is the fourth of

March will produce that error indefinitely, and it is nearly invisible because

the output looks well formed.

The lesson stops here

1 more paragraph to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents