Judging Whether the Whole Thing Worked

Last timeText From Outside Is Not an Instruction

A tool using system has to be judged on finished tasks, not on individual replies, and every failure has to be attributed to the model, a tool or the loop you wrote.

The unit is a finished task

A system that answers questions is evaluated on answers. A system that does

things has to be evaluated on whether the things got done, which is a different

kind of test.

A test case for a tool using system has three parts. A starting situation,

usually a small database in a known state. A request, in the words somebody

would actually use. And a checkable final state: this order is refunded for

this amount, this record exists with these fields, this message was sent to

this address and no other.

The final state is what makes it a test. A transcript cannot be graded reliably,

because a reply that says the refund has been issued is produced just as readily

when it has not.

FIG 1Two ways to grade the same run

	can be checked mechanica	catches a wrong amount	catches a double action	catches a confident fals
grade the final reply	0	0	0	0
grade the final state of	1	1	1	1
grade both	1	1	1	1

Coded 1 for yes and 0 for no, and the first row is all zeroes on purpose. Grading the reply catches nothing in this table, because every one of these failures is compatible with a confident and well written reply. The second row catches all four and it takes a test database rather than a list of expected answers, which is the main reason evaluations of these systems are so often skipped: the harness is real work, and there is no shortcut that does the job.

How per step quality compounds

The arithmetic here is simple and the consequence is not intuitive.

If each step of a task succeeds independently with some chance, the task

succeeds only if every step does, so the task success rate is the step rate

raised to the number of steps. Ninety seven per cent per step sounds like a

system working well. Over twelve steps it is sixty nine per cent of tasks

finishing, which is one task in three failing.

FIG 2Task success against the number of steps

99 per cent per step97 per cent per step90 per cent per step

All three curves start in a respectable place and the bottom one has fallen below one task in ten by twenty five steps. Two lessons follow. The first is that the single most effective way to improve a system of this kind is to shorten its tasks, because removing steps raises the success rate faster than improving any individual step. The second is that per step metrics should never be quoted as though they described the system.

FIG 3Attributing the failures

: the share of tasks that fail
: the share failing because of the model
: the share failing because of a tool
: the share failing because of the loop

Twenty two per cent of tasks failing, nine of them from the model and five from the tools, leaves eight per cent belonging to the loop, which is the code you wrote. In most systems measured honestly this third number is the largest of the three, which is the opposite of where attention goes. It is also the cheapest to fix, since a loop fault is an ordinary bug with an ordinary repair, while a model fault needs a changed prompt or a changed model and a tool fault may belong to somebody else.

Attributing each failure

Three causes, three completely different repairs.

A model failure is a wrong decision given correct information: the wrong tool

chosen, a sensible result ignored, a conclusion that does not follow from what

came back, an answer given when more work was needed. The repair is a clearer

tool description, a changed instruction, or a different model.

A tool failure is a tool returning something wrong or nothing at all when it

should have worked. The repair is in the tool, and sometimes in somebody else

service.

A loop failure is everything in between: a step limit cutting a task off, a

budget exhausted by a task that was going fine, an error returned in a form the

model could not act on, a validation rule rejecting something legitimate, two

calls run at once that should not have been, a result truncated without saying

so.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents