Basis Lab

Answers · Putting a model into service

Why does the model have to be asked again after the tool hands back its result?

In shortBecause a tool call is a request, not an answer. The result must be added to the conversation and the model called again to read it and reply.

Because a tool call reply is not an answer. It is a request, and the result only becomes visible to the model when it is appended to the message list and the model is called on that longer list. The first call ended with reply.calls non-empty, so the program has no text to return to the person. What it has is a request for run(call), and running that request produces a result that exists in the program, not in the model. The second call is not a retry of the first. It is a different call with a longer input, and it is the first call whose input contains the thing the model asked for.

r1=model(messages,tools=TOOLS),messages←messages+[r1],result=run(c1),messages←messages+[{role:tool,id:c1.id,content:result}],r2=model(messages,tools=TOOLS)r_1 = \text{model}(\text{messages}, \text{tools}=\text{TOOLS}),\quad \text{messages} \leftarrow \text{messages} + [r_1],\quad \text{result} = \text{run}(c_1),\quad \text{messages} \leftarrow \text{messages} + [\{\text{role}:\text{tool},\text{id}:c_1.\text{id},\text{content}:\text{result}\}],\quad r_2 = \text{model}(\text{messages}, \text{tools}=\text{TOOLS})

Take a question and one tool call. The model reads a message list holding only the question and returns a reply whose calls list contains one call, say c1. The program appends that reply. It runs c1, gets a string, and appends {'role': 'tool', 'id': c1.id, 'content': result}. At this point the conversation has three messages: the question, the assistant's call, and the tool result. The model has still never read the third one. Only model(messages, tools=TOOLS) on that three message list can read it. If the program returns the result itself, it is returning a database row or a web page to a person who asked a question. It is not returning an answer.

FIG 1The loop
python
messages = [{'role': 'user', 'content': question}]

for step in range(MAX_STEPS):
    reply = model(messages, tools=TOOLS)
    messages.append(reply)
    if not reply.calls:
        return reply.text
    for call in reply.calls:
        result = run(call)
        messages.append({'role': 'tool', 'id': call.id, 'content': result})

return 'stopped after the step limit'
The append and the second call are separate lines because the result is data for the next call, not a return value from this one.

The result is a message, not a return value

The natural guess is that the tool result travels back along the same request. It does not. For the purposes of the loop, a model call is a function of the message list and the tool descriptions. It has no memory between calls beyond what is in messages. The previous reply is an object in the program, and the tool result is another object. Neither is inside the model. Appending them to the list is how the next call sees them. This is also why the tool result is appended as a message with a role and an id rather than returned as a value from run. A return value is for the program. A message is for the model.

FIG 2What each call can see
stepcallmessages passed to modelwhat comes backwhat happened
1firstthe question onlya tool call, c1, and no final textThe reply is a request for run(c1).
2secondquestion, assistant call c1, tool message with resultthe text answerThe result is now in the only place the model can read it.
2 steps
The second call is the first one whose input contains the result, so it is the first one that can produce an answer grounded in that result.

The assistant message that carried the call is part of the history for a reason. Without it, the tool message has no matching request: the model would see a result with no record of what was asked, and the id would point at nothing. The loop appends the assistant reply before running the call, then appends each tool result with the id of the call it answers. That pairing is what lets the model connect result to request. When there are several calls, the program appends one tool message per call, all before the next model call. The next call reads all of them together.

Where the second call disappears

Nothing about this changes when a framework hides the second call. A hosted tool runner may perform the append and the follow-up call on the server, so the client sees one request and one response, but the sequence is the same: the result is added to the conversation and the model is invoked again. The place it genuinely stops being true is when there is no tool call to answer. If the first reply has reply.calls empty, the loop returns reply.text immediately and no second ask happens. A fatal tool error can also end the loop before the second call, but then the task is not finished; the program is choosing to stop. The second call is required only when the model has asked for something and the program intends to let it use the result.

The cost of the second call is that it reads the whole history, not just the result. That is why a long task becomes expensive faster than the number of steps suggests, and why trimming what goes into the tool result matters. A loop that appends the result but never calls the model again has done the work and thrown away the only turn in which the answer could be written.

Worth keeping

  • A tool call reply contains no final text, so returning it to the person would return a request rather than an answer.
  • The model sees a tool result only after the result is appended to the message list and a new model call is made on that list.
  • If the first reply has no tool calls, no second call is needed, because that reply is already the answer.

Where this goes next

This question is one step of Models That Use Tools, a course that works through its subject from the beginning. The page that teaches this part of it is The Loop, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why does the model pick the wrong tool when two of them sound almost the same?The model only sees the written description, so when two descriptions overlap it cannot tell the tools apart and confidently picks the more familiar reading.
  2. Can a web page my agent reads make it do something I never asked for?Yes. A fetched web page can contain a sentence aimed at the model, and nothing in the model's view marks it as different from something you wrote yourself.
  3. How do I know if a failed task was the model's fault or the tool's?You cannot tell from the reply alone: grade the final state of the data, log what each tool actually returned, and assign every failure to the model, a tool, or your own loop code.
  4. Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
  5. Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
  6. How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.

Read alongside