Answers · Putting a model into service
How does a document the model retrieved end up giving it instructions?
In shortA retrieved document gives instructions because its text is pasted into the same prompt as your instructions, so the model cannot tell data from command.
Retrieved text gives instructions because it is concatenated into the prompt string alongside the system message and the user question, and the model reads one channel in which all of it has the same status. The step that changes the text from data into instruction is the join, not the retrieval and not anything the model decides.
Count the sources in a prompt: system message, retrieved documents, fetched pages, tool results, uploaded files, earlier turns, a summary of earlier turns. Only the first is written by the team that owns the system. The rest are text other people produced, and each is a surface somebody can write to. The internal knowledge base is not an exception: it contains support tickets, pasted emails, wiki edits and old imports. Internal means inside the company, not vetted for this purpose.
The join is the instruction
A page an attacker can write is fetched by a crawler, chunked, embedded and stored in the index with its source URL. A user asks an ordinary question about the topic. Retrieval scores the attacker chunk as relevant, because the chunk is about that topic, and selects it. In the code that assembles the prompt, a template has a hole for retrieved context. The chunk goes into the hole. The source URL does not. At that line, a row in a database with a timestamp and a score becomes a run of characters inside the prompt. There is no function call named trust_this; there is a string concatenation or a template fill, and the text is now in the same place as the sentence you wrote about being a helpful assistant.
Before the join the chunk is data: it has a source, a fetch time and a retrieval score. After the join it is text with no provenance, because the prompt has one channel and nothing in it records which token came from which URL. The retrieval layer knew exactly where the chunk came from and discarded that fact at the join. Reconstructing it afterwards is impossible from the prompt alone. This is why an incident is hard to explain: the system never made a decision to trust the document, so there is no decision to audit, only a template with a hole and some text that went into the hole.
Why the natural guess fails
The natural guess is that the model should treat a document as data, especially if you wrap it in a sentence saying that the following text is information only and never instruction. That fails because the wrapper is also text in the same channel. The system message and the attacker sentence are both tokens; the model has no register that marks one as trusted instruction and the other as untrusted quotation. The join threw away the only thing that could have distinguished them, which was the source. A model told to ignore instructions inside documents is being given a tendency, not a control: a sufficiently convincing document gets past it, and the instruction itself sits in the same channel as the document it is meant to neutralise. The attacker also does not need to type into the chat. They write the page, and your crawler, index and retrieval do the rest.
If the retrieved text never reaches the prompt as free text, the mechanism changes. A separate process that extracts a single fixed field and inserts only that field leaves no room for a sentence of instructions. If the model can be convinced but its tools can only write to a queue your own code reads, the instruction may still be followed, but the consequence is bounded. A measured comparison of defences ranks limiting what tools can do above checking retrieved text before the join, above checking the outbound action, above instructions to ignore documents, above marking document boundaries. The top of that ranking does not depend on the model reaching the right conclusion about the text it just read. The prompt edits at the bottom are the smallest share.
That is the boundary of the mechanism. A document can give the model instructions because the join gives it the same status as your instructions. Nothing about the model's beliefs changes that. What changes the consequence is what the model is allowed to do after it has read the document and believed it. If the only outbound tool writes to a queue your own code reads, a model that has been fully persuaded to exfiltrate the context has no path out. The injection is still real; the blast radius is a number you chose at the tool boundary.
Worth keeping
- Retrieval does not create trust; the concatenation into the prompt does.
- The source of a chunk exists in the retrieval layer and is thrown away at the join, which is why the prompt has no provenance.
- Telling a model to ignore document instructions is a tendency, not a security control; limiting what its tools can do is the defence that survives a fully convinced model.
Where this goes next
This question is one step of What a Model Refuses, and Why, a course that works through its subject from the beginning. The page that teaches this part of it is Instructions Hidden in the Data, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- How many legitimate requests should I be willing to block in order to catch the bad ones?Block no more legitimate requests than one prevented harmful answer is worth: the break-even count is the harm cost divided by the refusal cost.
- If someone fully jailbreaks the model, what actually stops it from deleting my database?Nothing in the model stops it. What stops it is your code: the delete tool either cannot name your database or cannot delete irreversibly.
- How do I write a safety rule I can actually test, instead of just saying 'be harmless'?Name the action, the cases it covers, the cases it deliberately leaves out and what happens when it fires; then check that two readers label the same twenty borderline requests the same way.
- What's the difference between a model that refuses because of how it was trained and one that just gets blocked by a filter sitting in front of it?A trained refusal is a habit in the model's weights that an unusual framing can slip past; a filter is separate software that reads the text as data, so no wording reaches it.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.