The Attacker Does Not Have to Be the One Typing
Last timeChecking What Goes Out
If your system reads documents, pages or tool output, somebody who can write into any of those is writing into your prompt. Here is the path, step by step.
Every door into the prompt
The previous lessons assumed the attacker is the person typing. Usually they
are not, and the reason is that very little of a modern prompt is typed by the
user.
Count the sources. The system prompt, which you wrote. Retrieved documents from
your own index. Web pages fetched by a tool. The result of an API call. A file
the user uploaded. Earlier turns of this conversation. Possibly a summary of
earlier turns, written by the model.
Every one of those except the first is text somebody else produced, and each of
them is a surface somebody can write to.
| writeable by someone out | reviewed before it is in | treated as untrusted by | share of prompt tokens, | |
|---|---|---|---|---|
| system prompt | 0 | 1 | 1 | 0 |
| internal knowledge base | 1 | 0 | 0 | 12 |
| customer support tickets | 1 | 0 | 0 | 40 |
| web pages fetched by a t | 1 | 0 | 0 | 95 |
| uploaded files | 1 | 1 | 0 | 8 |
| results from a partner A | 1 | 0 | 0 | 30 |
The knowledge base deserves a word. It feels internal and safe. It usually
contains content written by support staff, pasted from customer emails, synced
from a wiki anybody in the company can edit, and sometimes imported from a
public source years ago. Internal means inside the company, not written by
somebody you have vetted for this purpose.
Tracing one through
The exercise worth doing is following a single attack from where it was written
to where it had an effect. Nobody believes the attack until they write out the
path for their own system.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents