What a Submitted Program Can Actually Ask For

A service accepts a program it did not write, runs it against ten inputs, and

returns what it printed. That is the whole job. The program came out of a

model, or out of a stranger, or out of a model being steered by text a stranger

supplied, and nobody read it before it ran.

Confining it is the work. But nothing can be confined before it has been

described, and the description people reach for first is wrong in a way that

costs money. They describe the program. They read the submission, decide it

looks harmless, and run it. The description that holds is not the program at

all.

Strip a running program down to what it can do unaided and there are three

things: arithmetic, reading and writing the memory it already holds, and

jumping about inside itself. That is the entire unaided repertoire, and none of

it touches anything the operator owns. A program doing only those three things

cannot read a file, cannot send a byte anywhere, and cannot tell you what time

it is.

Everything else is a request. Open this path. Write these bytes. Make another

process. Connect to that address. Give me more memory. Each one is handed to

the kernel, which looks at the asking process, decides, and either does the

work on that process's behalf or returns a refusal the program is free to read

and try around.

So the question of what this code can do has a precise answer, and the answer

is not a property of the code. It is the set of requests the kernel will still

answer for the process the code runs in, and that set is configured by the

operator.

Two consequences follow, and both are uncomfortable. The first is that reading

a submission tells you about one sample: this program, this time. The reachable

set is the same whatever any particular submission happens to contain, so the

reachable set is the thing worth writing down. The second is that the cost of

an allowed request is the operator's cost, measured on the operator's machine,

because the kernel is the one doing the work. It does not stay inside the run

that asked for it.

An inventory that cannot be recited is a feeling. Ten items cover what a

hostile submission to this job actually tries:

• read a file holding credentials
• open a connection out
• fill the disk
• keep making processes
• spin the processor forever
• allocate memory until the host suffers
• write outside the scratch folder
• read another reader's submission
• reach the address the cloud uses to hand out machine credentials
• leave a process running after the answer was returned

They fall into four kinds of reach, and the grouping is what makes ten items

memorable. Something to read, which is the first, the eighth and the ninth.

Somewhere to send, which is the second. Something to spend, which is the third,

fourth, fifth and sixth. And somewhere to leave something behind, which is the

seventh and the tenth. A wall that stops one member of a group usually stops

the rest of it, which is why this inventory is ten items long rather than a

hundred.

A cost with no machine under it is an adjective, so take a concrete host: eight

cores, 16 GB of memory, a 40 GB disk with 20 GB of it free, one of a pool, and

the service pushing about ten submissions a second across that pool. Nothing in

the job as described so far limits what a submission may spend, and that is the

state the inventory has to be priced in.

Filling the disk is the dullest item on the list and among the most expensive.

The free space is 20 GB, which is 20,000 MB, and a program writing as fast as

an ordinary disk will take it, call it 180 MB a second, exhausts that in

$20{,}000 / 180 = 111$ seconds. Under two minutes, from one submission. What it

costs is not that submission: it is every other submission on the host, each of

which now fails at its first write, along with the log that would have

explained why, because the log needs somewhere to write too.

Making processes is the same shape, faster. A program that makes two more of

itself in each round holds $2^{k}$ of them after $k$ rounds, and

$2^{15} = 32{,}768$, which is the whole process table on a stock host. Fifteen

rounds. The sixteenth is refused, and it is refused for everybody: the session

somebody would log in with needs a process, and so does the thing that was

going to kill the offender.

Spinning the processor costs a slot. Eight cores means eight submissions

running at full speed, so eight programs that never finish and the host serves

nobody while the queue behind them grows at ten a second. Allocating memory

costs the host's headroom: of the 16 GB, call 14 GB usable once the system has

taken its own, and a loop touching 2 GB a second reaches that in

$14 / 2 = 7$ seconds. The kernel then recovers memory by killing something, and

it chooses by its own arithmetic rather than by guilt, so the process that dies

may be the one that was about to return a different reader's answer.

The reading items are priced differently, and that difference is the one to

hold on to. Reading a file that holds credentials costs whatever those

credentials open, for as long as they are valid, and that is not bounded by

anything on this host. Reaching the address the cloud uses to hand out machine

credentials is the same item with the file taken out of it: the platform will

issue a token for the role the machine runs as to an ordinary request made from

the machine, so the cost is the whole of that role's reach. Neither item is

made cheaper by a tighter sandbox. They are made cheaper only by the credential

being smaller, or by not being reachable from inside at all.

Opening a connection out costs the operator's name. The far end sees the

operator's address, so the service has become a relay for whatever the

submission wanted done, and the complaint arrives at the operator with the

operator's own address quoted in it.

Writing outside the scratch folder costs every later run. If what a submission

can reach includes the thing that scores the next submission, then the attacker

is not grading their own work, they are grading everybody's. A host taking ten

submissions a second and replaced once an hour runs

$10 \times 3600 = 36{,}000$ submissions between replacements, every one of them

after the overwrite.

Reading another reader's submission costs the meaning of the result. Where runs

share a folder, the ten inputs arrive with somebody else's answers beside them,

which makes a score worthless. The expensive part is the sentence the operator

then has to say out loud, which is that any run that day could have done the

same, and the day holds $10 \times 86{,}400 = 864{,}000$ of them.

The tenth item is last on the list and first in practice, because it is the one

that looks like nothing at all. A run returns its answer, the request closes,

and the submission's process carries on running.

There are two clocks on this job. The request's clock starts when the

submission is accepted and stops when the answer goes back, somewhere around a

second and a half for a program doing honest work. The host's clock does not

stop there. Anything the submission left behind is running on the second clock,

and the second clock is the one the operator is billed on.

Price it. If one submission in a hundred leaves something behind and the

service takes ten a second, that is

$0.01 \times 10 \times 3600 = 360$ processes an hour arriving on hosts that

report themselves as idle, because the thing doing the reporting is counting

requests and every request finished. Nothing is alarming at any single moment,

which is why this item is usually discovered by an invoice.

Nothing in the inventory above mentions intent, and it should not.

A submission from a model is not a decision by the model. It is a function of

the text that reached the model, and that text includes a prompt, and may

include a document somebody uploaded, a page somebody fetched, or the error

message from an earlier attempt. The producing system belongs to the operator;

its input does not. Treating the output as trusted because the system that made

it is yours is the whole of the mistake.

So the safety claim worth making never contains the word hostile. This

submission is not trying anything is a statement about one sample, and it

cannot be checked at the rate submissions arrive. A submission trying all ten

of these reaches nothing it should not is a statement about the host. It is the

same sentence for every submission, it stays true while nobody is watching, and

it can be tested. Only the second kind of claim is worth building on.

What follows from the inventory is a short series of decisions, each refusing

one group of asks: where the wall goes, what identity the code runs as, what it

is allowed to spend, whether it has any route out, and what is within its reach

to be read. The inventory is what those decisions are answering. Anything left

off it is something nobody will think about again until it turns up in an

invoice or in somebody else's complaint.