Basis Lab

Answers · Putting a model into service

If someone fully jailbreaks the model, what actually stops it from deleting my database?

In shortNothing in the model stops it. What stops it is your code: the delete tool either cannot name your database or cannot delete irreversibly.

A successful jailbreak changes what the model will agree to do. It does not change which calls your code exposes, which arguments those calls accept, or which database credentials the process holds. If a tool exists with a signature like delete_database(database_id) and the runtime user has DROP, then a captured model can supply your database name and the deletion happens. Nothing in the model's training, refusal behaviour, or the jailbreak itself is the barrier at that moment. The barrier is the tool definition and the privilege behind it.

\[ \operatorname{calls}(T) = \{ T(a_1,\ldots,a_k) : a_i \in \operatorname{type}_i \} \]

The set of calls the signature allows

Write down the set of calls a tool can express. A jailbreak changes the model's policy over that set, not the set. Take delete_record(record_id: str) against a table with N rows. The id argument is free, so there are N possible row deletions. With N = 1,000,000, there are 1,000,000 expressible calls, and the worst one is among them. If the signature also takes table: str and database: str, the count multiplies across every table and database the credential can reach. A scoped version, delete_own_record(record_id), resolves the account and database from the session and has no table or database parameter. Outside that session account the number of expressible deletions is 0; inside it the number is the number of rows that account owns, not the whole database. A database drop is a separate capability. If no tool signature includes a schema operation and the database user lacks DROP, the number of expressible DROP calls is 0, whatever the model says.

FIG 1Two delete tools, two worst cases
python
def delete_record(record_id: str):
    db.execute('DELETE FROM records WHERE id = ?', (record_id,))

def soft_delete_own_record(record_id: str):
    account_id = session.account_id
    db.execute(
        'UPDATE records SET deleted_at = now() '
        'WHERE id = ? AND account_id = ?',
        (record_id, account_id),
    )

def request_database_drop(database: str, reason: str):
    queue.put(DropRequest(database, reason))  # separate credential approves
The first signature can name any row the database credential can reach. The second cannot name another account and cannot hard delete. The third can only enqueue a request for a separate credential and a person to approve.

A jailbreak is not a privilege escalation. The model process already had whatever credentials the tool code gave it; the jailbreak only changes the instructions the model is following. That is why the useful audit is not how strong the refusal layer is, but what the worst call this signature can express, and what that call does to the database. If the answer is drop the production schema with no restore, the defence is missing at the tool boundary, not at the text boundary. If the answer is soft delete one row in the caller's own account, the worst case is boring, and a jailbreak is an incident report rather than a data loss.

The natural guess is that a system prompt, refusal training, or a jailbreak detector is what stands between the model and your data. Those change the probability that the model proposes the call, not the effect if it does. A detector is a check, and a check has two error rates. If it catches 99 of every 100 attacks, then 100 - 99 = 1 attack in 100 still gets through; one is enough when the action is DROP DATABASE. The same detector also refuses some legitimate requests, and that is the other error rate. A tool with no delete parameter has no false-negative rate for deletion, because there is no deletion for it to miss. A broad tool with a validator in front is better than a broad tool with nothing, but it is still a check that can be wrong. The narrow tool cannot be talked past, because the argument it would need does not exist.

Where the guarantee stops

This does not make a captured model harmless. A scoped delete can still delete every row inside its scope, so scope the credential and the session as tightly as the product allows. Recoverability is a property of storage, not of the prompt: a soft delete is only recoverable if retention exists and a restore has been tested. If retention is 30 days and a restore takes 2 days, the worst case is 30 + 2 = 32 days of missing data, not zero. Backups bound the loss; they do not stop the deletion. A human confirmation helps only when it shows the literal call your code built, one action at a time. If the confirmation text is a summary written by the model, the user is confirming the model's honesty, which is the thing under attack. If you must keep a hard delete, put it behind a separate credential the request process does not hold and a queue that a person approves; then a captured model can at most enqueue a request. A raw SQL tool is the broadest delete tool there is, with an argument for every statement the credential can run. Remove it, or accept that the jailbreak has the same power as the credential.

Worth keeping

  • A jailbreak changes what the model will agree to do, not which calls your code can express.
  • A narrow tool has no false-negative rate on the action it cannot express.
  • Backups bound recovery time; they do not stop a deletion.

Where this goes next

This question is one step of What a Model Refuses, and Why, a course that works through its subject from the beginning. The page that teaches this part of it is What the Model Is Allowed to Touch, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. How many legitimate requests should I be willing to block in order to catch the bad ones?Block no more legitimate requests than one prevented harmful answer is worth: the break-even count is the harm cost divided by the refusal cost.
  2. How do I write a safety rule I can actually test, instead of just saying 'be harmless'?Name the action, the cases it covers, the cases it deliberately leaves out and what happens when it fires; then check that two readers label the same twenty borderline requests the same way.
  3. How does a document the model retrieved end up giving it instructions?A retrieved document gives instructions because its text is pasted into the same prompt as your instructions, so the model cannot tell data from command.
  4. What's the difference between a model that refuses because of how it was trained and one that just gets blocked by a filter sitting in front of it?A trained refusal is a habit in the model's weights that an unusual framing can slip past; a filter is separate software that reads the text as data, so no wording reaches it.
  5. Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
  6. Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.

Read alongside