Answers · Putting a model into service
How many legitimate requests should I be willing to block in order to catch the bad ones?
In shortBlock no more legitimate requests than one prevented harmful answer is worth: the break-even count is the harm cost divided by the refusal cost.
You should be willing to block as many legitimate requests as one prevented harmful answer is worth, measured in the cost of the two mistakes. Write h for the cost of one harmful answer getting out and c for the cost of one wrongly refused legitimate request; the break-even count is n = h/c. That is not a moral limit. It is the point where the refusal cost of the stricter setting equals the harm it prevents.
The arithmetic
Take h = $40,000 and c = $1. Then n = 40,000/1 = 40,000. One escaped harmful answer has to be worth 40,000 wrongly refused requests before a threshold that blocks those 40,000 is too strict. For a business product, take h = $200 for a mildly embarrassing answer and c = $80 because a refusal can cost a seat in an account. Then n = 200/80 = 2.5. Two wrongly refused requests cost $160 and are cheaper than one $200 harmful answer; three cost $240 and are not. The line is between two and three, so a defensible setting blocks at most two legitimate requests for every harmful answer it prevents.
| harm cost h | refusal cost c | break-even n = h/c | reading |
|---|---|---|---|
| $40,000 | $1 | 40,000 | one harmful answer is worth 40,000 refusals |
| $200 | $80 | 2.5 | three refusals cost $240, more than the $200 harm |
| $200 | $1 | 200 | one harmful answer is worth 200 refusals |
| $40,000 | $80 | 500 | one harmful answer is worth 500 refusals |
The decision rule is a comparison, not a slogan. If you refuse r legitimate requests in order to prevent one harmful answer, the refusal cost is r c and the benefit is h. Strictness pays only while r c < h, which is the same as r < h/c = n. At h = $200 and c = $80, n = 2.5, so r = 2 pays and r = 3 does not. At h = $40,000 and c = $1, n = 40,000, so r = 3,000 would still be far inside the strict region. The same arithmetic that makes a strict setting rational for one product makes it ruinous for another.
The obvious target is zero harmful answers, and the obvious way to get there is to keep tightening until nothing bad passes. That target treats h as infinite and c as zero. It also sees only half the evidence. A harmful answer that gets through produces an incident and a review. A wrongly refused legitimate request produces a polite decline, a user who concludes the product is broken or timid, and silence. The affected person tries once more, sometimes, and then goes elsewhere. Nothing arrives to push the threshold back, so every incident moves it stricter and the setting drifts past the point where the next refusal costs more than the harm it prevents.
Counting the hidden side
To use n you need c, and c comes from reading refusals rather than waiting for complaints. Take 300 refusals and count how many should have been answered. Suppose 36 should have been refused and 264 were legitimate. Then 36/300 = 12% were correctly refused and 264/300 = 88% landed on people who had done nothing wrong. The refusal cost for that sample is 264c. If the 36 correctly refused requests each prevented one harmful answer, the benefit is 36h. Strictness pays for this sample only if 264c < 36h, which rearranges to 264/36 < h/c, or about 7.3 < n. At h = $200 and c = $80, n = 2.5, and 7.3 > 2.5, so this sample is over-refusing. At h = $40,000 and c = $1, n = 40,000, and 7.3 < 40,000, so the same sample is not over-refusing.
Where it stops being true
The ratio is per category, not global. A clinical vocabulary request, a historical question, and a security administration question can have different h and c, so one n for the whole system will be wrong for most of it. The left end of a threshold is also not safe. It is the region where total cost is highest, made up almost entirely of refusals landing on people who did nothing wrong. And some over-refusal is not a threshold at all. If refused requests still refuse with the filters turned off, the model has learned to decline the surface vocabulary of a category rather than the category. No dial fixes that. You prompt around it, choose a different model, or live with it, and you know which by running the refused requests again with the filters off.
Worth keeping
- The break-even number of legitimate requests you may block per prevented harmful answer is the harm cost divided by the refusal cost.
- A threshold tuned only by reacting to incidents drifts stricter than the cost minimum, because over-refusals do not generate complaints.
- Measure the cost of a wrongly refused request by sampling and reading your own refusals; the affected users will not report it.
Where this goes next
This question is one step of What a Model Refuses, and Why, a course that works through its subject from the beginning. The page that teaches this part of it is The Price of Saying No, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- If someone fully jailbreaks the model, what actually stops it from deleting my database?Nothing in the model stops it. What stops it is your code: the delete tool either cannot name your database or cannot delete irreversibly.
- How do I write a safety rule I can actually test, instead of just saying 'be harmless'?Name the action, the cases it covers, the cases it deliberately leaves out and what happens when it fires; then check that two readers label the same twenty borderline requests the same way.
- How does a document the model retrieved end up giving it instructions?A retrieved document gives instructions because its text is pasted into the same prompt as your instructions, so the model cannot tell data from command.
- What's the difference between a model that refuses because of how it was trained and one that just gets blocked by a filter sitting in front of it?A trained refusal is a habit in the model's weights that an unusual framing can slip past; a filter is separate software that reads the text as data, so no wording reaches it.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.