How do I decide whether a small but significant win is actually worth shipping?

The natural guess is that crossing the significance line is the ship signal, and it fails because the line answers a different question. It reports how surprising the data would be if the change did nothing. It says nothing about the relative size of the two errors you can make. A fixed one in twenty is right for exactly one ratio of costs, the one where the loss is about nineteen times the gain, and that ratio is not common in either direction. So the convention ships duds on cheap-to-reverse products, where it should be far more permissive, and it discards winners on expensive ones, where it should be far stricter.

The probability you compare against $L/(G+L)$ is not the false positive rate either. It is your honest chance that this particular change works, given the interval and the base rate. The interval gives you the first part: a gain of +0.9 points with a 95 per cent interval of +0.1 to +1.7 says the effect is probably positive and probably smaller than the point estimate suggests. The base rate gives you the second, and it is the step people skip. If one change in five works and your test detects real effects eighty per cent of the time, then out of a hundred experiments twenty are real and sixteen of those get reported; of the eighty duds, four clear the line by chance. That is twenty reported wins of which sixteen are real, so eighty per cent. Drop the base rate to one in twenty and the same arithmetic gives five real changes, four detected, and about five false wins from ninety five duds, so four in nine, or forty five per cent. Most of the celebrated results in that organisation are noise, and nothing in any single result reveals which.

The ratio assumes the costs are known, roughly stable and additive, and it assumes the gain persists. If a win decays over a few months, $G$ is an overstatement and the bar rises; if the change compounds with later ones, $G$ is an understatement and the bar falls. For hard-to-reverse changes $L$ has a tail, since the cost of a bad refund system is not the expected cost but the worst plausible one, and you should price the tail rather than the average. And for a small but significant win whose interval sits entirely below the threshold you wrote down before the run, the answer is neither ship nor discard: it is that the change is real and too small to be worth the complexity, which is a settled question and worth recording as one.

One consequence of the arithmetic is that most of the value in a programme of experiments comes from a small number of large wins against a background of failures. If a third to a tenth of changes produce the improvement their authors expected, then a team that ships only on the conventional threshold is discarding cheap winners and keeping expensive losers, and is doing so systematically, because the threshold is the same for both. Judge the programme on the duds it stopped rather than on its hit rate, and write down the interval, the priced mistakes and the decision together, including for the experiments that found nothing. A null result with a narrow interval closes a question. A result too small to separate from zero closes nothing, and saying so is more useful than a verdict.