Basis Lab

Answers · Putting a model into service

Why does promising that 99% of requests are fast cost so much more capacity than promising the average is fast?

In shortA promise about the slowest requests forces you to keep machines idle so queues never build, while an average promise can be met by making ordinary requests slightly faster.

Because the two promises constrain different requests, and only one of them can be satisfied by speeding up the bulk of traffic. An average is held down by improving the many ordinary requests, which is cheap because there are so many of them; a ninety-ninth percentile is held down only by making the slowest request in a hundred rare, and slow requests are made by queues. Preventing queues means paying for machines that sit idle.

Take a service where 98 requests in a hundred complete in 100 ms and the remaining two take 2 seconds because they land on a path that waits for something. The average is (98 x 0.1 + 2 x 2.0) / 100 = 13.8 / 100 = 138 ms. The ninety-ninth percentile is 2 seconds. A promise of an average under 150 ms is kept with 12 ms to spare; a promise that 99% finish under 150 ms is missed by 1,850 ms. Nothing about the service changed between those two sentences.

The average is a sum, so it can be traded. One request that takes 1.9 seconds longer than it should adds 1.9 seconds to the total for the window. Spread over a thousand requests that is 1.9 ms on the mean, and it can be bought back exactly by shaving 1.9 ms off each of the other 999. Every slow request has a price in fast requests and the exchange rate is fixed, so a team working to an average can let the occasional disaster through and pay for it elsewhere.

A percentile has no such rate. The ninety-ninth percentile is an order statistic: it is the duration of the request sitting at position 990 when a thousand are sorted by duration. To move it from 2 seconds to 150 ms you cannot improve a handful of slow requests, because there are only twenty of them and improving nineteen leaves the percentile exactly where it was. You have to move essentially all of them, which means removing the cause of the slow path rather than averaging it away.

The natural guess is that 99% under 150 ms is the same promise as an average under 150 ms with a stricter number attached, so the same design with a little more headroom will hold it. Headroom at the average rate does nothing here. It makes the ordinary requests faster, and the ordinary requests were never the problem; the slow ones are waiting behind each other, and that wait grows much faster than the load does.

The queue sets the tail

Model the service as one queue with mean service time SS and utilisation ρ\rho, the fraction of time the server is busy. The mean time from arrival to completion is S/(1−ρ)S/(1-\rho). The same model gives the whole distribution: the chance a request takes longer than tt is ρe−t(1−ρ)/S\rho e^{-t(1-\rho)/S}, which falls away exponentially but at a rate that collapses as ρ\rho approaches one.

E[T]=S1−ρ,t99=Sln⁡(100ρ)1−ρ\mathbb{E}[T] = \frac{S}{1-\rho}, \qquad t_{99} = \frac{S \ln(100\rho)}{1-\rho}

Set S=10S = 10 ms and promise 50 ms. For the mean, 50=10/(1−ρ)50 = 10/(1-\rho) gives ρ=0.8\rho = 0.8, so a server completes 0.8/0.010=800.8/0.010 = 80 requests a second. For the ninety-ninth percentile, 50=10ln⁡(100ρ)/(1−ρ)50 = 10\ln(100\rho)/(1-\rho) has to be solved by trial: ρ=0.312\rho = 0.312 gives 10×3.440/0.688=50.010 \times 3.440 / 0.688 = 50.0 ms, and the same server now carries 0.312/0.010=31.20.312/0.010 = 31.2 requests a second.

FIG 1
promiseequation to solveutilisationrequests per second
mean under 50 ms50 = 10/(1 - rho)0.8080.0
99% under 50 ms50 = 10 ln(100 rho)/(1 - rho)0.31231.2

At 31.2 requests a second the machine is busy 31% of the time, so roughly two thirds of what you paid for is idle. That idleness is not waste in the ordinary sense; it is what the promise is made of, because every request that would otherwise have waited occupies that spare room instead. On the mean promise at ρ=0.8\rho = 0.8 only a fifth of the machine is spare, and the queue sometimes grows into it.

FIG 2Where each promise runs out of room
0.00250.00500.00750.001000.000.10.30.50.70.9utilisation, rho
mean99th percentile
Both curves are the same machine with a 10 ms service time. A 50 ms promise is crossed by the mean at utilisation 0.8 and by the ninety-ninth percentile at 0.312; the vertical distance between the curves is the capacity the stricter promise costs.

Where the multiple changes

The size of the penalty depends on how much room the promise leaves above the raw work. Loosen the same promise to 200 ms with the same 10 ms service time and the mean allows ρ=0.95\rho = 0.95, or 95 requests a second, while the ninety-ninth percentile allows ρ=0.782\rho = 0.782, or 78.2 a second: 21% more machines rather than 156%. Tighten it and the cost climbs steeply. At 99.9% inside 50 ms, 5(1−ρ)=ln⁡(1000ρ)5(1-\rho) = \ln(1000\rho) is satisfied by ρ=0.093\rho = 0.093, which is 9.3 requests a second, eight and a half times the capacity of the mean promise and more than three times the 99% one.

Two further things move these numbers. Service times heavier tailed than the exponential make the real ninety-ninth percentile worse than the formula, so the utilisation that holds it sits below 0.312. And a promise measured at the client spends part of its 50 ms on the network and the connection, leaving less inside the service, which pushes utilisation down again.

The permitted one percent is what keeps the price finite. Over a million requests in a week, a 99% promise allows 10,000 of them to exceed the threshold: it is a licence to be slow ten thousand times, and the spare capacity is what buys the choice of which ten thousand. Promise 99.9% and the licence shrinks to a thousand, and the utilisation falls from 0.312 to 0.093 to pay for it. Each added nine multiplies the idle machines that have to exist, which makes the number in the promise a purchasing decision before it is an engineering one.

Worth keeping

  • A mean latency can be held down by improving ordinary requests, because an average is a sum and a slow request can be paid for elsewhere; a ninety-ninth percentile cannot, because it names the slow requests themselves.
  • With a 10 ms service time and a 50 ms promise, meeting the mean allows 80 requests a second per server, and meeting the ninety-ninth percentile allows 31.2.

Where this goes next

This question is one step of Serving a Model to Many People, a course that works through its subject from the beginning. The page that teaches this part of it is Promising a Number and Keeping It, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. How do I tell whether a slow request spent its time waiting in line or actually being worked on?Time a request spent waiting is the total measured where it arrived minus the time anything was actually computing for it; whatever is left was queue.
  2. Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
  3. Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
  4. How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.
  5. Why does the memory used by a model keep growing as the conversation gets longer?The model's weights do not grow. It keeps a cache of keys and values for every earlier token, so memory rises with conversation length and with how many conversations run at once.
  6. How many legitimate requests should I be willing to block in order to catch the bad ones?Block no more legitimate requests than one prevented harmful answer is worth: the break-even count is the harm cost divided by the refusal cost.

Read alongside