Basis Lab

Answers · The systems around the model

Should the cache go in front of my app or between the app and the database?

In shortPut the shared cache between the app and the database by default, and push a copy in front of the app only for the part of a response that every reader gets identically.

Put the shared cache between the application and the database, and treat that as the default. A cache in front of the application earns its place only for the portion of a response that every reader receives identically; anything assembled per person stays behind the application, where the code can see who is asking.

Start with what a hit costs at each rung, because that is the argument usually made for keeping the copy as close to the code as possible. A map inside the service process answers in the time it takes to follow a pointer, 0.0002 milliseconds. The shared tier beside the store is one round trip away, 0.4 milliseconds. Dividing one by the other:

0.4 ms0.0002 ms=2000\frac{0.4\ \text{ms}}{0.0002\ \text{ms}} = 2000

Two thousand times faster is real, and on its own it is the wrong number to optimise. The question that decides placement is how many copies you have to keep honest. One process with a local map is one copy. Forty processes are forty copies, forty expiry schedules, and forty places an invalidation has to reach, none of which knows the others exist. A single popular item that goes cold produces forty simultaneous misses and forty fetches of the same row, where one shared tier would have produced one fetch and served the other thirty nine from it.

What the outer copy cannot know

Move the copy the other way, in front of the application instead of behind it, and the gain and the loss both change. A layer near the reader removes geography, and geography is most of what a distant reader waits for: in one traced request the nearby point of presence is reached in 18 milliseconds, the crossing to the data centre costs 112 milliseconds, so 94 of those milliseconds are distance and no work inside the data centre touches them. What the outer layer cannot do is see who is asking. It answers before your code runs, so it has no user, no session and no permissions row. A page assembled per person therefore cannot be held there whole: key it by address and one reader is served another reader's page, key it by user and each entry is wanted by exactly one person, which is a cache with a hit rate near zero and a privacy problem attached. The remedy is to split the response, with the part everybody shares going out to the near layer at 18 milliseconds and the small personal remainder coming back for the 112 millisecond crossing.

The arithmetic that makes an outer layer worth its risk is the nesting of the two costs. Let the near layer answer a share pp of reads at ℓ\ell milliseconds, let the shared tier answer a share qq of what gets past it at mm milliseconds, and let the store cost ss. Then

L=p ℓ+(1−p)(q m+(1−q) s)L = p\,\ell + (1-p)\left(q\,m + (1-q)\,s\right)

With p=0.7p = 0.7, ℓ=18\ell = 18, q=0.9q = 0.9, m=112.4m = 112.4 and s=130s = 130, the second bracket is 0.9×112.4+0.1×130=101.16+13=114.160.9 \times 112.4 + 0.1 \times 130 = 101.16 + 13 = 114.16 milliseconds, and the whole expression is 0.7×18+0.3×114.16=12.6+34.25=46.850.7 \times 18 + 0.3 \times 114.16 = 12.6 + 34.25 = 46.85 milliseconds. Without the near layer the same traffic averages the full 114.16. The outer copy wins not by being fast in itself but by multiplying everything underneath it by the thirty percent that got past, which is why it belongs in front of the part of the page that is shared rather than in front of the part that is personal.

The in-process map has a matching failure at the other end, and it is arithmetic rather than opinion. With only a shared tier, a share of 0.08 of reads falls through to it. Add a local map to every process and the share that reaches the tier becomes 0.08+0.30(1−e−n/16)0.08 + 0.30\left(1 - e^{-n/16}\right) for nn processes: at one process that is 0.08+0.30×0.0606=0.0980.08 + 0.30 \times 0.0606 = 0.098, and at sixty four processes it is 0.08+0.30×0.9817=0.3730.08 + 0.30 \times 0.9817 = 0.373. The local layer barely changes the picture when there is one process and makes it nearly five times worse for the tier below when there are sixty four, because each process now sees a sixty fourth of the traffic, warms slowly, and duplicates the misses of its neighbours. Forty copies also disagree with each other, so two readers asking the same question land on different processes and get answers of different ages.

Where the ordering stops mattering

Two conditions flip the advice. If the service runs as a single process, the local map and the shared tier collapse into the same decision: one copy, one expiry schedule, and the objection to keeping it in the process disappears. If the answer is wanted by one person and must be current, neither rung holds it and it should not be cached at all. Between those extremes the test is two questions asked of each field in the response: who else wants this answer, and how old may it be? Something every reader wants that may be minutes stale goes as far out as it can reach. Something one reader wants that must be seconds fresh stays close in, or goes uncached.

The shared tier beside the store costs 0.4 milliseconds against the local map's 0.0002 and repays it with one warm copy for the whole fleet, one invalidation, and capacity you provision rather than capacity that happens to be spare. That is why it goes between the application and the database. The local map is then an optimisation applied to the few entries whose round trip you cannot afford, and the layer near the reader is an optimisation applied to the shared shell of the page, not to the page.

Worth keeping

  • A copy inside each process is two thousand times faster to read than a shared tier and comes in as many disagreeing copies as you have processes.
  • A cache in front of the application answers before your code runs, so it cannot see the user and can only hold what every reader receives identically.
  • Splitting a page into a shared shell held near the reader and a small personal remainder fetched from further in is what turns a slow response into a fast one.

Where this goes next

This question is one step of Keeping the Answer Nearby, a course that works through its subject from the beginning. The page that teaches this part of it is Where to Put It, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. What's the difference between a CDN cache and caching inside my own app?A CDN cache sits near the user and can only hold answers that are identical for everyone, while a cache inside your app can be personal but exists once per process.
  2. What's the difference between LRU and LFU, and which one should I use?LRU drops the item unused for longest; LFU drops the item requested least often. Both guess at future use, and the cheap win is refusing one-off arrivals rather than counting requests.
  3. Why can a higher hit rate still mean the cache is making things slower?Hit rate counts hits equally and ignores what each hit costs, so a cache with many cheap hits can save less than one with few expensive hits, and can even add more work than it removes.
  4. Why can't a system just stay consistent and available when the network splits?During a split, a cut-off machine can only answer from the data it already holds, so it must either risk being wrong or refuse to answer.
  5. How can I tell if my traffic is bursty or just high volume?Compare the variance of the gaps between arrivals with the square of their average gap: high volume alone does not make traffic bursty.
  6. Why does average latency hide the slow requests users actually feel?Waiting times are lopsided, so the average sits between the common case and the bad case and describes neither, and it is the bad case that users keep meeting.

Read alongside