Answers

The systems around the model

The machinery a model sits in: storage, speed, doing several things at once, and staying up.

1. What's the difference between a CDN cache and caching inside my own app?A CDN cache sits near the user and can only hold answers that are identical for everyone, while a cache inside your app can be personal but exists once per process.
2. What's the difference between LRU and LFU, and which one should I use?LRU drops the item unused for longest; LFU drops the item requested least often. Both guess at future use, and the cheap win is refusing one-off arrivals rather than counting requests.
3. Why can't a system just stay consistent and available when the network splits?During a split, a cut-off machine can only answer from the data it already holds, so it must either risk being wrong or refuse to answer.
4. How can I tell if my traffic is bursty or just high volume?Compare the variance of the gaps between arrivals with the square of their average gap: high volume alone does not make traffic bursty.
5. Why does average latency hide the slow requests users actually feel?Waiting times are lopsided, so the average sits between the common case and the bad case and describes neither, and it is the bad case that users keep meeting.
6. Should the gradients be summed or averaged across GPUs?Sum the gradients when each device already divides its loss by the global batch size; average them when each device uses its own local mean. Doing both makes the gradient too small by the number of devices.
7. Why can a higher hit rate still mean the cache is making things slower?Hit rate counts hits equally and ignores what each hit costs, so a cache with many cheap hits can save less than one with few expensive hits, and can even add more work than it removes.
8. Should the cache go in front of my app or between the app and the database?Put the shared cache between the app and the database by default, and push a copy in front of the app only for the part of a response that every reader gets identically.
9. Does a server being up mean my service is available?No. A server being up only says a check got an answer; availability is the share of user requests that succeeded, measured where users are.
10. Why does concurrent code fail only sometimes even when there is no syntax error?Concurrent code can fail only sometimes because a multi-step update can be interrupted between its read and its write, so two workers act on the same stale value and one update is lost without any error.
11. Why do retries make an overloaded service worse?Retries add extra requests precisely when a service is already too busy, so they turn a small overload into a much larger one and can keep the failure going by themselves.
12. How can I estimate wait time from the number of requests in the system?Divide the average number of requests inside the system by the rate at which they arrive, and you get the average time each request spends there.
13. Why does latency explode when utilization gets close to 100%?Waiting time grows as one over the capacity you have left, so halving the spare capacity doubles the wait and the last few percent of load dominate.
14. What does it actually mean to say a copy will be 'eventually' consistent?It means the copies will agree in the end, and that for some unspecified time a read may return an out of date value.
15. What's the difference between a nested loop join and a hash join?One looks each outer row up in the inner table, fast only when the outer side is tiny and indexed; the other builds a lookup table in memory and reads each side once, which suits two large tables.

The other subjects