Answers · The network and the web
Why does a DNS lookup sometimes have to ask several servers before it gets an answer?
In shortBecause when no cache already holds the address, the resolver has to ask one server at a time down the hierarchy, and each reply only tells it who to ask next.
It asks several servers only when nothing closer to hand already knows the answer, and then it has to ask them one after another. Each reply names the next server to ask, so the queries cannot be sent in parallel and their costs add rather than overlap.
Nearly every lookup stops before that. The browser keeps answers, the operating system keeps answers, and the resolver your network provides keeps answers for everybody it serves. The hierarchy is what happens when all of those are empty, which is rare: over nine in ten lookups are answered without a full walk, and about three quarters never leave the machine at all.
The walk, and why it is three steps
The resolver begins with a short list of root servers built into it. A root server will not know the address and never will; it replies with a referral, the addresses of the servers for the top level domain. Ask one of those and you get a second referral, for the domain's own servers. Ask one of those and you finally get the address. Three exchanges, three different machines, usually on three different networks.
The addition is not an accident of implementation. The second query cannot even be written until the first reply arrives, because the first reply is what supplies the address it must be sent to. A slow first server delays the second by exactly as much. That is the difference between a lookup and a page load: a page's images download concurrently, the steps of a cold lookup cannot.
| step | lookup | answered by | cost in milliseconds | running total | what happened |
|---|---|---|---|---|---|
| 1 | the site itself | full walk | 180 | 180 | Nothing cached anywhere. Three exchanges plus the resolver's own overhead. |
| 2 | the image host | resolver cache | 22 | 202 | A different domain, but this resolver had it recently from another user. |
| 3 | a second name in the site domain | partial walk | 60 | 262 | The referrals from the first lookup were cached, so only the last exchange was needed. |
| 4 | the analytics host | full walk, one server slow | 1100 | 1362 | One server did not answer and the resolver waited out its timeout before trying another. |
That last row is the ordinary way a third party makes your page slow. The four costs add as 180 + 22 + 60 + 1100 = 1362 milliseconds, and every one of them is spent before the request to the analytics host exists, so no amount of work on your own server touches it.
The natural guess is that this walk happens on every lookup, which would make name resolution obviously expensive and the first thing to fix. The walk is rare. What separates a good day from a bad one is not how often it happens but what it costs when it does, and the arithmetic settles it. Take a hit rate of 0.93, so 7 per cent of lookups miss. A miss costing 200 milliseconds gives an average of milliseconds. The same hit rate, with a miss costing 5 seconds because one server did not answer and the resolver waited out its timeout, gives milliseconds. No cache got worse between those two cases; only the far end of the chain did.
At 14 milliseconds nobody notices the lookup. At 350 milliseconds the same code path has added a third of a second to every page load, and it leaves no trace in the page and no entry in your own server's logs, because the request has not been made yet.
Lifetimes, and the cost of a long one
Every answer is handed over with a lifetime in seconds, and every cache along the chain obeys it and cannot be told to forget. A record kept for 300 seconds is fetched at most once every five minutes per cache; the same record kept for 86400 seconds is fetched times less often. That saving sounds large and is not, because repeat lookups cluster: the second visit to a name usually happens within minutes, so a five-minute lifetime already captures most of the benefit a day-long one would. What does not flatten out is the other side. A machine that has failed is still being handed out by every cache that holds it, for up to the full 86400 seconds, and during an incident that number is what everybody is waiting on.
Two things move these numbers. A resolver that has just started or been flushed pays the full 180 milliseconds, or more, for every name until it has learned them, which is why the first load after a reboot feels slow and the tenth does not. And a page that spreads itself over many distinct hosts multiplies the chances to pay the miss cost, since each host is a separate name that can be cold, separate from the others and discovered only once the first response has been parsed.
Worth keeping
- A cold lookup costs three sequential round trips, roughly 180 milliseconds at 60 milliseconds per exchange, because each reply supplies the address of the next server to ask.
- At a 0.93 hit rate a 200 millisecond miss averages 14 milliseconds, but a 5 second miss caused by a timeout averages 350 milliseconds, so the tail and not the hit rate decides whether lookups are felt.
- A record cached for 86400 seconds is fetched 288 times less often than one cached for 300 seconds, and cannot be revoked, so the long lifetime trades a small traffic saving for up to a day of stale answers during a failure.
Where this goes next
This question is one step of From an Address to a Page, a course that works through its subject from the beginning. The page that teaches this part of it is Turning a Name Into a Number, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does opening a connection take three messages instead of two?Three messages are needed because the server must also learn that its own reply arrived; the third message is that proof, not a formality.
- What does a CDN actually change about my request?A CDN puts a machine near the reader that answers the request itself, so your origin often never sees it and the address it logs is not the reader's.
- Why does a download start slow and then speed up?The sender starts with about fourteen kilobytes in flight and doubles that each round trip until something is lost, so the opening round trips measure the path rather than move data.
- Where should a session ID be stored so JavaScript can't read it?Store it in an HttpOnly cookie set by the server; that is the only browser storage JavaScript running on the page cannot read.
- What does it mean for an API method to be safe?A safe method is one that changes nothing on the server when it is called, so a caller may repeat it freely and a cache or a crawler may use it without anything being altered.
- Does signing out actually invalidate my token on the server?No. Signing out clears your browser's copy of the token, but the server keeps accepting that token until it expires.