You Asked for Four Bytes and Sixty-Four Arrived
Last timeHow Far Away Memory Is
Memory is not moved a byte at a time. It moves in blocks of sixty-four, and whether you use the other sixty is the difference between fast code and slow code.
The block is the unit
Read a single byte from memory. The hardware fetches sixty-four, aligned to a
sixty-four byte boundary, and puts all of them in the cache.
This is not a rounding-up convenience. It is the only operation available. The
path from memory to processor moves blocks, the caches store blocks, and the
protocol that keeps several cores consistent talks about blocks. There is no
transaction anywhere in the machine that moves four bytes.
The consequence is a change of question. Asking how much data your program
reads is close to meaningless. The question that predicts performance is how
many distinct blocks your program touches, and of the sixty-four bytes in each
one, how many it looked at.
Why it fetches extra
The block size looks wasteful until you price the alternative, and the pricing
turns on where the cost of a memory access actually sits.
Almost all of it is getting there. The request travels to the memory
controller, the controller selects a chip and a bank, a row of several
thousand bits is activated into a buffer, and only then are bytes read out.
Activating the row is the expensive part and it is already done. Reading
sixty-four bytes out of the open row instead of four costs a few additional
nanoseconds against the eighty already spent.
| step | stage | nanoseconds | bytes delivered so far | what happened |
|---|---|---|---|---|
| 1 | request leaves the processor | 15 | 0 | Travel. This part is geometry and is paid whatever you asked for. |
| 2 | row activated in the chip | 45 | 0 | The expensive internal step. Thousands of bits are sensed into a buffer, and nothing has been sent back yet. |
| 3 | first eight bytes returned | 65 | 8 | The first bytes arrive. Note that seventy-five per cent of the total time has already been spent before a single byte came back. |
| 4 | remaining fifty-six returned | 80 | 64 | Fifteen nanoseconds for eight times as much data. This is why the block is sixty-four bytes and not eight. |
That is the whole argument. The hardware is betting that if you touched a
byte, you will shortly touch its neighbours. The stake is a few nanoseconds
and some cache space. The payoff is eighty nanoseconds avoided. Programs
written without a thought for any of this still win the bet most of the time,
because arrays, structures and stack frames are all contiguous by nature.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents