The New Version Is Eight Per Cent Faster, and So Is the Old One If You Run It Enough Times
Last timeFour Ways a Profile Misleads
A single before and after pair proves nothing, because the same program run twice differs by more than most improvements. Here is the comparison that survives that, and how to report it.
The same program, twice
Before any comparison, measure the thing that decides whether a comparison is
possible: run the unchanged program ten times and look at the spread.
Most people are surprised. The numbers are not the same, and they are often not
close. Several per cent between the fastest and slowest run of identical code
on an idle machine is ordinary, and on a shared or virtualised machine it can
be far worse.
| typical swing, per cent | can be reduced | can be eliminated | shows up in a single pai | |
|---|---|---|---|---|
| where the code and data | 4 | 1 | 0 | 1 |
| cache and branch predict | 3 | 1 | 0 | 1 |
| other processes on the m | 2 | 0 | 1 | 1 |
| processor frequency and | 8 | 0 | 0 | 0 |
| which machine in the fle | 3 | 0 | 0 | 1 |
| the order the runs happe | 1 | 1 | 0 | 1 |
The memory layout figure deserves a moment, because it is the one that feels
wrong. Where functions and data happen to land determines which of them share
cache sets and which share pages, and that can move run time by several per
cent with the instructions entirely unchanged. Address randomisation guarantees
it differs between runs. This means a single measurement of a single build is
partly a measurement of one accidental layout.
The consequence is the whole lesson. If the same program varies by four per
cent, a single before and after pair showing a five per cent improvement is
almost no evidence at all.
Running them alternately
The second problem is worse than noise, because averaging does not fix it.
Suppose you run the old version twenty times, then the new version twenty
times. During those forty runs the machine warmed up, a background update
started, and the processor dropped a frequency step. All of that lands on the
second half, which is entirely the new version. The comparison now contains a
real difference between the versions plus an unknown difference between the
first and second halves of the afternoon, and nothing in the numbers separates
them.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents