My change made it 4% faster — how do I know that isn't just noise?

You cannot, from one pair of runs. What you have is the difference between two single measurements, and until the run to run spread of the unchanged program is known, a four per cent gap has nothing to be compared against.

The measurement that decides it costs ten minutes. Take the version you did not change, run it ten times back to back on the same machine, and look at the spread of those ten numbers before you look at anything else.

The ten times sum to 1006.0 ms, so the mean is 100.60 ms. The deviations in the third column square to 0.36, 6.76, 3.61, 0.81, 13.69, 12.25, 1.69, 4.84, 9.00 and 1.69, which total 54.70. Divide by nine, since ten runs determine one mean, and the variance is 6.08 ms²; take the root and the standard deviation is 2.47 ms. Against a mean of 100.60 ms that is 2.5 per cent. The fastest run, 96.9 ms, sits 7.2 per cent below the slowest, 104.1 ms, on code that did not change by a single instruction.

The natural guess is that four per cent is comfortably bigger than the odd wobble, so it must be real. It is not bigger than what is actually on the machine. Four per cent of 100.60 ms is 4.0 ms, and the fifth unchanged run, at 96.9 ms, already landed within 1 per cent of a 96.0 ms build. A single pair is one draw from each of two heavily overlapping distributions, and nothing in the pair says which part of the overlap you drew. The trap runs the other way too: a genuinely six per cent faster build can measure two per cent on an unlucky pair and get thrown away.

Put the numbers through it. With ten runs of each version the uncertainty on the difference is 2.45 × √(2/10) = 2.45 × 0.447 = 1.1 percentage points. Doubling that for a 95 per cent interval gives ±2.2, so the result to report is 4.0 ± 2.2 per cent, somewhere between 1.8 and 6.2. Zero is outside that interval, so on this machine ten interleaved runs of each settle a four per cent question, and no more than that.

The square root is the whole economics. Halving the interval to ±1.1 takes four times the runs, forty of each. And if the same machine had a ten per cent spread, ten runs of each would give 2 × 10 × 0.447 = ±8.9, wider than the effect itself; the count that would settle it comes from 2 × 10 × √(2/n) < 4, so n > 50, and the honest answer is fifty-one runs of each.

This second failure is not noise and averaging will not touch it. Run the old version twenty times, then the new version twenty times, and the machine changes underneath you: it warms up, a background job starts, the processor drops a frequency step. Suppose the drift costs 3 per cent across the session and the change is really worth 4. In the old-then-new order the second half carries the 3 points, so the new version measures 1 per cent faster. In the new-then-old order the drift penalises the old version and the same change measures 7 per cent faster. Same forty runs, same machine, opposite headlines.

Alternating turns the drift from a bias into a nuisance shared by both versions, because consecutive runs happen under nearly identical conditions; comparing inside each adjacent pair removes most of what is left, which is why pairing buys precision for nothing. Discard the first run or two of each version as well, since a cold start measures the file system and the allocator rather than the program.

All of that assumed an idle machine and one workload. On a shared or virtualised host the spread is routinely 8 to 10 per cent, and the run counts above grow to fifty or more per version; the useful thing is to work that out before spending the afternoon rather than after. The result is also tied to its input, because a four per cent win on one workload is not a win on another, and tied to its processor, because the same change can help on one and do nothing on the next. Address randomisation guarantees that memory layout differs between runs, so no amount of care makes a single measurement of a single build reproducible. Only the distribution is reproducible.

Write it so it cannot be misquoted: the difference with its interval rather than as a bare percentage, the number of runs behind it, the workload, and the machine. Four per cent faster on its own is a number that will be quoted into a context where it does not hold.

When the interval does cover zero, do not report the midpoint. Report that the change was too small to measure with the runs taken, and how many more runs would settle it. An improvement whose interval runs from below zero to well above the measured difference is not a small win; it is no result yet, and saying so is the finding.