Answers · How a program runs

How many samples does a profiler need before a percentage in the report means anything?

In shortIt depends on how many samples landed on that line: you need a few hundred samples on the line you care about, because the relative error is roughly one over the square root of that count.

A percentage in a sampling profile is a count divided by the total number of samples, so its uncertainty is set by how many samples landed on that line, not by the total alone. A line needs a few hundred samples before its share is worth quoting to two significant figures: at a hundred samples the relative error is about ten per cent, at nine it is about a third, and at two it is about seventy per cent.

e = \sqrt{\frac{p(1-p)}{n}}, \qquad \frac{e}{p} \approx \frac{1}{\sqrt{k}}

One run, line by line

Take a run of 3000 samples. A function reported at 35 per cent received 1050 samples. Its standard error is $\sqrt{0.35 \times 0.65 / 3000} = \sqrt{0.0000758} = 0.0087$ , or 0.87 percentage points. That is a relative error of $0.0087/0.35 = 0.025$ , about 2.5 per cent, so the true share is near 35 per cent and the printed figure can be trusted. The same arithmetic at the bottom of the table gives a different answer.

FIG 1

function	samples on line	share	standard error, points	relative error
parser	1050	35.0 per cent	0.87	2.5 per cent
hash lookup	400	13.3 per cent	0.62	4.7 per cent
retry path	60	2.0 per cent	0.26	13 per cent
logger	9	0.3 per cent	0.10	33 per cent
configuration read	2	0.07 per cent	0.05	71 per cent

The first two rows are fine. The last two are not. Nine samples give a relative error of $1/\sqrt{9} = 1/3 = 33$ per cent, so the logger's 0.3 per cent could as easily be 0.2 or 0.4. Two samples give $1/\sqrt{2} = 0.71$ , so the configuration read at 0.07 per cent could be 0.02 or 0.12. The absolute error column makes these look safe because it is small in points; the relative error is the one that matters when the share is small.

The gap you are arguing about

If two entries print as 1.1 and 0.9 per cent in a 3000 sample run, they received about 33 and 27 samples. The relative errors are $1/\sqrt{33} = 0.17$ and $1/\sqrt{27} = 0.19$ , so the first is $1.1 \pm 0.19$ per cent and the second $0.9 \pm 0.17$ per cent. Those one standard error intervals overlap from 0.91 to 1.07, so the order is an accident of this run. The same applies to a change from 13.3 to 13.0: each line at that share carries a standard error near 0.6 points, and the difference of two such estimates has error $\sqrt{0.62^2 + 0.62^2} = 0.88$ points, which is three times the change.

How long to run

Decide the smallest share you need to resolve, then give it a few hundred samples. For a one per cent line resolved to ten per cent relative error, you need 100 samples on that line, so $n = 100 / 0.01 = 10{,}000$ samples in total. At 1000 samples per second that is ten seconds. For a 0.1 per cent line at the same relative error, $n = 100 / 0.001 = 100{,}000$ samples, or 100 seconds. For five per cent relative error, which is enough to claim a change, you need 400 samples on the line: a one per cent line then needs 40,000 total samples, or forty seconds.

Beyond some run length more samples stop helping. The samples are not independent: the workload changes, caches warm and cool, scheduling varies. The error computed here is a lower bound. If the profiler misses inlined functions or attributes a sample to the wrong frame, the true error is wider by an amount no formula gives. At that point the fix is several separate runs and a comparison between them, not a longer single run.

Worth keeping

A sampling profile's percentage is an estimate from a finite count, so the uncertainty on a line depends on that line's sample count.
The relative error on a line is roughly one over the square root of its samples: a hundred samples gives ten per cent, nine gives a third.
To resolve a share p to relative error r, run until the line has about 1/r^2 samples, which means total samples about 1/(r^2 p).

Where this goes next

This question is one step of Reading a Profile Honestly, a course that works through its subject from the beginning. The page that teaches this part of it is How Many Samples Is Enough, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course