The Sample Landed Three Instructions After the One That Was Slow, and the Function It Blames Does Not Exist Any More
Last timeThe Time It Was Not Running
Even a correctly taken profile of the right thing puts time on the wrong line in four specific ways. All four are recognisable, and knowing them changes what you trust.
The sample lands late
The first distortion is in the mechanism itself. When a sampling interrupt
fires, the processor does not stop instantly. It finishes what is in flight,
takes the interrupt, and the handler reads the instruction pointer. By then the
program has moved on.
| step | moment | what the processor was doing | attributed to | error | what happened |
|---|---|---|---|---|---|
| 1 | 1 | a load that misses all the caches | nothing yet | none | This is the expensive instruction. It will stall for a few hundred cycles. It is the thing a reader of the profile wants to find. |
| 2 | 2 | stalled, waiting for memory | nothing yet | none | Hundreds of cycles pass here. If the interrupt fires now, the recorded pointer is somewhere after the load, because the hardware reports the next instruction not yet retired rather than the one stalling. |
| 3 | 3 | the interrupt is taken and handled | the instruction after the load | a few instructions | On straightforward hardware this is a small error and the sample lands within the same statement. The line attribution is wrong and the function attribution is right. |
| 4 | 4 | an out-of-order processor with many inst | tens of instructions later | can cross a function boundary | Here the error becomes qualitative. The sample can be attributed past the end of a short function to the caller, or past a branch into code that had nothing to do with the cost. |
The practical rule is proportional to the distance. At function granularity,
skid is usually harmless. At line granularity it is unreliable unless precise
sampling is on. And for a hot line inside a short function, treat the
attribution as pointing at a neighbourhood rather than at a line.
Functions that are not there
The second distortion is the compiler doing exactly what it should.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents