Answers · How a program runs

Why is a system call so much slower than calling a function?

In shortA system call is slow because the processor has to switch into the kernel and back, and that fixed cost is paid in full even when the request itself is trivial.

A system call is slow because the processor has to leave your code running as your code, run kernel code with your process still underneath it, and then put everything back exactly as it was. A function call costs about 2 nanoseconds. A request to the kernel costs about 500. One is two hundred and fifty times the other, for two lines that look identical where you write them.

Nothing about the instruction that makes the request is expensive. It is the sequence behind it that has a fixed price: save the register state, move to a stack the kernel controls, raise the privilege level, install the kernel's address mappings, check every pointer you passed against what your process is allowed to touch, and then do all of it backwards on the way out. Those steps do not read your arguments. Ask for one byte or one megabyte and the same work happens.

FIG 1

step	what it does	nanoseconds
1	your code puts the arguments in registers	2
2	the trap instruction, privilege switch, mapping change	180
3	the kernel validates arguments and finds the handler	60
4	the actual work, on data already in a kernel buffer	90
5	return, undoing everything from step 2	170

Add the column and the shape of the problem appears. 2 + 180 + 60 + 90 + 170 = 502 nanoseconds for a request whose data was already sitting in a buffer the kernel held. The step that did what was asked accounts for 90 of those, and 90/502 is under a fifth. The rest, 180 + 60 + 170 = 410 nanoseconds, plus the 2 to get there, is boundary work that would have been paid whatever the request was. Divide 502 by 2 and you get 251: one request is worth about two hundred and fifty function calls.

The natural guess is that the call costs what it does because of what it does: that opening a file or reading a directory is inherently heavier than adding numbers, and the price lives in the operation. It does not. The price lives in the crossing, and the crossing is identical for a request that returns one integer and one that returns a page of bytes. The same mistake runs the other way too. A call that looks trivial from outside, such as asking whether a file exists, is not cheap because the answer is short. It pays the same four to five hundred nanoseconds, which is why a loop that asks the kernel a thousand small questions is slow in a way that no tuning inside the loop will touch.

Where the crossing sits

The comparison only means anything against the other waits in the same program. These are nanoseconds, and the third column is each figure divided by two, the price of one function call, so the ratios are visible.

FIG 2

operation	nanoseconds	in function calls
a function call	2	1
a request to the kernel	500	250
a minor page fault	3000	1500
a switch to another process	16000	8000
a read from a solid-state disk	100000	50000
a read from a spinning disk	10000000	5000000

Read the third column rather than the second. A page fault is 3000/2 = 1500 function calls, a solid-state read is 100000/2 = 50000, and a spinning read is 10000000/2 = 5000000. That last figure says one disk read costs what five million function calls cost, which is why a program doing one needless read per iteration is not slow by a factor of two but by a factor of everything. It also puts the crossing in its place: 500 nanoseconds is enormous next to 2 and almost nothing next to 50000, so a system call is worth batching and a disk read is worth not doing.

Why the price went up in 2018

The crossing got dearer in 2018, so advice written before then understates it. Until that year the kernel's pages were mapped into every process and marked inaccessible, which made entry cheap: no mapping change was needed. The speculation attacks published that January showed the marking was advisory, because a processor speculating past a check leaves measurable traces. The fix chosen was to unmap the kernel from the user address space, so every entry installs the mapping and every exit removes it, and removing it flushes the caches that translate addresses, which slows the memory accesses just after the crossing as well. The entry step above is given as 180 nanoseconds; on a machine without those mitigations it is nearer 60.

Paying once instead of a thousand times

Cost per item is the crossing divided by how many items you put in it, plus the work each item genuinely needs.

a = \frac{e}{n} + w

With a crossing of 500 nanoseconds and 20 nanoseconds of real work per item: one item per request gives 500/1 + 20 = 520 nanoseconds per item; eight gives 500/8 + 20 = 62.5 + 20 = 82.5; sixty-four gives 500/64 + 20 = 7.8 + 20, about 28. Going from one to eight removes 437.5 nanoseconds of crossing per item, and going from eight to sixty-four removes a further 54.7. That is the entire case for buffered writes, for reading in blocks rather than bytes, and for the shared queues where the kernel watches a ring your program fills and the crossings per request approach zero.

Batching stops paying when the second term dominates. Raise the work per item to 200 nanoseconds and four per request already cost 500/4 + 200 = 325 against 500/1 + 200 = 700 for one: most of the win is in the first four, and by sixteen items the crossing contributes 31 against 200 and is noise. Two things move the figures above and are worth checking on the machine in question. Hardware since 2018 has clawed back part of the mitigation cost, so the crossing varies between machines more than it used to. And the 90 nanosecond step becomes a page fault of 3000 if the data is not already in memory, which is 3000/90 = 33 times the step it replaced, and 3000/502 = 6 times the entire crossing you just paid. No amount of batching removes that.

Worth keeping

A function call costs about 2 nanoseconds and a request to the kernel about 500, so a single crossing is worth roughly 250 function calls.
The crossing is a fixed price: reading one byte and reading a megabyte pay the same entry and exit, so the fix is fewer requests rather than faster ones.
Unmapping the kernel from user address space after the 2018 attacks roughly doubled the entry cost, which makes pre-2018 rules of thumb about it wrong.

Where this goes next

This question is one step of What Happens While You Wait, a course that works through its subject from the beginning. The page that teaches this part of it is The Cost of Asking, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course