One Kernel, Many Processes, and No Boot Sequence Anywhere

A container is a process the running kernel created with a few extra flags on the call, and both numbers people quote about containers follow from that.

Somebody starts a container for the first time, gets a prompt, asks for a list

of processes and sees one line. So they conclude they are inside a small

machine, and then they go looking for the things a small machine has. There is

no system log. There is nothing managing services. There is nothing listening

on the address they would log in to. An hour goes on that search, and the hour

is the cost of the picture rather than a fault in the tool.

There is no second machine and no second kernel. What was started is a

process. The kernel already running on your machine created it the same way it

creates every other process, with a few extra flags on the call saying which

parts of the system the new process is allowed to see. Everything in this

subject follows from that one sentence, and the two numbers everybody quotes,

the memory saving and the start-up time, are both consequences you can work out

on paper.

Why it looks like a machine from the inside

Four signals arrive at once, and they are convincing.

The prompt says root, with a short unfamiliar host name after it. The process

list has one entry and that entry is numbered one. There is a full directory

tree, with the usual places a distribution installs itself into. The network

has an address of its own, which is not the address of the machine the command

was typed on.

Those four are exactly the four things a person uses to decide whether they are

on a machine, and they are exactly four of the views that get separated. The

illusion is nobody's design. It is what separation looks like from the inside

when it is working, and nothing in the container is lying to you: the kernel is

answering every one of those questions with respect to what this particular

process is allowed to see.

What the picture costs is three searches that cannot succeed. There is no boot

log, because nothing booted. There is nothing bringing services up, because

nothing asked for anything to be brought up, and the one process running is the

service itself. There is nothing to log in to, because logging in would need a

program listening on a port, and no such program was started. Every one of

those absences is correct behaviour, and every one of them reads as a broken

installation to somebody holding the wrong picture.

FIG 1A container start, with the orchestration taken away

static int start_the_service(void *image_root) {
    pivot_root_to(image_root);          /* the mounted layers become / */
    execv("/usr/bin/gunicorn", argv);   /* this process becomes the service */
    return 1;                           /* reached only if exec failed */
}

int flags = CLONE_NEWNS | CLONE_NEWPID | CLONE_NEWNET
          | CLONE_NEWUTS | CLONE_NEWIPC | SIGCHLD;
pid_t pid = clone(start_the_service, stack_top, flags, NULL);

The whole mechanism is on the last three lines. Take the five CLONE flags off and this is the ordinary way any program on the machine starts another one, which is the point: a container is not a different kind of thing being created, it is the same thing created with five bits set.

What starting one actually does

The call is clone, and it is what process creation goes through anyway. Each of

the flags asks for one view to be given to the new process instead of being

inherited. CLONE_NEWUTS is the host name, CLONE_NEWPID the process numbers,

CLONE_NEWNS the mounted file systems, CLONE_NEWNET the network, and

CLONE_NEWIPC the shared memory and message channels. There is a sixth view, the

user numbers, asked for with CLONE_NEWUSER, and it is usually not asked for at

all, which has its own reasons and its own consequences.

Notice what is not in the call. Nothing allocates a machine. Nothing loads a

kernel. The flags change what one process can see and they create nothing

capable of answering a system call, so every request this process makes from

here on is answered by the kernel that was already running before the command

was typed.

FIG 2What happens between the command and the service answering

Read the chain and count the steps that involve a second kernel. There are none. The reason a start takes milliseconds is not that these steps are fast but that the expensive steps of starting a machine are absent from the list.

The exec on the last line is worth pausing on. The new process replaces itself

with the image's program, so when the start finishes there is exactly one

process and it is the service. Nothing started it on the service's behalf and

nothing is watching it from inside, which is a fact with consequences that turn

up later as mysteries.

FIG 3Where the 180 MB of the image is

120 plus 48 plus 12 is 180, and the two layers not drawn here carry no bytes at all, only a note about the environment and the command to run, so they add 0 MB each. The application's own share is 12 of 180, which is one fifteenth, or 6.7 per cent: almost everything shipped is a distribution the application did not write.

The bytes, counted once instead of ten times

Take the image as fixed: a small web service, 180 MB, in five layers. Three of

them carry bytes. The base system is 120 MB, the dependencies 48 MB and the

application code 12 MB, and $120 + 48 + 12 = 180$ . The remaining two layers

carry metadata only, an environment variable and the command to run, so they

contribute 0 MB each and the total is still 180 MB. The application's share is

$12 / 180 = 0.067$ , so 6.7 per cent of what is shipped is the part anybody

wrote on purpose.

Now run ten of them. The layers are read only, and a second container from the

same image does not get a copy of them: it is shown the same bytes, with an

empty writable layer on top for anything it writes. Say each container writes

2 MB of its own, a log file and a small cache. Then ten containers cost

$180 + 10 \times 2 = 200$ MB, which is 20 MB an instance. Ten small virtual

machines at 1.2 GB each cost $10 \times 1200 = 12{,}000$ MB, which is 12 GB, and

$12{,}000 / 200 = 60$ , so the same ten workloads come out sixty times smaller.

The marginal number is sharper than the ratio. The eleventh container adds 2 MB,

taking 200 MB to 202 MB, a rise of $2 / 200 = 0.01$ , or 1 per cent. The

eleventh virtual machine adds 1,200 MB, taking 12,000 MB to 13,200 MB, a rise

of $1200 / 12{,}000 = 0.1$ , or 10 per cent. One of those grows with the number of

instances and the other very nearly does not.

FIG 4Ten of each, counted

step	what the host is asked to run	stored bytes it has to hold	time until it is serving	what happened
1	one container from the image	180 shared + 2 written = 182 MB	80 ms	The 180 MB is the image's read-only layers, as files in the host's own file system. The 2 MB is this container's own writable layer.
2	ten containers from that image	180 + 10 x 2 = 200 MB	10 x 80 ms = 800 ms, started one at a time	The first column of this sum does not change with the count. That is the entire saving, and it is sharing rather than compression.
3	one small virtual machine	1,200 MB	25,000 ms	Inside that 1.2 GB is a kernel of its own, its own copy of the same base system, and its own set of services.
4	ten small virtual machines	10 x 1,200 = 12,000 MB	10 x 25,000 ms = 250,000 ms, or 4 min 10 s	Nothing is shared because nothing can be: each copy is inside a disk image only that guest can interpret, and the host sees one opaque file.

4 steps

Compare the two middle rows against the two outer ones. Both quantities multiply by ten for the virtual machines and only the small one multiplies for the containers, so the ratio between the two arrangements is not fixed: it grows with the number of instances.

The reason the machine cannot share is structural rather than a missed

optimisation. Its 1.2 GB sits inside a disk image that only the guest knows how

to interpret, so when two guests hold the same library the host sees two opaque

files and has no way to notice they contain the same bytes. The container's

180 MB is ordinary files in the host's own file system, and the host is the one

reading them.

That argument repeats in memory. The read-only layers are mapped once into the

host's page cache, and every container reading the same library gets the same

physical pages, while each guest's copy of that library is a separate page the

host cannot recognise as a duplicate. The stored bytes are the easier number to

work out, so they are the one worked here, but the saving is the same saving

twice.

FIG 5Stored bytes against the number of instances

containers from one imagesmall virtual machines

One line has a slope of 1,200 and the other a slope of 2, which is the whole figure. The container line starts higher than the origin, because the image is paid for once whether one instance runs or twenty, and then is nearly flat; at twenty instances the two lines read 220 MB and 24,000 MB.

Where the 25 seconds goes

A container starts in about 80 milliseconds and a virtual machine in about 25

seconds, which is 25,000 milliseconds, so the ratio is

$25{,}000 / 80 = 312.5$ . That number gets quoted as though containers had a

faster boot. They have no boot.

FIG 6

what a virtual machine does first	what the container does instead
firmware, then a bootloader	nothing, the host booted long ago
unpack a kernel, probe the devices	nothing, the kernel is already running
a service manager brings up services	nothing, the one process is the service
the application process finally starts	this is the only step on the list

Three of the four rows are absent rather than quick, so the difference

$25{,}000 - 80 = 24{,}920$ milliseconds is almost exactly the boot:

$24{,}920 / 25{,}000 = 0.997$ , or 99.7 per cent of the virtual machine's

start-up.

The honest comparison is available, and it is a useful one to know. A virtual

machine built for fast starts, with a stripped kernel, no firmware to speak of

and no service manager, starts in about 125 milliseconds. Against the

container's 80 that is $125 / 80 = 1.56$ , so having a kernel of your own costs

around $125 - 80 = 45$ milliseconds. The rest, $25{,}000 - 125 = 24{,}875$

milliseconds, is 99.5 per cent of the 25 seconds and it is a boot sequence,

not virtualisation. Anyone quoting three hundred times is comparing against a

boot.

The 80 milliseconds has a practical shape too. Replacing fifty copies of a

service one at a time costs $50 \times 0.08 = 4$ seconds of start-up with

containers and $50 \times 25 = 1250$ seconds, or 20 minutes and 50 seconds,

with machines. That difference is why a deployment can be done by replacing

instances one by one and watching, rather than being an event somebody schedules.

What one kernel already implies

None of the following needs any isolation to be configured. They follow from

there being one kernel, and none of them can be configured away.

One kernel version for everybody. The image carries no kernel, so a program

that needs a newer system call does not get it by using a newer base image.

A kernel module cannot sensibly be loaded from inside, because loading one

changes the kernel every other container is also using.

A flaw in that kernel is on both sides of the wall, which puts a ceiling on

how much separation any of this can give you.

One scheduler and one page cache. Containers compete for the same processors

and the same cached pages, so a noisy neighbour turns up in your timings.

No boot means nothing arranging services, which leaves an ordinary program

holding process number one and inheriting the duties that number carries.

The last is the one that bites hardest, because the duties of process number

one are real and the program holding it was written assuming something else

would do them.

The test to apply instead

The replacement for the small machine picture is a question. When something

surprises you, ask what an ordinary process with a narrowed view of the system

would do here, and the answer is usually immediate. The container did not

restart, a process exited and something outside started another one. The

container is not slow, a process is sharing a processor with the rest of the

machine. The log is not missing, the process wrote to its own output and nobody

had arranged for that to go anywhere.

Each of the views the clone call asked for narrows one specific thing, and each

one explains a different class of surprise. The place to start is the process

numbers, because that is the view which makes a container look like a machine

with one program on it while the host's own list shows something quite

different.

Recap

Starting a container is one ordinary process creation with a handful of extra flags on it, so there is no second kernel to boot and nothing a boot sequence would normally arrange exists unless the image's own program arranges it.
Ten containers from one 180 MB image cost the host 180 MB once plus whatever each writes, which is 200 MB in total against the 12 GB that ten 1.2 GB virtual machines need, and the saving is sharing rather than compression.
The gap between 80 milliseconds and 25 seconds is the absence of a boot rather than a faster one, which is why a virtual machine stripped down to a kernel and one program starts in about 125 milliseconds and closes almost all of that gap without removing any virtualisation.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextA Different View of the System →