ContentsThe library

Keeping a Secret in Production

Follow the Chain Until It Stops

Last timeWhere It Can Live Instead

A process has no credentials and must end up with all of them. Every scheme for that is a chain, and the only question worth asking is where the chain stops.

What starting up looks like

A process begins with nothing. It has a user identity, a working directory, a

set of file handles, and no knowledge of any password anywhere. A few hundred

milliseconds later it is talking to a database, a payment provider and an

object store. Something happened in between, and that something is the subject

of this lesson.

The shape is always the same. The process presents one thing it already has.

It receives a short-lived token in exchange. It uses that token to fetch the

credentials it needs. It holds those in memory and uses them until it exits.

FIG 1A process going from nothing to everything
Eleven nodes, and the interesting one is the second. Everything after it is machinery that can be bought or copied from documentation; the second node is a design decision specific to your deployment, and it determines how much all the machinery is worth.

The one it already has

There is a step that cannot be fetched. Before anything can be requested,

something has to be presented, and that something was placed there by someone

or something else. This is the bootstrap credential, and it is the quietest

part of every secrets architecture.

It is quiet for a reason that is almost funny. The diagram on the whiteboard

shows the secret service in the middle with arrows coming out of it, and

everybody looks at the arrows. The thing that opens the service is drawn as a

small unlabelled line at the edge, because it was handled during setup, by one

person, probably in an afternoon, and never revisited.

The consequence is a system where the credentials are held carefully, rotated

on a schedule, and audited on read, and all of them are reachable by anybody

who can present one unrotated value that three people know and that is also in

a configuration management repository from four years ago.

FIG 2Four bootstrap mechanisms, compared on who else could stand there
nothing is stored on disworks unattendedsurvives moving to anothresists somebody who own
platform identity grante1100
certificate installed wh0110
a value a person types a1111
hardware root on the mac0011
The marked cells are the two extremes of the trade. A person typing a value is the only mechanism that nothing on the machine can replay, and it is also the only one that cannot survive an unattended restart at four in the morning. Hardware is the only column-four one, and it ties the credential to a specific physical machine.

Where the chain ends

Four endings, and each one is an assumption worth stating out loud.

A platform identity is the common modern answer. The process runs somewhere,

and the place it runs vouches for it: anything running in this position is this

service. Nothing is stored anywhere, which is a genuine improvement, and the

assumption is that nothing else can get into that position. That assumption is

exactly as strong as the controls on who can start something there, which is a

different team and a different review.

A certificate placed into the image at build time pushes the problem to the

build system, which now holds a credential that can obtain every credential.

Build systems are frequently the least protected part of an estate, with broad

access, many contributors and extensive logging.

A person typing a value at start is the strongest against replay and the

weakest operationally. Nothing on the machine can reproduce it, and nothing

restarts without a human. A few systems genuinely warrant this, and most do

not, and the ones that adopt it informally usually end up with the value saved

somewhere so that the restart works at night.

A hardware root ties the ability to bootstrap to a specific physical machine.

It is the strongest ending and it ends there: somebody who can run code on that

machine can still ask it to do the bootstrap.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the planThe contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The rest of this course

  1. 01Longer Than the List You Would Write From Memory
  2. 02Deleting It Does Not Undo Itopening only
  3. 03Two Questions Decide Every Place a Secret Can Sitopening only
  4. 04Follow the Chain Until It Stopsyou are here
  5. 05One Component Falls, and Then Whatopening only
  6. 06Both Values Have to Work for a Whileopening only
  7. 07Nobody Published It and It Is Publishedopening only
  8. 08The Order Is Not the One That Feels Urgentopening only

Read alongside