ContentsThe library

Keeping a Secret in Production

Longer Than the List You Would Write From Memory

Ask anybody which credentials their system holds and you get four. The real number is thirty, and the ones left off the list are the ones nobody is looking after.

The list from memory is wrong

Ask the person who built a service to name the credentials it holds. You will

get four: the database password, the payment key, the mail key, and something

about a certificate. The answer arrives quickly and confidently, and it is

wrong by roughly a factor of seven.

This is not carelessness. It is a property of how the list is produced. Asked

to remember secrets, people remember the things that were called secrets at the

moment they were set up, which is a small subset of the things that behave like

secrets now. Everything added since by a script, inherited from a predecessor,

or created by a platform on your behalf is absent, because it was never

introduced as a secret and nobody held it in their hand.

So do not produce the list by remembering. Produce it by walking.

FIG 1Walking the inventory instead of recalling it
The procedure takes about two hours for a service of moderate size, needs no tooling, and reliably produces a list three to eight times longer than the one produced from memory. The value is not in the tooling; it is in asking every connection the same question.

The ones nobody calls secrets

The entries that get missed are missed systematically, which means they can be

listed in advance. Seven classes account for nearly all of them.

A connection string is a password with a hostname glued to the front, and

because it is called a string and is stored in configuration beside things like

the page size, it does not feel like a credential at all.

A signing key feels like mathematics rather than like a password, and is often

treated as configuration. Anybody holding it can mint a session for any account

in the system, which makes it the single most valuable item in most inventories

and the one least likely to be on the list.

A webhook secret is the value that proves an inbound message really came from

your payment provider. It is set up once, in a web form, by whoever did the

integration, and it is never thought about again.

An internal service token is the credential one of your own services uses to

call another. These are frequently permanent, frequently shared across every

service, and frequently absent from the list because internal sounds safe.

A backup is a copy of everything your database holds, usually sitting under

rules looser than the database itself.

A personal credential doing a system job is the one that causes the most

disruption. A scheduled task that runs as a named person works perfectly until

that person leaves, and in the meantime their access is the systems access.

And a cloud role is a credential even though it has no password. It is granted

by position rather than by knowledge, which means anything that can get into

that position inherits it.

FIG 2Seven classes, and how each one does on the four questions
called a secret by the ton the list from memoryplaces a copy livesreaches customer data
database password1131
session signing key0061
webhook secret0041
internal service token0050
backup file0191
job running as a person0071
cloud role0021
The first two columns are almost entirely zeros and the last is almost entirely ones, which is the finding of this lesson in one picture. The marked cells are the two worst: a signing key that can mint any session and is on nobody list, and a backup that exists in nine places.

What it unlocks

An inventory with no ordering is a wall, and a wall produces no work. Rank it,

and rank it by consequence rather than by how secret each thing feels.

The ranking question is a single sentence. If somebody outside the company had

this value and nothing else, what could they do by tomorrow morning. Not what

could they do with enough time and other holdings, which makes everything look

equally bad, but what could they do by tomorrow morning with this value alone.

The answers separate quickly into three bands. Some credentials read or write

customer data directly, which is the top band and usually has four or five

members. Some grant access to the systems that hold the credentials, which is

the same band dressed differently and is easy to miss. And some unlock a

third-party service that costs money or sends mail in your name, which is real

but recoverable.

FIG 3Copies, from credentials and places
the number of credential copies in existence
the number of distinct credentials in the inventory
the average number of places each one lives
Thirty credentials at six places each is a hundred and eighty copies, every one of which is a way out. The reason this number is worth computing is that most effort goes into protecting the first place, which is typically the best protected of the six already.

Counting the copies

The last column of the inventory is the one that changes behaviour, because it

is the one people get most wrong. Asked where a credential lives, almost

everybody names one place: the secret store, or the environment, or the

configuration file. The real answer is a list.

FIG 4One database password, followed to every place it exists
stepwhere the copy iswho can read itis the read recordedsurvives a rotationwhat happened
1the secret storetwo people and four services10The place everybody names. It is also the best protected of the eight, which is why protecting it harder is rarely the useful work.
2the running process environmentanybody who can run a command in the con00Readable from the process list and from the process directory by anybody with access to the host, and recorded nowhere at all.
3a laptop configuration filewhoever has the laptop01Set up during onboarding, never cleaned up, and still working after the rotation because nobody knew the copy was there.
4a chat message to a colleagueeverybody in the channel, forever01Sent once to unblock somebody at six in the evening. Searchable by every current and future employee with access to that channel.
5a database backupwhoever can read the backup bucket01Contains the credentials other systems use, under storage rules usually looser than the database it came from.
5 steps
Five of the eight usual places, and the pattern is consistent: the one everybody protects is the one with the tightest access and the only audit record, and the four nobody thinks about all survive a rotation. Rotating a value without finding the copies leaves most of the exposure in place.
FIG 5The inventory worksheet
plaintext
FOR EVERY CREDENTIAL, WRITE DOWN
  1. the name people actually use for it
  2. what it unlocks, in one sentence
  3. every place a copy of it lives
  4. who can read each of those places
  5. when it was last changed
  6. what stops working if it changes

FOUR QUESTIONS THAT FIND THE MISSED ONES
  does anything here connect without proving who it is
  does any scheduled job run as a person rather than as itself
  has anybody pasted a value into chat to unblock a colleague
  does a backup contain a file that a live process reads

WHEN THE LIST IS DONE
  sort it by what each one unlocks by tomorrow morning
  the top five are the whole of the next six lessons
Two hours with this worksheet beats any amount of scanning, because the scanner finds the copies that look like secrets and this finds the credentials that nobody ever called one.

The inventory is not the interesting part of handling secrets, and it is the

part that decides whether any of the rest helps. Everything in the next seven

lessons is applied to a list. A short list produces a system that is carefully

protected in four places and wide open in twenty-six.

Recap

  • Build the inventory by walking outward from the service, asking of every system it talks to how it proves who it is, rather than by trying to remember.
  • The dangerous entries are the ones nobody calls a secret: a connection string, a signing key, a webhook value, a backup file, a job running as a person.
  • Rank by what each one unlocks and by how many copies exist, because effort is finite and those two numbers decide where it goes.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextWhy Not in the Repository →

The rest of this course

  1. 01Longer Than the List You Would Write From Memoryyou are here
  2. 02Deleting It Does Not Undo Itopening only
  3. 03Two Questions Decide Every Place a Secret Can Sitopening only
  4. 04Follow the Chain Until It Stopsopening only
  5. 05One Component Falls, and Then Whatopening only
  6. 06Both Values Have to Work for a Whileopening only
  7. 07Nobody Published It and It Is Publishedopening only
  8. 08The Order Is Not the One That Feels Urgentopening only

Read alongside