Keeping a Secret in Production
Longer Than the List You Would Write From Memory
Ask anybody which credentials their system holds and you get four. The real number is thirty, and the ones left off the list are the ones nobody is looking after.
The list from memory is wrong
Ask the person who built a service to name the credentials it holds. You will
get four: the database password, the payment key, the mail key, and something
about a certificate. The answer arrives quickly and confidently, and it is
wrong by roughly a factor of seven.
This is not carelessness. It is a property of how the list is produced. Asked
to remember secrets, people remember the things that were called secrets at the
moment they were set up, which is a small subset of the things that behave like
secrets now. Everything added since by a script, inherited from a predecessor,
or created by a platform on your behalf is absent, because it was never
introduced as a secret and nobody held it in their hand.
So do not produce the list by remembering. Produce it by walking.
The ones nobody calls secrets
The entries that get missed are missed systematically, which means they can be
listed in advance. Seven classes account for nearly all of them.
A connection string is a password with a hostname glued to the front, and
because it is called a string and is stored in configuration beside things like
the page size, it does not feel like a credential at all.
A signing key feels like mathematics rather than like a password, and is often
treated as configuration. Anybody holding it can mint a session for any account
in the system, which makes it the single most valuable item in most inventories
and the one least likely to be on the list.
A webhook secret is the value that proves an inbound message really came from
your payment provider. It is set up once, in a web form, by whoever did the
integration, and it is never thought about again.
An internal service token is the credential one of your own services uses to
call another. These are frequently permanent, frequently shared across every
service, and frequently absent from the list because internal sounds safe.
A backup is a copy of everything your database holds, usually sitting under
rules looser than the database itself.
A personal credential doing a system job is the one that causes the most
disruption. A scheduled task that runs as a named person works perfectly until
that person leaves, and in the meantime their access is the systems access.
And a cloud role is a credential even though it has no password. It is granted
by position rather than by knowledge, which means anything that can get into
that position inherits it.
| called a secret by the t | on the list from memory | places a copy lives | reaches customer data | |
|---|---|---|---|---|
| database password | 1 | 1 | 3 | 1 |
| session signing key | 0 | 0 | 6 | 1 |
| webhook secret | 0 | 0 | 4 | 1 |
| internal service token | 0 | 0 | 5 | 0 |
| backup file | 0 | 1 | 9 | 1 |
| job running as a person | 0 | 0 | 7 | 1 |
| cloud role | 0 | 0 | 2 | 1 |
What it unlocks
An inventory with no ordering is a wall, and a wall produces no work. Rank it,
and rank it by consequence rather than by how secret each thing feels.
The ranking question is a single sentence. If somebody outside the company had
this value and nothing else, what could they do by tomorrow morning. Not what
could they do with enough time and other holdings, which makes everything look
equally bad, but what could they do by tomorrow morning with this value alone.
The answers separate quickly into three bands. Some credentials read or write
customer data directly, which is the top band and usually has four or five
members. Some grant access to the systems that hold the credentials, which is
the same band dressed differently and is easy to miss. And some unlock a
third-party service that costs money or sends mail in your name, which is real
but recoverable.
- the number of credential copies in existence
- the number of distinct credentials in the inventory
- the average number of places each one lives
Counting the copies
The last column of the inventory is the one that changes behaviour, because it
is the one people get most wrong. Asked where a credential lives, almost
everybody names one place: the secret store, or the environment, or the
configuration file. The real answer is a list.
| step | where the copy is | who can read it | is the read recorded | survives a rotation | what happened |
|---|---|---|---|---|---|
| 1 | the secret store | two people and four services | 1 | 0 | The place everybody names. It is also the best protected of the eight, which is why protecting it harder is rarely the useful work. |
| 2 | the running process environment | anybody who can run a command in the con | 0 | 0 | Readable from the process list and from the process directory by anybody with access to the host, and recorded nowhere at all. |
| 3 | a laptop configuration file | whoever has the laptop | 0 | 1 | Set up during onboarding, never cleaned up, and still working after the rotation because nobody knew the copy was there. |
| 4 | a chat message to a colleague | everybody in the channel, forever | 0 | 1 | Sent once to unblock somebody at six in the evening. Searchable by every current and future employee with access to that channel. |
| 5 | a database backup | whoever can read the backup bucket | 0 | 1 | Contains the credentials other systems use, under storage rules usually looser than the database it came from. |
FOR EVERY CREDENTIAL, WRITE DOWN
1. the name people actually use for it
2. what it unlocks, in one sentence
3. every place a copy of it lives
4. who can read each of those places
5. when it was last changed
6. what stops working if it changes
FOUR QUESTIONS THAT FIND THE MISSED ONES
does anything here connect without proving who it is
does any scheduled job run as a person rather than as itself
has anybody pasted a value into chat to unblock a colleague
does a backup contain a file that a live process reads
WHEN THE LIST IS DONE
sort it by what each one unlocks by tomorrow morning
the top five are the whole of the next six lessonsThe inventory is not the interesting part of handling secrets, and it is the
part that decides whether any of the rest helps. Everything in the next seven
lessons is applied to a list. A short list produces a system that is carefully
protected in four places and wide open in twenty-six.
Recap
- Build the inventory by walking outward from the service, asking of every system it talks to how it proves who it is, rather than by trying to remember.
- The dangerous entries are the ones nobody calls a secret: a connection string, a signing key, a webhook value, a backup file, a job running as a person.
- Rank by what each one unlocks and by how many copies exist, because effort is finite and those two numbers decide where it goes.
This is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents