Keeping a Secret in Production
One Component Falls, and Then What
Last timeGetting It at Run Time
Most estates give every service the same identity and therefore the same reach. Scoping is the work that turns one compromise into a small compromise.
What a shared identity costs
Everything so far has been about the credentials. This lesson is about the
grant, which is the part that decides what a bad afternoon looks like.
The common arrangement is one identity for the whole estate. Every service
presents the same thing, every service can read every secret, and the secret
store is configured once by one person and never touched again. It is
convenient, it works, and it has a single property that matters: every
compromise is a total compromise.
Consider the image resizing service. It takes an uploaded file, makes a
thumbnail, and writes it to object storage. It needs one credential. Under the
shared arrangement it can read the payment key, the signing key, the mail
credentials and the primary database password, because everything can. It is
also, by some distance, the most likely component in the estate to be
compromised, since it parses files that strangers upload.
Granting only what is used
Having given each component its own identity, the next question is what that
identity may read, and the way to answer it matters.
Do not ask the team. The answer from memory names the two or three obvious
connections and omits the metrics backend, the feature flag service, the
internal search index and whatever a script added two years ago. Starting from
that list produces an outage on the first deployment and a revert, after which
nobody attempts scoping again for a year.
Read the code instead. Every connection the component opens is in there, and
an afternoon of grepping for the ways connections are made produces a complete
list. Supplement it by watching one week of what the component actually reads
from the secret store, which catches the paths that only run during a monthly
job.
Then grant exactly that list, deploy to one component, and wait a full billing
cycle before declaring it done, because the thing that breaks is always the
quarterly report.
BEFORE CHANGING ANYTHING
1. grep the code for every place a connection is opened
2. list the credentials those connections need
3. watch one month of reads from the store, add what is missing
4. compare that list with what the component can read today
MAKE THE CHANGE
5. create an identity for this component alone
6. grant it the list from step 3 and nothing else
7. deploy it to one instance first
8. leave the old grant in place for one week
CONFIRM IT
9. present the new identity, try to read something not on the list
10. check that the refusal was recorded
11. only then remove the old grantThe credential everybody uses
There is one arrangement that defeats all of this, and it is extremely common.
Every service connects to the same database as the same database user. The
secret store can be scoped beautifully, with an identity per component and a
careful grant for each, and it achieves nothing for that credential, because
the value each component is permitted to read is the same value. Scoping who
may read a key is pointless when the key itself is shared.
The fix is at the other end. Give each component its own database account,
with its own password, and with privileges matching what that component does:
read-only where it only reads, no access to the tables it never touches. Then
the scoping in the secret store means something, because the credentials being
scoped are actually different.
This is more work than the store-side scoping and it is where most of the
benefit lives, which is an uncomfortable ordering, since the store-side part is
the part that gets funded.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents