ContentsThe library

Keeping a Secret in Production

One Component Falls, and Then What

Last timeGetting It at Run Time

Most estates give every service the same identity and therefore the same reach. Scoping is the work that turns one compromise into a small compromise.

What a shared identity costs

Everything so far has been about the credentials. This lesson is about the

grant, which is the part that decides what a bad afternoon looks like.

The common arrangement is one identity for the whole estate. Every service

presents the same thing, every service can read every secret, and the secret

store is configured once by one person and never touched again. It is

convenient, it works, and it has a single property that matters: every

compromise is a total compromise.

Consider the image resizing service. It takes an uploaded file, makes a

thumbnail, and writes it to object storage. It needs one credential. Under the

shared arrangement it can read the payment key, the signing key, the mail

credentials and the primary database password, because everything can. It is

also, by some distance, the most likely component in the estate to be

compromised, since it parses files that strangers upload.

FIG 1The same compromise under two arrangements
Nine nodes and one branch. Everything expensive about a secrets programme sits upstream of this diagram, and this diagram is where the value is realised or lost. A perfect storage arrangement with one shared identity gives the right-hand outcome for every component in the estate.

Granting only what is used

Having given each component its own identity, the next question is what that

identity may read, and the way to answer it matters.

Do not ask the team. The answer from memory names the two or three obvious

connections and omits the metrics backend, the feature flag service, the

internal search index and whatever a script added two years ago. Starting from

that list produces an outage on the first deployment and a revert, after which

nobody attempts scoping again for a year.

Read the code instead. Every connection the component opens is in there, and

an afternoon of grepping for the ways connections are made produces a complete

list. Supplement it by watching one week of what the component actually reads

from the secret store, which catches the paths that only run during a monthly

job.

Then grant exactly that list, deploy to one component, and wait a full billing

cycle before declaring it done, because the thing that breaks is always the

quarterly report.

FIG 2Scoping one component, in order
plaintext
BEFORE CHANGING ANYTHING
  1. grep the code for every place a connection is opened
  2. list the credentials those connections need
  3. watch one month of reads from the store, add what is missing
  4. compare that list with what the component can read today

MAKE THE CHANGE
  5. create an identity for this component alone
  6. grant it the list from step 3 and nothing else
  7. deploy it to one instance first
  8. leave the old grant in place for one week

CONFIRM IT
  9. present the new identity, try to read something not on the list
  10. check that the refusal was recorded
  11. only then remove the old grant
Eleven steps, of which three are about not breaking anything and two are about proving the change did what it claimed. Teams that skip steps nine and ten end up with a diagram that says scoped and a store that still answers every request.

The credential everybody uses

There is one arrangement that defeats all of this, and it is extremely common.

Every service connects to the same database as the same database user. The

secret store can be scoped beautifully, with an identity per component and a

careful grant for each, and it achieves nothing for that credential, because

the value each component is permitted to read is the same value. Scoping who

may read a key is pointless when the key itself is shared.

The fix is at the other end. Give each component its own database account,

with its own password, and with privileges matching what that component does:

read-only where it only reads, no access to the tables it never touches. Then

the scoping in the secret store means something, because the credentials being

scoped are actually different.

This is more work than the store-side scoping and it is where most of the

benefit lives, which is an uncomfortable ordering, since the store-side part is

the part that gets funded.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the planThe contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The rest of this course

  1. 01Longer Than the List You Would Write From Memory
  2. 02Deleting It Does Not Undo Itopening only
  3. 03Two Questions Decide Every Place a Secret Can Sitopening only
  4. 04Follow the Chain Until It Stopsopening only
  5. 05One Component Falls, and Then Whatyou are here
  6. 06Both Values Have to Work for a Whileopening only
  7. 07Nobody Published It and It Is Publishedopening only
  8. 08The Order Is Not the One That Feels Urgentopening only

Read alongside