Two Questions Decide Every Place a Secret Can Sit

Last timeWhy Not in the Repository

File, environment, secret manager, kernel keyring, hardware module. They differ on exactly two things that matter, and comparing them on anything else wastes the afternoon.

The two questions

Discussions about where to keep credentials go wrong by comparing too many

things at once. Encryption at rest, vendor names, whether something is managed,

how the architecture diagram looks. None of those separate the options, because

every option encrypts at rest and every vendor says the same words.

Two questions do separate them.

The first is who can read the value. Not who should: who can, counting

everybody with the access they actually hold today, including the people

administering the thing that holds it.

The second is whether a read leaves a record. If somebody did read it, would

you ever know, and would you know when and from where. This question is almost

never asked and it is the one that decides what an incident looks like six

months from now.

Everything else is either implied by these two or is not a security property.

FIG 1Applying the two questions to a place you already use

plaintext

QUESTION ONE: WHO CAN READ IT
  name every person with the access today
  name every process running on the same host
  name the administrators of the holding system
  name anybody who can restore a backup of it
  count them, and write the number down

QUESTION TWO: IS THE READ RECORDED
  if somebody read it an hour ago, is there a line somewhere
  does that line say who, when and from where
  how long is the line kept
  would anybody ever look at it

THE ANSWER IS USUALLY
  question one: more people than anybody expected
  question two: no

Run this on the place your most valuable credential sits in right now. The exercise takes ten minutes and almost always produces a count that nobody in the room would have guessed, which is the point of counting rather than estimating.

The five places

There are five real options, and they form a clean ordering rather than a

confusing field.

A file on disk with restrictive permissions is the simplest. Readable by the

owner and by anybody who becomes the owner, by anybody with administrative

access to the host, and by anybody holding a backup of the filesystem. No

record of reads. It is better than people assume, because the permission check

is enforced by the operating system every time rather than by convention.

The process environment is the most common and has the worst properties, which

is the subject of the next section.

A secret service that the process fetches from over the network is the usual

answer at any scale. The value is held centrally, access is granted per

identity, and every fetch is recorded with who and when. That recording is the

real improvement, more than the central storage is.

An operating system facility designed for key material holds the value outside

your process address space, with ownership and permissions attached to each key

and with the kernel mediating every access. It keeps the value out of crash

dumps and out of the environment, and it is underused.

Dedicated hardware is the only option that changes the shape of the problem,

because it never returns the value at all. You send it data and it sends back a

signature or a decryption. Nobody reads the key, including you, including an

attacker who completely owns the machine.

FIG 2The five places on the questions that distinguish them

	readable by anybody on t	the read is recorded	appears in a crash dump	readable by your own pro	needs a bootstrap creden
file with restrictive pe	1	0	1	1	0
process environment	1	0	1	1	0
secret service over the	0	1	0	1	1
operating system key fac	0	0	0	1	0
dedicated hardware	0	1	0	0	1

The fourth column is the one that surprises people: four of the five hand the actual value to your process, which means code running as your process can read it however carefully it was stored. The marked cells are the extremes, the environment readable by anything on the host and the hardware that never gives the value up at all.

What is wrong with the environment

Putting credentials in environment variables became the default through a

combination of convenience and a widely read piece of deployment advice, and it

has the weakest properties of the five. Four specific problems.

It is inherited. Every child process your service starts receives the entire

environment, including the credentials, whether or not it needs them. A service

that shells out to a tool to resize an image has just handed that tool the

database password.

It is visible. On many systems the environment of a running process is

readable through the process filesystem by the owner and by administrators, and

on some older systems and some container setups it is readable more widely than

that. It is also frequently visible in whatever lists running processes.

It leaks into reports. Crash handlers, error pages and monitoring agents

routinely capture the environment as diagnostic context, which republishes

every credential into a third-party service that was never meant to hold them.

This is a common and under-noticed route for a value to escape.

And reads are recorded nowhere. There is no moment at which the value is

fetched, so there is nothing to log. Six months later there is no way to ask

whether anything read it.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents