Keeping a Secret in Production
Nobody Published It and It Is Published
Last timeChanging It Without an Outage
Most credentials that escape were never mishandled. They were logged, captured in an error report, printed by a build, or visible in a process list.
The recurring list
A credential that escapes was usually not mishandled by anybody. It was
written down by a machine, as a side effect of something useful, into a place
that nobody thought of as storage.
The list is short and it is the same in every organisation. Application logs,
where a request or a configuration object was printed to help with debugging.
Error and crash reports, where the capture of context is the entire feature.
Build output, where a deployment step echoed its command. Process listings,
where a value passed as an argument is visible to anybody on the host. Core
dumps, which are the contents of memory written to a file. Support bundles,
which are a deliberate collection of everything diagnostic. And screenshots
pasted into tickets, which are a human route with the same shape.
None of these involves anybody being careless with a secret. Every one of them
is a system doing its job.
| written by a machine | leaves your systems | kept for months or years | readable by more people | |
|---|---|---|---|---|
| application logs | 1 | 1 | 0 | 1 |
| error and crash reports | 1 | 1 | 1 | 1 |
| build output | 1 | 0 | 1 | 1 |
| process listings | 1 | 0 | 0 | 0 |
| core dumps | 1 | 1 | 1 | 1 |
| support bundles | 1 | 1 | 1 | 1 |
| screenshots in tickets | 0 | 1 | 1 | 1 |
Why a log is worse than a repository
Teams that would never commit a credential write them into logs every day, and
the comparison is worth making explicitly, because the log is the worse of the
two.
Readers. A repository is read by engineers. A log is read by engineers,
operators, support staff, analysts, anybody with access to the search tool, and
whoever is on call at a company you outsourced monitoring to.
Rules. Access to a repository is usually deliberate and reviewed. Access to
the log search tool is usually granted on the first day because it is needed
for the job, and removing it is nobody responsibility.
Retention. Code history is permanent, which is the problem in the previous
lesson. Logs are kept for ninety days or a year, which sounds better until you
notice that they are also exported, aggregated, backed up and sampled into
other systems, each with its own clock.
Copies. A log line written once is forwarded to an aggregator, indexed by a
search service, archived to object storage, and summarised into a dashboard.
That is four copies in systems with four sets of access rules, and the one you
know about is the first.
How it gets there
Three mechanisms account for almost all of it, and each has a specific fix.
An object printed whole. Somebody logs a request, a configuration or a client
object to see what is in it, and the printed form includes every field,
including the authorisation header or the password. The fix is to give these
objects a printed form that redacts credential fields, so that printing them is
safe by default rather than safe by remembering.
Context captured on an error. An error reporter attaches the environment, the
request headers and the local variables, because that is what makes a report
useful. The fix is the deny list in the reporting library, configured once,
plus a decision not to send the environment at all.
A value passed on a command line. A deployment step, a database tool or a
script receives the credential as an argument, which makes it visible in the
process listing to anybody on the host and prints it into the build output. The
fix is to pass it on standard input or through a file descriptor, which every
serious tool supports and almost nobody uses.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents