The Order Is Not the One That Feels Urgent

Last timeWhere It Leaks From

A credential is exposed. The instinct is to find out what happened. The correct first move is to make the value worthless, and everything else comes after.

Make it worthless, then think

Somebody finds a credential where it should not be. A key in a log line, a

password in a ticket, a token in a public repository. What happens in the next

ten minutes is almost always the wrong thing, and the wrong thing is natural.

The instinct is to understand. Where did it come from. How long has it been

there. Did anybody see it. Who committed it. Is this one still in use. Those

questions are all reasonable and they all take hours, and during every one of

those hours the credential still works.

Invert it. Revocation takes minutes. Investigation takes hours. Doing the

investigation first costs exactly the difference, and it buys nothing, because

the answers do not change what has to be done.

There is one argument people reach for here, and it should be named and

dismissed. The argument is that rotating immediately might break something, so

we should first understand what uses it. That argument is a request to leave a

known-exposed credential working during business hours in order to avoid an

outage, and the honest version of it is a decision somebody should make out

loud rather than a reason to delay by default.

FIG 1Two responses to the same discovery, in minutes

Both teams reach the same place by minute one hundred and eighty, having done the same work in a different order. The difference is a hundred and twelve minutes during which the credential was known to be exposed and still worked, which is the only number in this lesson that an attacker cares about.

The whole order

Seven steps. The first two are the response; the rest is the clean-up, and the

clean-up matters.

Revoke or rotate. Make the value stop working, by whichever mechanism exists.

Confirm it failed, by presenting the old value. A revocation that silently did

not take effect is the worst state available, because everybody now believes

the problem is solved.

Establish the exposure window, from the moment it was exposed to the moment it

stopped working. Not from when somebody noticed. The push, the log line, the

screenshot, whatever it was, has a timestamp, and that is the start.

Read what it did during the window. The access logs of whatever the credential

unlocked, filtered to that credential and that window. This is the step that

replaces speculation with a list.

Tell whoever must be told. Usually that is a small, specific list decided in

advance, and the thing that makes it hard at the time is not knowing who is on

it.

Replace it properly, with the full rotation from the previous lesson, with its

overlap and its checks. The emergency revocation was a blunt act; the proper

replacement is the careful one.

Fix the route it escaped by, which is the only step that prevents the next one.

FIG 2The response, with the trap drawn in

Twelve nodes, and the whole lesson is in the first branch. Everything after it is ordinary work that any competent team does well when it is not under pressure, which is why the preparation that matters is making the first decision automatic.

What it could have done

Once the value is dead, the question becomes what happened while it was alive,

and the answer should come from records rather than from reasoning.

Take the window. Take the access logs of whatever the credential unlocked.

Filter to that credential, in that window, and read what comes back. If the

credential was a database account, that is the connection log and the query

log. If it was an interface key, the request log of that interface. If it was a

signing key, the question is different and harder: anything it signed is

indistinguishable from something you signed, which is why signing keys sit at

the top of every inventory.

Three outcomes. Nothing used it, which is the usual answer, and it is worth

recording with the evidence. Something used it from an address you recognise,

which is one of your own systems and means the rotation may have broken

something. Or something used it from an address you do not recognise, which is

a different kind of day and is the one the next question is for.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents