Finding Structure Without Labels
Flagging the Odd One Out
Last timeDescribing It With Fewer Numbers
Scoring how unusual a row is can be done three ways, and the threshold on that score is not a statistical question but an arithmetic one about what a false alarm costs.
The third question
The first lesson named three questions. Two of them are done. The third is
which rows do not look like the rest, which is how fraud, faults, broken
sensors and corrupted records are found when nobody has labelled any examples.
This is the question where the no answer key problem bites hardest, because
there usually are a handful of labelled examples, from the cases somebody
caught by hand, and a handful is enough to mislead and not enough to train on.
Three ways to score a row
Every method here gives each row a number saying how unusual it is. The
threshold on that number is the next section; the score comes first, and there
are three families.
Distance from the neighbours. Take the average distance from a row to its ten
nearest neighbours, and that is the score. Simple, and it fails whenever the
data has regions of different density: a point sitting comfortably inside a
sparse group scores higher than a point on the edge of a dense group, which is
backwards.
Sparseness relative to the neighbourhood. Compare how crowded a row
neighbourhood is to how crowded its neighbours neighbourhoods are. A row in a
sparse region whose neighbours are also in a sparse region is normal for where
it is. This repairs the uneven density problem and is the right default for
most tables.
- the k nearest neighbours of row i
- how crowded row i neighbourhood is, roughly one over the average distance to its neighbours
- the score, about one for a normal row and above one for an unusual one
Reconstruction error. Reduce the data to a few directions as in the previous
lesson, then rebuild each row from those few numbers and measure how far the
rebuild is from the original. A row that the structure of the data cannot
express comes back wrong. This family finds something different from the other
two: not rows that are far away but rows that break the pattern, which is
often what fraud looks like, since a fraudulent transaction may be
unremarkable in every column and impossible in combination.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents