Judging a Result With No Truth to Check

Last timeFlagging the Odd One Out

Two tests carry almost all the honest evidence available here, whether the result survives being refitted on different rows and whether it predicts something deliberately kept out of it.

The lesson this course exists for

Seven lessons of methods, and every one of them returns an answer whatever the

data. So the last question is the only one that was ever difficult: given a

result, with no truth to check it against, what can honestly be said?

Three things, and this lesson is all three. Two of them are tests and the

third is a warning about the evidence everybody finds most convincing.

Does it survive being refitted

Take nine tenths of the rows at random. Run the whole method. Do it again with

a different nine tenths. Compare the two results.

The comparison needs a measure, and the usable one is pairwise. Take every

pair of rows that appears in both samples, and ask whether the two runs agreed

about them: either both put the pair together, or both put them apart.

FIG 1Agreement between two runs, corrected for chance

: the share of pairs the two runs agreed about
: the share they would agree about by chance, given how many groups each found and how large
: the corrected agreement, one for identical and about zero for unrelated

The correction matters more than it looks. With two groups of equal size, two unrelated groupings already agree about half the pairs, so a raw agreement of 0.5 is no agreement at all. With ten groups the chance level is above 0.8, because most pairs are apart in both runs and that counts as agreement. Always use the corrected figure, which has chance at zero wherever the grouping sits.

FIG 2Twenty resamples of four data sets

step	the data	groups asked for	average corrected agreement	verdict
1	four separated blobs	4	0.94	real
2	four separated blobs	7	0.52	too many groups asked for
3	two overlapping blobs	2	0.71	real but the boundary is arbitrary
4	points spread evenly, no groups	4	0.11	imposed
5	one long band of points	3	0.08	imposed

5 steps

The last two rows are the ones that matter, since both produced four tidy regions and a confident chart. Their agreement sits barely above chance because nothing in the data determined where the boundaries went, so each refit put them somewhere else. The second row shows the same test diagnosing a number of groups that was too high, with the four real groups stable and the three extra splits landing differently every time.

Two cautions keep this honest. A grouping can be stable and meaningless: if

the data contains two obvious groups and somebody asks for two, the answer is

stable whether or not two is the interesting division, so stability confirms

that a structure is determined by the data and not that it is the structure

worth having. And a very unstable result is more informative than a stable

one, because it is a definite finding that the question was wrong.

The lesson stops here

6 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Judging a Result With No Truth to Check

The lesson this course exists for

Does it survive being refitted

6 more paragraphs to go

This is the reading half

Read alongside