How can I tell whether my clusters are real or just something the algorithm made up?

Check whether the clusters survive being refitted on different rows, and whether they predict a column you deliberately kept out. Those are the only two tests that give honest evidence when there is no true labelling to compare against. A convincing two-dimensional plot is the weakest evidence available, because the methods that draw such plots can produce separated islands from data that contains no groups at all.

Take nine tenths of the rows at random, run the whole clustering method, and record which pairs of rows end up in the same group. Do this again with a different nine tenths. For every pair of rows that appears in both samples, ask whether the two runs agreed: either both put the pair together or both put them apart. The share of pairs that agree is $a$. Raw agreement is misleading because two unrelated groupings already agree about half the pairs when there are two equal groups. Correct for chance using the formula below, where $e$ is the agreement expected by chance given the number and sizes of groups in each run.

With two groups of 50 rows each from 100 rows, the probability that two distinct rows fall in the same group is $2450/4950 \approx 0.495$, since there are $2 \times \binom{50}{2} = 2450$ within-group pairs out of $\binom{100}{2} = 4950$ total pairs. If two independent runs both use two equal groups, the chance agreement is $0.495^2 + 0.505^2 \approx 0.5$. A raw agreement of 0.5 then gives a corrected agreement of $(0.5 - 0.5)/(1 - 0.5) = 0$, which is no agreement at all. Always use the corrected figure. A value near one means the two runs are essentially identical; a value near zero means they are unrelated.

The last two rows are the ones that matter. Both produced four tidy regions and a confident chart, yet their agreement sits barely above chance because nothing in the data determined where the boundaries went; each refit put them somewhere else. The second row shows the same test diagnosing too many groups: the four real groups are stable, but the three extra splits land differently every time. A very unstable result is more informative than a stable one, because it is a definite finding that the question was wrong. A stable result can still be meaningless: if the data contains two obvious groups and somebody asks for two, the answer is stable whether or not two is the interesting division.

The natural guess is to judge the grouping by how convincing it looks on a two-dimensional plot. This fails because the neighbourhood preserving methods used to draw such plots are built to pull near points together and push far points apart, which produces separated islands on data with no group structure whatsoever. Run one on points scattered uniformly in a cube and it will return a picture of distinct islands, every time. Three things in that picture are artefacts rather than findings: the sizes of the islands, which depend on the settings and not on how many points are in each; the gaps between them, which carry no information about how different the groups are; and the positions, since two islands side by side are no more related than two across the page.

The stronger test uses information the method never had. Choose a column that is known for each row and deliberately exclude it from the clustering. Run everything. Then ask whether the resulting groups differ on that column. A grouping built from purchase history alone, whose groups turn out to have cancellation rates of three, eleven and thirty-one per cent, has found something real about the customers. The spread is $31 - 3 = 28$ percentage points. No internal measure can produce that conclusion and no amount of chart polishing can fake it.

The second row is the honest negative result: five stable groups, cleanly separated, that predict nothing anybody cares about. The grouping is real and useless. That distinction is invisible to every measure in the earlier lessons. The held-out column is the only test that uses information the method could not see, so it cannot be gamed by the method.