Basis Lab

Answers · How models work

Why can't I just measure accuracy on data that has no labels?

In shortThere is no accuracy to measure, because accuracy compares predictions with true answers and unlabelled data has none.

There is no accuracy to measure, because accuracy is a comparison between a prediction and a known answer, and the known answers are exactly what is missing. What can be computed on unlabelled data is a property of the method's own output, not a property of the data.

What a score would need

In a supervised problem some rows carry an answer yy, the model predicts y^\hat y, and accuracy is the fraction of rows where y^=y\hat y=y. Remove yy and the fraction has no numerator: there is no yy for y^\hat y to equal. A held-out set without labels is just more unlabelled data, so hiding rows changes nothing. Any number you compute is a function of the inputs alone.

Grouping makes the problem visible

Take four points on a line at 0, 2, 4 and 6. Ask a grouping method for four groups and it can put each point in its own group, giving a within-group spread of zero. Ask for two groups, {0,2}\{0,2\} and {4,6}\{4,6\}, with centres 1 and 5, and the spread is (0−1)2+(2−1)2+(4−5)2+(6−5)2=1+1+1+1=4(0-1)^2+(2-1)^2+(4-5)^2+(6-5)^2=1+1+1+1=4. Ask for one group with centre 3, and the spread is (0−3)2+(2−3)2+(4−3)2+(6−3)2=9+1+1+9=20(0-3)^2+(2-3)^2+(4-3)^2+(6-3)^2=9+1+1+9=20. The number fell from 20 to 4 to 0 without the data changing; it fell because more groups were asked for.

W=∑g=1k∑i∈gd(xi,cg)2W = \sum_{g=1}^{k} \sum_{i \in g} d(x_i, c_g)^2

The natural guess is to call that spread a score. It is not one. The method chose the centres cgc_g to make WW small, so quoting WW says the method did what it was asked, not that there was anything there to find. It also falls automatically as kk rises and reaches zero when every row is its own group. A chart of four groups looks the same whether the data had four real blobs or was an empty square.

Three failures, not one

Grouping fails because a group label has no true label to match. Compression fails for a different reason: reconstruction error measures how much of the input was kept, not whether the few numbers mean anything. Outlier flagging fails for a third reason: a flag is a ranking, not a prediction with a known truth. Set a threshold that flags the top 400 of 40,000 rows and you have flagged 400/40000=1%400/40000=1\% by construction, whether or not any of those rows are faults.

What can stand in for a score

Two kinds of evidence are honest because the method does not manufacture them. The first is stability. Refit on a different sample and see whether the same structure returns. If ten samples of 90 per cent of the rows give the same four groups in nine, that is 9/10=90%9/10=90\%; if they give it in three, that is 3/10=30%3/10=30\%. A division imposed on structureless data does not survive resampling, because nothing was determining where the boundaries fell.

The second is prediction of a held-out column. Keep one known column out of the grouping and ask whether the groups differ on it. If group A has 40 cancellations among 500 customers and group B has 120 among 500, the rates are 40/500=8%40/500=8\% and 120/500=24%120/500=24\%, a difference of 16 percentage points. That column was never available to the method, so the difference is evidence the grouping picked up something outside its own distance function.

Neither substitute is a proof. Stability can be stable and still wrong if the sample is biased, and a held-out difference can be caused by something other than the groups. But they are the evidence that would not appear on data with no structure. When no held-out column exists and resampling is impossible, there is no honest score left, and the result remains a description rather than a finding.

Worth keeping

  • Accuracy needs known answers; unlabelled data supplies none, so any number computed is a property of the method, not a check on the data.
  • Grouping, compressing and flagging fail for different reasons, but every one of them returns an answer on structureless data.
  • Stability under resampling and difference on a held-out column are evidence because neither is produced by the method itself.

Where this goes next

This question is one step of Finding Structure Without Labels, a course that works through its subject from the beginning. The page that teaches this part of it is Working Without an Answer Key, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. How many principal components should I keep?There is no single correct number: keep the smallest number whose combined variation you can afford to lose, and report the share you discarded.
  2. Why does k-means split one long stretched-out group into two?Because k-means keeps only one centre per group, so a long group's ends end up closer to a second centre and the group is cut across its length.
  3. How can I tell whether my clusters are real or just something the algorithm made up?Check whether clusters survive refitting on different rows and whether they predict a held-out column; a convincing 2D plot is weak evidence because such plots can invent clusters.
  4. How does DBSCAN decide that some points are noise and just leave them out?A point is left out when too few neighbours lie close to it and no dense point is close enough to attach it to a group.
  5. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  6. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.

Read alongside