Should my evaluation cases be real user queries or ones I write myself?

Real queries from your own logs, sampled by behaviour rather than at random. Cases you write yourself are a sample of your imagination of your users, and they will flatter the system precisely where it is weakest.

Take a log where 70 per cent of requests are one simple intent that every version already handles, and 2 per cent are a hard intent where the versions you are choosing between actually differ. Sample 300 uniformly and you spend $0.70 \times 300 = 210$ cases on the easy intent and $0.02 \times 300 = 6$ on the hard one. Weight each group by how often the versions disagree, 2.1 per cent in the easy group and 24.5 per cent in the hard one: $210 \times 0.021 = 4.4$ informative cases against $6 \times 0.245 = 1.5$. The comparison rests on about six cases in total, and six cases will report no difference whatever the truth is.

The natural guess runs the other way: write the cases yourself and you get clean, unambiguous, correctly spelled queries, each with a known right answer. But a query with no typo, no missing context and no half-finished sentence is not a harder version of a user query, it is a different distribution. Where the failures live in the mess, in a pronoun with no referent or a request that changes topic mid-sentence, the hand-written set scores well because you removed the thing that breaks the system. Cleanliness is a property of the set, not of the product, and the score has no idea which one it is measuring.

The fix is equal quotas, not proportional ones. Sixty per group instead of the traffic share turns 1.5 informative adversarial cases into $60 \times 0.245 = 14.7$, and the rare group starts to carry the comparison. Over-sampling a group does not distort anything provided you report two numbers from the one run: the per-group scores, which say where a change landed, and the aggregate reweighted back to real traffic shares, which estimates the user experience.

Here $w_g$ is the traffic share of group $g$ and $p_g$ its score in the set. With per-group scores of 0.95, 0.90, 0.85, 0.80 and 0.75 and the traffic shares above, $0.70 \times 0.95 = 0.665$, $0.14 \times 0.90 = 0.126$, $0.09 \times 0.85 = 0.077$, $0.05 \times 0.80 = 0.040$, $0.02 \times 0.75 = 0.015$, and the total is 0.923. The unweighted average of the five, 0.85, would be a number no user experiences.

Size follows from the difference you intend to detect. The standard error on a score near 50 per cent is $0.5/\sqrt{n}$. At 100 cases that is $0.5/10 = 5$ points, and the error on a difference between two versions is $\sqrt{2}$ times it, 7.1 points, so a four-point improvement is invisible. At 500 cases it is $0.5/22.4 = 2.2$ points each and 3.2 on the difference, comfortably under five. At 4000 it is $0.5/63.2 = 0.8$ points each and 1.1 on the difference, which is what seeing a one-point change costs; at that price the honest move is a paired comparison on the same cases rather than a bigger set.

Traffic has blind spots. It contains nothing for a capability shipped yesterday, nothing for a request nobody has thought to make yet, and nothing for the safety case that must never happen. So a minority of the set is deliberately written by hand, each case carrying a note of what it is for, reported as its own group and never weighted as though it came from the user distribution. That is different from writing the whole set. Real logs also give you inputs, not answers, so every case still needs a label, and labelling difficulty is a reason to write a case, not a reason to replace the sample with your own.

Two slow failures arrive once the set exists. People tune against it, not dishonestly but because a visible number does that to anyone improving something, so reserve a quarter that nobody reads: 75 cases out of 300, scored separately. When the visible score rises and the held-out one does not, the gain was fitting. And traffic drifts, so a set assembled in January measures a product that no longer exists by June; replace about a tenth each month, 30 of 300, from recent traffic, keeping the group quotas, so the set stays current and stays comparable.