Why does my test set stop being a fair test if I keep tuning on it?

That is $0.0095$ as a proportion, or $0.95$ percentage points. Now train fifty variants that are all truly equal, so each has the same true error rate of 10 percent. Their measured scores scatter around 10 percent with a standard deviation of about 0.95 points. The best of fifty noisy measurements is roughly 2.8 standard deviations below the mean, because picking the minimum of many noisy numbers picks a lucky one. The arithmetic is $2.8\times 0.95 = 2.66$, and $10 - 2.66 = 7.34$. You report about 7.3 percent, but the truth is 10 percent. The 2.7 point gap is not a training effect. It is a selection effect.

The natural guess is that the test set is safe because no weight was ever updated on it. That is the wrong condition. The condition is that the held-out examples influenced nothing: not the weights, not which features were built, not which model was chosen, not when training stopped. A decision made by comparing scores on the test set lets information from those examples into the model. Each comparison is weak fitting, and a sequence of comparisons is stronger fitting. The test set has become a second validation set, whatever it is called in the code.

One look at the sealed set is defensible. Three looks with the reasons written down is defensible. Thirty looks means the final number should be described as a validation score, not as an estimate of performance. The effect is small for a handful of comparisons and large for the hundreds that accumulate in a real project. It is the ordinary explanation for a long run of benchmark improvements that fails to reproduce elsewhere. The discipline costs nothing and prevents the most common form of unreproducibility, which is not fraud but many small choices each made honestly against the same examples.

There is a second limit that no amount of sealing removes. The estimate has uncertainty, and that uncertainty shrinks only with the square root of the number of held-out examples. At $p=0.1$ and $n=100$, the same formula gives $\sqrt{0.09/100}=0.03$, or 3 percentage points. At $n=1000$ it is 0.95 points. To halve the uncertainty again you need four times as many examples, so $n=4000$ gives $\sqrt{0.09/4000}=0.0047$, or about 0.47 points. A small held-out set cannot distinguish a 10 percent system from a 9 percent system, no matter how carefully it is sealed.

The fairness also depends on splitting the right unit. If several rows describe the same patient, user or document, a random row split puts parts of one group on both sides. The model then recognises the group rather than learning the task, and the held-out score measures recognition. Splitting by group fixes that. For forecasting, a random split lets the model see later examples while being tested on earlier ones, which is a form of knowing the future; split by date instead. These failures are separate from tuning on the test set, but they break the same promise: the held-out examples must have influenced nothing.

Write down before the project starts how many times the sealed set may be opened. Once is ideal. Three times with recorded reasons is defensible. Thirty times means the final number is a validation score and not an estimate of performance, because that is what it has become.