Using the Data You Have Twice

Last timeRunning the Whole Process

There is no formula for the error bar on most statistics worth computing. Resampling your own measurements supplies one, using nothing but the data already collected.

A team reports that the median response time is 214 milliseconds. How much

would that number move if the same week were lived again?

There is a formula for the error on an average. There is no usable one for a

median, and none at all for most of what gets reported.

FIG 1Common reported quantities, and what is available for each

	textbook formula exists?	holds on real data?	resampling works?
the average of a sample	1	1	1
the median	1	0	1
the ninety fifth percent	0	0	1
the ratio of two measure	0	0	1
the gap between two grou	0	0	1
the largest value seen	0	0	0

The first marked cell is the quiet problem: a formula for the median does exist and it assumes a shape that response times never have, so it is available and should not be used. The last row is marked because it is the one case resampling genuinely cannot handle, and the reason is worth holding onto: a resample is built from values already observed, so it can never contain anything larger than the largest one already seen, and the method has no way to say how much bigger next week's maximum might be.

FIG 2An interval for a median, built from the data alone

step	step	what happens	result
1	start	a sample of 500 measured response times	median is 214 ms
2	resample	draw 500 of them at random, with replacement	some appear twice, about a third never appear
3	recompute	take the median of the resampled set	209 ms
4	again	a second resample and median	221 ms
5	repeat	two thousand times in all	two thousand medians, spread from about 198 to 231
6	sort and read off	the 2.5 and 97.5 per cent points of those medians	203 ms to 226 ms
7	report	the median with its interval	214 ms, between 203 and 226

7 steps

Only the original 500 measurements are ever used, and no formula for the median appears anywhere, which is the whole appeal: the same seven steps produce an interval for any statistic at all, including one defined by a function somebody wrote this morning. Drawing with replacement is essential rather than incidental, since drawing without it would return the same 500 values in a different order and every recomputed median would be identical. The two thousand only controls how smooth the final interval is; its width comes from the data.

FIG 3Resampling, and the version that resamples whole groups

python

import numpy as np
rng = np.random.default_rng(2)

times = rng.lognormal(5.3, 0.6, 500)                  # a skewed sample of 500
boots = [float(np.median(rng.choice(times, 500, replace=True))) for _ in range(2_000)]
print(round(float(np.median(times))), np.round(np.percentile(boots, [2.5, 97.5])))

sess = np.repeat(np.arange(50), 10)                   # 50 sessions, 10 requests each
vals = np.repeat(rng.lognormal(5.3, 0.6, 50), 10) * rng.lognormal(0, 0.1, 500)

rows = [float(np.mean(rng.choice(vals, 500, replace=True))) for _ in range(2_000)]
grps = [float(np.mean(vals[np.isin(sess, rng.choice(50, 50, replace=True))]))
        for _ in range(2_000)]
print(np.ptp(np.percentile(rows, [2.5, 97.5])), np.ptp(np.percentile(grps, [2.5, 97.5])))

The first block builds an interval for a median from a skewed sample, needing nothing but the measurements. The second block is the important correction: when measurements arrive in groups, such as several requests from the same session, resampling individual rows pretends each row is fresh information and produces an interval far too narrow. Resampling whole sessions instead keeps the dependence intact. The two printed widths differ by a large factor on the same data, and the wider one is the honest one.

What it assumes

The sample has to stand in for the population, which it does badly when it is

very small. Resampling thirty measurements gives an interval, and that

interval inherits every peculiarity of those thirty.

The measurements also have to be independent, or the resampling has to respect

whatever groups they fall into. This is the failure that occurs most often in

practice, because data collected from real systems almost always arrives in

clusters: requests within a session, measurements within a user, trades within

a day.

And statistics that depend on the extremes are outside the method entirely.

The lesson stops here

1 more paragraph to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Using the Data You Have Twice

What it assumes

1 more paragraph to go

This is the reading half

Read alongside