Answers · The mathematics underneath
How do I estimate the error of a statistic when I don't have a formula for it?
In shortResample your own data many times, recompute the statistic each time, and use the spread of those results as the error bar.
Resample your own measurements with replacement, recompute the statistic on each resample, and use the spread of those recomputed values as the error bar. No formula for the statistic is needed; the ingredients are the data and a way to recompute the number.
The procedure on a real sample
Take 500 measured response times whose median is 214 milliseconds. Draw a new set of 500 from those 500 at random, with replacement, so a value can be chosen more than once. The chance that any particular measurement is never chosen in one draw is ; over 500 independent draws it is , which is about , so roughly 36.8 per cent of the original values are absent from a given resample and some others appear twice. Recompute the median on that resample. One resample gives 209 ms; another gives 221 ms.
| step | step | what happens | result | what happened |
|---|---|---|---|---|
| 1 | start | a sample of 500 measured response times | median is 214 ms | The only data used from here on. |
| 2 | resample | draw 500 of them at random, with replacement | some appear twice, about a third never appear | With replacement is the point, not a detail. |
| 3 | recompute | take the median of the resampled set | 209 ms | A different sample gives a different median. |
| 4 | again | a second resample and median | 221 ms | The movement between resamples is the raw material. |
| 5 | repeat | two thousand times in all | two thousand medians, spread from about 198 to 231 | The spread of these medians is the error bar. |
| 6 | sort and read off | the 2.5 and 97.5 per cent points of those medians | 203 ms to 226 ms | This is the interval. |
| 7 | report | the median with its interval | 214 ms, between 203 and 226 | No formula for a median appears anywhere. |
Do this 2000 times. You now have 2000 medians, spread from about 198 ms to 231 ms. Sort them and read off the 2.5 per cent and 97.5 per cent points: 203 ms and 226 ms. Report the median as 214 ms with interval 203 to 226 ms. The 198 to 231 range is the raw spread of all 2000 medians; the 203 to 226 interval is the central 95 per cent of that spread.
The logic is that the 500 measurements are a stand-in for the population. Drawing from them with replacement imitates drawing a fresh sample of 500 from that population, with the important difference that the imitation can only use values already seen. Repeating the imitation many times shows how much the median moves from sample to sample. The spread of those moves is the error bar, and it is measured in the same units as the statistic.
The natural guess is to use the formula for the standard error of a mean, perhaps after checking that the median is close to the mean. That formula needs the standard deviation and a sampling distribution, and for a median the textbook formula assumes a symmetric shape such as the normal. Response times are skewed, so the formula is available and wrong. For a ninety fifth percentile, a ratio of counts, or a measure defined by a function written this morning, there is no usable formula at all. Another tempting shortcut is to resample without replacement. That returns the same 500 values in a different order, so every recomputed median is identical and the spread is zero, which is not an error bar.
What it assumes
The interval is only as good as the sample. The 500 measurements must stand in for the population; if they were collected from one busy hour, the resampled medians inherit that narrow view, and a biased sample produces a confident interval around the wrong number. The measurements must also be independent, or the resampling must follow the groups they arrive in. Requests within a session are not independent: ten requests from one session are closer to one another than ten requests from ten sessions. If you have 50 sessions with 10 requests each, resampling individual rows treats all 500 rows as fresh information and gives an interval that is far too narrow. Resample whole sessions instead, keeping each session's ten requests together, and the interval widens to reflect the real uncertainty.
Statistics that depend on the extremes are outside the method. A resample is built only from values already observed, so it can never contain a response time larger than the largest one in the original 500. If the statistic is the maximum, every resampled maximum is at most the observed maximum, and the method has no way to say how much larger next week's maximum might be. The same caution applies to a ninety ninth percentile or any quantity driven by the tail beyond the data.
How many more samples would halve the error? The width of a resampling interval shrinks roughly as , where is the number of original measurements. To halve the half-width from 11.5 ms to 5.75 ms, you need four times as many measurements: . Resampling more times from the same 500 does not narrow the interval; it only makes the 2.5 and 97.5 per cent points steadier. The data, not the number of resamples, sets the width.
Worth keeping
- Drawing with replacement is essential: without it every resample is just a reordering of the same values, so the recomputed statistic never moves.
- A resampling interval cannot repair a biased sample; it only shows how much the statistic would move if the same sampling process were repeated.
- For a statistic that depends on the largest observed value, resampling cannot say how much larger a future value might be, because no resample can contain anything beyond the data already seen.
Where this goes next
This question is one step of Answering With Random Samples, a course that works through its subject from the beginning. The page that teaches this part of it is Using the Data You Have Twice, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- When should I give up on finding an exact formula and simulate instead?Give up on an exact formula when the quantity can be written as an average of something drawable, since simulation then gives an estimate with a measurable error.
- Why does the average of many random samples get closer to the true value as I take more samples?Because the average of independent draws is centred on the true value at every sample size, and its typical distance from that value shrinks like one over the square root of the number of draws.
- Why does cutting the error in half require four times as many samples?The error shrinks in proportion to one over the square root of the sample count, so halving it needs four times the samples.
- What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
- How do you make a vague claim precise enough to tell whether it's true?Fix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.
- How do you find the parabola that best fits a function near a point?Match the function's height, slope and bending at the chosen point; those three conditions single out exactly one parabola.