An Unbiased Estimate That Is Almost Useless
Last timeA Gradient Through a Sample
The sampled policy gradient is correct on average and still barely usable, because its noise grows with the length of the run, with the scale of the rewards and with how rarely anything is earned.
There is now a gradient that can be computed from sampled runs. It is correct on
average. It is also, used directly, close to unusable, and understanding exactly
why is what makes every later method look inevitable rather than clever.
- how many runs were collected before an update
- what the ith run earned after step t, the weight on that action
- the direction that makes that action more likely
Unbiased, and what that does not mean
Unbiased means that if this were computed infinitely many times and averaged, the
result would be the true gradient. Nothing in that sentence says anything about
what happens with thirty-two runs.
| step | run | return | weight on every action in that run | what happened |
|---|---|---|---|---|
| 1 | 1 | 0 | 0 | Nothing earned. Every action in this run is pushed by zero, so the run taught nothing at all. |
| 2 | 2 | 0 | 0 | Again nothing. Two thirds of the batch has contributed nothing to the update. |
| 3 | 3 | 1 | 1 | One run found something. The entire update is now whatever this single run happened to do, including everything irrelevant it did along the way. |
Where the spread comes from
Three independent sources of randomness enter the single number each action is
weighted by. The action draws themselves, since the policy is a distribution. The
environment's answers. And the rewards, when they are noisy. These do not cancel;
they accumulate along the run, so the spread of the weight grows with how long the
run is, even when the task is behaving perfectly.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents