Answers · How models work

Why is the REINFORCE gradient so noisy even though it is unbiased?

In shortBecause unbiased only fixes the average, while one noisy return is multiplied by every action's score, so the estimate's spread grows with episode length and reward scale.

Unbiased means that if the same update were computed from infinitely many independently collected batches and averaged, the average would be the true gradient. The noise is the spread of one finite batch around that average, and in REINFORCE it is large because one sampled return multiplies the score direction of every action in the run, so the randomness of the whole episode is carried into every term of the sum.

\hat{g} = \frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^{T} G_i^{(t)} \, \nabla_\theta \log \pi_\theta(a_t^{(i)}\mid s_t^{(i)})

Unbiased is a statement about the mean

The estimate is an average of \(N\) runs. Each run contributes a sum over time of the return after step \(t\) times the score of the action taken at step \(t\). The return is not a fixed number: it contains the action draws, the environment's transitions and the rewards. A single run therefore injects the same episode-level randomness into every action that run took, and different runs disagree about the whole sum, not just about one term.

G_i^{(t)}=\sum_{k=t}^{T} r_k,\qquad \operatorname{Var}(G_i^{(t)})=\operatorname{Var}\left(\sum_{k=t}^{T} r_k\right).

If the per-step rewards were independent with variance 1, then at \(t=1\) a 20-step run has variance 20 and standard deviation \(\sqrt{20}\approx4.47\), while a 100-step run has variance 100 and standard deviation 10. Put a score component with standard deviation 0.1 against those returns. At the first step of a 20-step run the weighted term has standard deviation about \(4.47\times0.1=0.447\) from the return alone. At the first step of a 100-step run it is \(10\times0.1=1\), more than twice as large. The sum over \(t\) adds more such terms, so the single-run estimate spreads over a range comparable to the total return, not to the small differences between good and bad actions.

The natural guess that fails

The natural guess is that unbiasedness means the estimate is close, or that collecting more runs will repair it. Neither follows. Averaging \(N\) independent runs divides the standard deviation by \(\sqrt N\), so going from 1 run to 4 halves the noise, but going from 100 to 400 also only halves it at four times the cost. If the initial noise is five times the signal, reaching noise equal to the signal needs \(5^2=25\) times as many runs, and reaching a tenth of the signal needs \(50^2=2500\) times as many. That is a structural problem, not a sample-size problem.

\operatorname{sd}\left(\frac{1}{N}\sum_{i=1}^{N}X_i\right)=\frac{\sigma}{\sqrt N}.

Add 10 to every reward over a 20-step run. The true gradient cannot change, because adding a constant to every reward cannot change which policy is better. But the return at the first step moves from a typical 2 to \(2+10\times20=202\), while the spread between runs stays at 2. The signal is the same difference between runs; the zero-mean part multiplying each score has been multiplied by 100.

FIG 1The same task with ten added to every reward, over twenty steps

	typical weight on an act	spread between runs
as given	2	2
every reward raised by t	202	2

Adding ten to every reward over twenty steps moves the typical weight from 2 to 202 and leaves the run-to-run spread at 2, so the signal is the same difference and the zero-mean part multiplying each score is a hundred times larger.

\mathbb E_{a\sim\pi_\theta(\cdot\mid s)}\left[\nabla_\theta\log\pi_\theta(a\mid s)\right]=\sum_a \nabla_\theta \pi_\theta(a\mid s)=\nabla_\theta 1=0.

That identity is why the added term is pure noise on average. For a fixed state, the score direction averages to zero over the policy's own actions, so the extra \(cH\) times the score contributes nothing to the mean. Its variance does not vanish. With \(c=10\) and \(H=20\), \(cH=200\). If one score component has variance 0.01, the added term has variance \(200^2\times0.01=40000\times0.01=400\), so its standard deviation is 20. The original weight 2 gives variance \(2^2\times0.01=0.04\) and standard deviation 0.2. The ratio of standard deviations is \(20/0.2=100\). Unbiased, and a hundred times noisier.

Where the guarantee ends

Unbiasedness here assumes the runs are sampled from the current policy, the returns are computed for the actions actually taken, and any subtracted quantity does not depend on the action. A state-dependent baseline independent of the action keeps the mean and can remove the constant-shift noise; an action-dependent baseline, a truncated return without correction, or an off-policy batch can bias the estimate. Discounting changes the size of the horizon term but not the argument: the estimator's variance still grows with the number of random rewards it multiplies into the score.

Worth keeping

Unbiasedness fixes the mean of an estimator, not its spread, and REINFORCE has a spread that grows with the length of the run.
Adding the same constant to every reward leaves the true gradient unchanged but can multiply the zero-mean part of the estimate by that constant times the horizon.
Averaging N runs cuts noise only by the square root of N, so halving noise costs four times the data.

Where this goes next

This question is one step of Learning From Reward, a course that works through its subject from the beginning. The page that teaches this part of it is Why the Estimate Is So Noisy, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course