Answers · How models work

Why does the value of a state have to satisfy a recursive equation with the next state's value?

In shortA state's value equals the reward from the next step plus the discounted value of wherever that step lands, because every possible future divides that way and averaging over the first step leaves exactly the next state's value.

The value of a state must satisfy $V^\pi(s)=\mathbb{E}[r+\gamma V^\pi(s')]$ because a value is an average over futures, and every future splits into a first reward and a discounted remainder that is itself the value of the state it lands in. This is not an extra rule bolted onto the definition: it is what the definition becomes once you group the futures by their first step. A state is worth what tends to follow from it under the policy, so if you know what tends to follow from the places it can lead to, you already know what tends to follow from it.

V^\pi(s) = \mathbb{E}\left[ r + \gamma V^\pi(s') \right]

Splitting one step off

Write a single future as a list of rewards, $r_1, r_2, r_3, \ldots$ , with discount $\gamma$ . The return is $G = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdots$ . The first reward is $r$ , and the rest is $\gamma$ times a return that begins in the next state: $G = r + \gamma G'$ . Take the expectation conditional on the starting state $s$ . By the law of total expectation, average first over the first step and then over what follows. For a fixed next state $s'$ , the expected value of $G'$ is exactly $V^\pi(s')$ , because that is what the value of $s'$ means. So the inner average collapses to $V^\pi(s')$ and only the first step's randomness remains.

V^\pi(s) = \mathbb{E}[G \mid s_1=s] = \mathbb{E}[r+\gamma G' \mid s_1=s] = \mathbb{E}\left[r+\gamma \mathbb{E}[G' \mid s_2=s'] \mid s_1=s\right] = \mathbb{E}\left[r+\gamma V^\pi(s') \mid s_1=s\right].

A number you can check by hand

Take $\gamma=0.9$ and a state $s$ with two equally likely first steps. With probability $0.5$ the step gives reward $2$ and lands in a state whose value is $10$ . With probability $0.5$ it gives reward $0$ and lands in a state whose value is $4$ . The first branch is worth $2+0.9\times 10=2+9=11$ from $s$ , and the second is worth $0+0.9\times 4=0+3.6=3.6$ . Averaging the two branches gives $0.5\times 11+0.5\times 3.6=5.5+1.8=7.3$ . So $V^\pi(s)=7.3$ . If you tried to use $r+V^\pi(s')$ without the discount you would get $2+10=12$ on the first branch and $0+4=4$ on the second, average $8$ , which double counts the timing: the $10$ and the $4$ are already sums measured from the next step, so they must be pulled back one step by multiplying by $0.9$ .

Why the natural guess fails

The natural guess is that the recursion is an assumption about how values behave, or a computational trick that happens to work when the tables are small. It is neither. A value is defined as an average over complete futures, and the recursion is that same average rewritten by splitting every future at its first step. If a proposed value failed the equation, it would be because the average of the tail after the first step was not the value of the next state, which would mean the next state's value had been computed under a different policy or from a different starting point. The equation is a consistency test on a whole table of values at once, not a definition of any one entry.

The factor $\gamma$ is there because the next state's value is measured from the next step. If $\gamma=0.9$ , one unit of return one step later is worth $0.9$ units now, so a next-state value of $10$ contributes $9$ to the current state. Setting $\gamma=1$ removes that pull-back and the equation becomes $V^\pi(s)=\mathbb{E}[r+V^\pi(s')]$ , which is valid whenever the undiscounted return is finite. Setting $\gamma=0$ gives $V^\pi(s)=\mathbb{E}[r]$ , because nothing after the first step counts. The recursion does not say the future is ignored; it says the entire future after the first step has been compressed into one number, and that number arrives one step late.

Where it stops being true

The equation belongs to a fixed policy. Change $\pi$ and the same state has a different value, so a table learned under one policy is not a table for another. It also assumes the next state carries everything relevant about the future. If two histories arrive in the same nominal state but lead to different reward distributions, the one-step recursion fails until the state is enlarged to include the missing history. With $\gamma=1$ the identity still holds for finite returns, but the average can diverge, and then the value is infinite rather than a number to solve for. The same splitting argument gives the action value version, $Q^\pi(s,a)=\mathbb{E}[r+\gamma V^\pi(s')]$ , with the first action forced.

Worth keeping

The value of a state is an average over futures under a fixed policy, not a property of the state alone.
The Bellman equation is a consistency condition: the average return from a state equals the expected first reward plus the discounted average return from the next state.
Replacing a waited-for return with a bootstrapped next-state value trades the noise of a long sum for the bias already present in the estimate.

Where this goes next

This question is one step of Learning From Reward, a course that works through its subject from the beginning. The page that teaches this part of it is What a Position Is Worth, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course