The Value of Being Somewhere, and What It Must Agree With
Last timeThe Number Being Made Large
The value of a state is the average return from there onwards, the value of an action is the same thing after one forced choice, and both must agree with the values of whatever follows them.
The last lesson left one number for a whole run and no way to say which decision
earned it. Before that can be fixed, there has to be a way of saying what a
position is worth, so that a decision can be judged against what was already
expected rather than against nothing.
- the value of state s under the policy pi
- the discounted return collected from that point onwards
- the policy that keeps acting for the rest of the run
The value of a state
A position is worth what tends to follow from it. Not what did follow once, which
is a sample of this quantity and can be wildly off it, but the average over
everything that could follow.
- the value of taking action a in state s and continuing with pi
- the first action is forced to be a, whatever the policy would have preferred
The value of a state and one decision
The relationship between the two is immediate: the value of a state is the
average of the action values over whatever the policy would have done there. If
every action in a state is worth the same, the policy has nothing to learn in
that state, however large the numbers are.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents