A Rule That Reads What Arrived Before Deciding
Last timeA Linear Recurrence Done Properly
A fixed decay rate treats every token the same, so the state fills with filler. Letting the current input set the keep and write amounts turns the state into something selective.
Everything so far treats the input as something to be absorbed. The slot keeps
its fixed fraction and adds what arrived, at every position, with no regard for
what arrived. That is a strange way to run a limited memory, and it is the last
thing standing between a fixed state and a useful one.
The problem with a constant rate
Consider a model holding one fact through a passage. The fact arrives, the slot
takes it, and then four hundred words of unremarkable text go by. Each of those
words costs the slot the same fraction of its contents as the fact did, because
the rate does not know the difference. By the time the fact is needed it has
been diluted four hundred times by material that mattered less than it did.
| step | step | how much the step mattered | keep factor actually used | share of the fact still held | what happened |
|---|---|---|---|---|---|
| 1 | 1 | 1 | 0.99 | 1 | The fact arrives and is written in. |
| 2 | 50 | 0.1 | 0.99 | 0.61 | Fifty steps of low-value text have each taken a hundredth, and nearly forty percent is gone. |
| 3 | 200 | 0.1 | 0.99 | 0.13 | The keep factor has not changed, because nothing about it can change. |
| 4 | 400 | 0.9 | 0.99 | 0.02 | The question finally arrives and the fact is effectively gone. Nothing was wrong with the rate; it was simply applied to the wrong things. |
This is not a small effect that shows up only on contrived tasks. Natural text is
enormously uneven in how much each token is worth remembering: a name, a number
or a negation can change the meaning of everything that follows, while long
stretches of connective material can be dropped with no loss. A rule that
treats those the same is throwing away most of the capacity it has. The models
in the previous lesson were already good at holding things for a long time; what
they could not do was decide which things.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents