Answers · How models work

How can a model remember a sequence without keeping every past input?

In shortA model can remember a sequence by keeping a fixed-size block of numbers and rewriting it at every step from its old value and the new input.

A model remembers by keeping a fixed-size block of numbers, called the state, and rewriting it at every step from its previous value and the current input. It does not keep the past inputs; anything older than the current input reaches the present only if it survived inside that block.

FIG 1The only inputs to one update

: the state after step t: a fixed block of numbers, the same size at every step
: the previous state, the only route by which anything older can arrive
: the current input, the one new piece of information at this step
: the update rule, the same function at every step and blind to how far into the sequence it is

There is no term for an input from ten steps ago and no term for how long the sequence has been running. The whole memory is the block on the left, and it is rewritten rather than extended.

The update, not the archive

The equation is the whole mechanism. At step one the model has some initial state $h_0$ , often all zeros. At step two it combines $h_1$ with $x_2$ ; at step three it combines $h_2$ with $x_3$ . The input $x_1$ never appears again. If any trace of $x_1$ is still present at step three, it is present because $h_2$ carried it and $h_3$ folded $h_2$ forward. There is no separate archive to consult, and no slot in the block that belongs to $x_1$ . The rule $f$ also does not see the step number. That is not a simplification: a rule that depended on position would have nothing to say about position ten thousand if it had only been trained up to position five hundred.

A scalar update makes the fading visible. Take $f(h,x)=0.5h+x$ and start from $h_0=0$ . After the first input the state is exactly the first input. After the second it is half the first input plus the second. After the third it is a quarter of the first, a half of the second, and all of the third.

h_1=x_1,\qquad h_2=\tfrac12 x_1+x_2,\qquad h_3=\tfrac14 x_1+\tfrac12 x_2+x_3.

The coefficient on $x_1$ after $t$ steps is $1/2^{t-1}$ . At $t=10$ that is $1/512$ , which is about $0.00195$ ; at $t=20$ it is $1/524288$ , about $0.0000019$ . The old value is not erased in one step. It is multiplied by one half at every step, so it shrinks geometrically while the block keeps the same size. This is the behaviour that becomes a serious problem over hundreds of steps: the state can still contain something of the first input, but that something is tiny compared with what arrived recently.

The size is the limit

The block has a fixed number of slots, so handling one more item costs the same at item 10 and at item 10000. If the state has 512 numbers, one update reads 512 and writes 512, about 1024 number operations. Ten updates cost $10\times1024=10240$ operations; ten thousand updates cost $10000\times1024=10240000$ operations. The memory kept is 512 numbers in both cases. A record of every input of the same width is different: after 10 items it holds $10\times512=5120$ numbers; after 10000 items it holds $10000\times512=5120000$ numbers. The ratio between those two lengths is $10000/10=1000$ , so the growing record is a thousand times larger at the longer length, while the state is unchanged.

The natural guess is that memory means storing the inputs themselves. That guess fails for two separate reasons. A list of inputs is not fixed-size, so if the model compares the current item against every stored item, the work per new item grows with the length, and the storage grows without bound while a long response is generated. If instead you truncate the list to the last $k$ items, you have a window, not a summary: anything older than $k$ is gone completely, and $k$ is still a fixed choice. The state does something different: it compresses every past item into the same block at the moment it arrives. That is why you cannot look at a trained state and read off what the third sentence said. The third sentence influenced the block, and the fourth sentence influenced it again on top of that, and there is nowhere else to look.

Where it stops being true

The fixed size is also the hard limit. A summary of a thousand items and a summary of ten items occupy the same space, so something must be discarded, and which things get discarded is the design problem. If the question is roughly what the document was about, a fixed summary is the right shape. If the question is what number appeared in the fourth line of a table nine thousand words ago, a summary is the wrong shape: at the time that number appeared, it was competing with everything else for a fixed amount of room and was never going to earn enough of it. Systems that need both usually mix the two approaches, carrying a fixed state through most of the work and looking back over a short recent window where exact recall of the recent past matters. The cost curve stays flat for the bulk, and exactness is bought only where it is needed.

Worth keeping

A fixed-size state makes the cost of each new item independent of how many items came before, because the block is rewritten rather than extended.
The price is that a summary of ten items and a summary of a thousand items occupy the same space, so exact recall of an arbitrary earlier item is not available.
Whether that trade is acceptable depends on the task: gist tasks suit a summary, exact lookup tasks do not.

Where this goes next

This question is one step of Models That Carry a Memory, a course that works through its subject from the beginning. The page that teaches this part of it is A Fixed Summary Carried Forward, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course