Slots That Forget at Rates You Chose
Last timeThe Sequential Bottleneck
Once the update is linear, the carried state can be given one independent decay rate per slot, and spreading those rates across many timescales is what makes the memory work.
The previous lesson removed the squash from inside the loop to make the training
cheap. That looks like a sacrifice. It turns out to be an opportunity, because
once the update is linear you can say exactly what the carried state is doing,
and then arrange for it to do something sensible.
Making the slots independent
A general linear update lets every slot of the state read every other slot. That
is more freedom than it sounds, and almost all of it is redundant, because a
mixing matrix can usually be rewritten as a change of coordinates around a
matrix that does not mix at all. Taking the non-mixing form directly is both
cheaper and easier to reason about.
- the carried state, a collection of independent slots rather than a block that mixes with itself
- a diagonal carry matrix: one number per slot, saying what fraction of itself that slot keeps each step
- how the input is written into the slots, which is where the mixing happens instead
- the input at this step
What one number buys
A slot with carry factor nine tenths has lost half of anything it held after
about seven steps. A slot with factor nine hundred and ninety nine thousandths
takes about seven hundred steps to do the same. That sensitivity is the whole
point: the memory length is controlled by how close the factor sits to one, and
it grows explosively as it approaches.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents