Answers · How models work
How does a gate let information pass through unchanged?
In shortA gate lets old information take a direct additive route through the steps, with a learned fraction deciding how much survives each step.
A gate lets information pass through unchanged because the previous state is multiplied by a learned keep fraction and then added to, rather than being sent through a matrix and a squashing function. With keep fraction one and nothing new written, the new state is exactly the old state; with keep fraction near one, the old component survives many steps because multiplying by a number close to one loses very little.
The default behaviour of a slot is to persist, not to be transformed. On the main route the previous state meets only one multiplication, by the keep gate, before it reaches the addition. The candidate route is where fresh content is proposed, and the write gate decides how much of that proposal enters. If the keep gate sits at one and the write gate at zero, the new state is the old state, however many steps have passed.
The arithmetic of a held slot
Take one slot holding value 1 and hold the keep fraction at 0.999 while the write gate is zero. After one step the slot holds 0.999. After ten steps it holds 0.999^{10}=0.990, so 99.0 percent remains. After a hundred steps it holds 0.999^{100}=0.905, so 90.5 percent remains. After a thousand steps it holds 0.999^{1000}=0.368, so 36.8 percent remains. The arithmetic is repeated multiplication by the keep fraction; the identity case is the same multiplication with factor one.
| keep fraction | after 10 steps | after 100 steps | after 1000 steps |
|---|---|---|---|
| 0.999 | 99.0 | 90.5 | 36.8 |
| 0.99 | 90.4 | 36.6 | 0.0 |
| 0.9 | 34.9 | 0.0 | 0.0 |
In a plain recurrence the per-step factor is fixed by the weights and applies to every slot and every step. If that factor is 0.95, after fifty steps the surviving fraction is 0.95^{50}=0.077, about 7.7 percent, and after a hundred steps it is 0.95^{100}=0.0059, about 0.59 percent. The same multiplication happens on every route from an early step to a late one, so the factor along a long route is a fixed number raised to a length. A gate replaces that with a number computed from the current state and input, so one slot can sit near one while another is rewritten.
Keep and write are separate
Keeping and writing are separate decisions. The keep gate asks how much of what the slot already holds should survive. The write gate asks how much of the candidate should be admitted. If one number had to do both jobs, admitting new content would force the old content out, and a long-held value could not coexist with active processing in the same step. With two numbers, a slot can be nearly closed to new writes and nearly open to retention, or the other way round. The candidate is where the usual matrix and squashing work happens; it is a proposal, not a replacement.
A gate is not extra memory
What gates do not do is add capacity. The state still has the same fixed number of slots, and the gates are numbers computed for those slots, not a second store. The gain comes from the route: on the main path the previous state is only multiplied by the keep fraction and added. If the keep fraction is one and the write gate is zero, that route is the identity, so the information is copied forward rather than transformed again. A plain recurrence has no such route, because every step applies the same weights and squashing function, so the per-step factor is fixed by the weights and applies to every slot and every step. A fixed factor cannot be held near one for one slot while another slot is being rewritten.
Because the keep fraction is computed from the current state and input, it differs by step and by slot. The model can learn to drive one slot near one when something worth holding arrives, leave it there while irrelevant steps pass, and drop it later. A fixed rule cannot do that, since it has no way to distinguish one slot or one step from another.
The unchanged pass is conditional. If the keep fraction is 0.9, after ten steps 0.9^{10}=0.349 remains, and after a hundred steps 0.9^{100}=0.0000266 remains, which is gone for practical purposes. If the keep fraction is one but the write gate is not zero, the old component is still present in the sum but the state changes because new content is added. The reach is finite: a few hundred steps reliably, not thousands, and the state is fixed-size, so a slot held open is unavailable for other content. Training is still a sequential walk through the sequence.
Worth keeping
- A gate does not add memory; it changes the route the existing state takes from one step to the next.
- If the keep fraction is one and nothing new is written, the state is copied forward exactly.
- A keep fraction of 0.999 leaves about 36.8 percent after a thousand steps, while 0.9 leaves about 0.0000266 after a hundred.
Where this goes next
This question is one step of Models That Carry a Memory, a course that works through its subject from the beginning. The page that teaches this part of it is Letting Things Through Unchanged, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does a transformer need a cache that grows with sequence length?A transformer stores the key and value for every token it has already seen so later tokens can look back at them, and that store grows with the conversation.
- Why can a linear recurrence be computed as a convolution?Because a linear recurrence adds up past inputs with fixed decaying weights, and that weighted sum is a convolution with the recurrence's impulse response.
- How can a model remember a sequence without keeping every past input?A model can remember a sequence by keeping a fixed-size block of numbers and rewriting it at every step from its old value and the new input.
- What is actually stored in the state that a model carries forward?It is a fixed-size block of numbers rewritten at each step from its old value and the new input, not a stored copy of earlier inputs.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.