What is actually stored in the state that a model carries forward?

Write the state as $n$ numbers. The update has no argument for $x_{t-10}$, no argument for the position $t$, and no separate store that the current input can be written into and left alone. Anything from step $t-10$ reaches step $t$ only by having been inside $h_{t-11}$, then $h_{t-10}$, and so on through every intermediate update. That is why the same function can run on a sequence of any length: it never needs to know how long it has been running, and the price is that it also cannot address an earlier position directly.

The fixed length is not a detail of notation. It means the model cannot defer the decision about what to keep: every input must be folded into the same $n$ numbers at the moment it arrives, and later inputs compete for the same coordinates. There is no growing buffer that can be sorted at the end.

Take the simplest update that mixes and forgets: $h_t=0.5h_{t-1}+x_t$, starting from $h_0=0$, with inputs $2$, $4$, $6$. Then

The natural picture is a compressed transcript: some record of each earlier item, shortened but still addressable. The update rule above has no place to put such a record. It has one block of $n$ numbers, and every coordinate is rewritten from the whole previous block and the current input. No coordinate is reserved for the third sentence, or for the item at position 900; after enough updates, the only thing that can be said about an earlier item is whatever is still detectable in the transformed block. A transcript can be compressed by dropping words, but it retains a correspondence between locations and earlier events. A state does not promise that correspondence, and a trained one typically has information spread across coordinates rather than stored in named slots.

That is why inspecting a state does not tell you a list of remembered items. You can ask what the state does on a task, but you cannot read the third item off coordinate seven. The coordinates are a code, not an index.

If the state has $n=512$ numbers, it has 512 numbers after one item and 512 after 10,000; the count is fixed before the sequence length is known. Keeping every position instead, at 128 numbers per item, gives $128 \times 100=12,800$ numbers after 100 items and $128 \times 10,000=1,280,000$ after 10,000. At 10,000 items that is 2,500 times the 512-number state, since $512 \times 2,500=1,280,000$. The fixed state is cheaper by a factor that grows with length, and the same fixed size is why something must be discarded: a summary of 10 items and a summary of 10,000 items have the same room, so the update rule has to decide what earns space while the sequence is arriving.

None of this says a fixed state must lose exact bits. If the update rule copies one coordinate straight through, that coordinate survives indefinitely, and gated rules can learn to hold a value for a long time. What the fixed size rules out is storing a separate entry for every past item. For a task that needs the gist, a fixed block is the right shape. For a question about a number in the fourth line of a table nine thousand words ago, a fixed summary is the wrong shape, and a design that keeps every position, or carries a fixed state plus a short exact window, is doing a different job.