Why LLMs Forget, and How Google's 'Sleep' Moves Memories Into a New Slot

Why LLMs Forget, and How Google's 'Sleep' Moves Memories Into a New Slot#

We built a tiny model to try the idea behind Google's new "Sleep" paper: before a model learns something new, copy what it already knows into a fresh, frozen slot. We expected the slot to protect the old knowledge. Our first version did the opposite. Over five test runs, the model kept 27-37% of the old facts with the sleep step and 38-53% without it. After we changed how the model learned (smaller steps, and stop as soon as every new fact is right), the sleep step started to help: across 20 fresh runs it kept 73.2% of the old facts on average, against 50.1% without it, and it came out ahead in every run.

That first failure turns out to be a good way into the paper. A new slot is only the first half of what the paper calls sleep. This post walks through the whole method, piece by piece, with the numbers the paper reports and with our toy where it helps. The paper is "Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories" by Ali Behrouz, Farnoosh Hashemi, Adel Javanmard and Vahab Mirrokni (Google and Cornell, arXiv 2606.03979). Google Research summarized it on its blog on October 27.

The result that shows forgetting most directly is about translation. The model learned two languages that were not in its pretraining data, one after the other. With the usual approach (in-context learning), both translation scores fell, from roughly 21 and 36 ChRF to roughly 10 and 15. With Sleep, the scores stayed near their single-language values on the graph, and the paper says its three-level variant "nearly recovers" them. The ChRF values are our reading of the paper's graph (Figure 5). So why does a model forget, and what does it mean for a model to sleep?

Why a model forgets#

Picture a whiteboard that is already full. To write anything new, you have to erase something old. That is catastrophic forgetting. A language model keeps what it knows in one fixed set of numbers, its weights. Train it on something new, and the same weights are changed again, so some of the old answers get written over.

Our toy shows it in a few lines. A small network learns 100 made-up facts (set A: a random key, an answer from 0 to 9) and gets all of them right. Then the same weights learn a second set, B. B ends at 100%, and A drops from 100% to 54% (seed 100).

Why not keep the new facts in the prompt instead? That works, but only for as long as the conversation lasts, like a note written on your hand that washes off. The prompt is short-term memory and changes with every token; the weights are long-term memory, and after pretraining they stop changing. The paper compares today's models to a person with anterograde amnesia: they remember the past, they can handle the present, but nothing moves from one to the other. Sleep is meant to be the step that moves knowledge across.

Memory with more than one speed#

Think of daily notes that change every day, a notebook you tidy once a week, and an album you put together once a year. The paper builds on an earlier architecture by the same group, Hope, that works like this. Instead of one block of weights that changes either always or never, Hope stacks several blocks that update on their own schedules. The paper's example: one block every 1,000 steps, the next every 5,000, the slowest every 10,000.

Speed alone does not stop forgetting. When the fast block's turn comes, its update overwrites what it was holding. So the knowledge has to move to a slower block right before that update. That moving step is what the paper calls sleep. The model alternates between two states: awake, where it takes in new input, and asleep, where it takes in almost nothing and works on what it already has.

Sleep has two stages. The paper borrows the names from human sleep research (memory consolidation, then dreaming), but it uses the brain as inspiration, not as a claim about how brains work. The idea it borrows is that deep sleep is thought to move the day's memories into longer-term storage.

Stage 1: move the knowledge into a new slot#

Consolidation runs in three steps (paper pages 7-8):

1. Compute. Work out the fast block's next update, but do not apply it yet.
2. Consolidate. The model before the update is the teacher, because its fast block still holds the recent knowledge. The student is the model with the update applied and one new, small slot added to a slower block. Only the new slot trains, until the student behaves like the teacher. Everything else is frozen.
3. Update. Apply the fast block's update and clear the slot it used before.

The new slot is a low-rank expert: two thin matrices, A (d × d_low) and B (d_low × d), with d_low much smaller than d. The blocks are mixture-of-experts layers, where a router picks which experts handle each input, so a new expert can join without touching the others. To make this practical, the paper suggests allocating the slots up front and masking them off until a sleep switches them on. In the paper's experiments the extra slots add up to five blocks of size 64, and the number of active weights stays the same as the original model's (page 25).

Trying it on our toy model#

Our toy keeps only the core of this step. A frozen base layer feeds a fast block (17,098 weights) that learns set A. Then sleep: we record the model's answers on A, add a new slot (4,256 weights, about a quarter of the fast block), and train only that slot to give the same answers. We freeze the slot, reset the fast block, and let it learn B.

def sleep(model, facts, steps=500):
    x, _ = facts
    with torch.no_grad():
        teacher_answers = model(x).argmax(-1)      # what the awake model says about A
    slot = new_slot()                              # 256 -> 16 -> 10, no bias
    old_slots = list(model.slots)
    opt = torch.optim.Adam(slot.parameters(), lr=1e-2)
    for _ in range(steps):
        h = model.base(x)
        out = slot(h) + sum(s(h) for s in old_slots)
        loss = F.cross_entropy(out, teacher_answers)
        opt.zero_grad(); loss.backward(); opt.step()
    slot.requires_grad_(False)                     # the slot is frozen from now on
    model.slots.append(slot)
    model.fast = fast_block()                      # fresh fast block for the next set

Several differences from the paper matter:

• Our student has no fast block at all while the slot trains, where the paper's student keeps the updated fast block.
• During sleep we feed the slot A's original keys again. The paper's setting has no outside data during sleep, which is the very problem its next piece is built for, so our toy skips the hard part.
• After sleep we replace the whole fast block with a fresh one. The paper applies the computed update and resets only the low-rank parts added to the fast block in earlier sleeps (page 9).
• There is no router; the slot's output is simply added.
• We copy hard answers, not the teacher's probabilities. And there is no stage 2.

With seed 100 again, a sleep step between A and B leaves A at 76% after B, not 54%. One run could be luck, so we ran 20 new seeds (100-119) that we had not used while choosing the settings. After B, A averaged 50.1% without sleep and 73.2% with it, and sleep came out ahead in all 20 runs. Both conditions learned B to 100% every time.

It took one failure to get here. In the first version the fast block learned B with a larger learning rate (1e-2) for a fixed 300 steps, and the slot copied the teacher's soft probabilities. On seeds 0-4, the sleep step left A at 27-37%, worse than the 38-53% without it. Our reading is that the fresh fast block learned B so hard that its outputs swamped the slot's answers on A's keys too, since the two are simply added. Separating A's and B's inputs and starting the fast block's last layer at zero did not clearly help. What did help was a learning rate of 1e-3, stopping the moment B was 100% right, and copying hard answers.

The toy still loses about a quarter of A. A fresh fast block learning B still shifts the answers on A's keys. We also tried a third set, A then B then C. After C, A was at 32-48% in both conditions on seeds 0-4: the second sleep protected B, not A. The slot helps, but it is not the whole method. The paper adds two more pieces on top of it.

How a small part teaches a bigger one#

In the usual distillation, a big model teaches a small one. Here it runs the other way: the small part that saw the new data has to teach a bigger one. The teacher is the model with only its fast, small memory holding the new knowledge; the student has more room, because of the new slot. The paper calls this knowledge seeding, and points out two problems. The student can express more than the teacher, so copying the teacher's outputs alone wastes the new room. And during sleep there is no outside data to learn from.

So the paper mixes two lessons:

1. Distillation on the student's own text. The student writes a passage, and at each token its probabilities are compared with the teacher's.
2. Imitation. The teacher writes a passage; the student gets the first part and continues it. The reward mixes two checks: does the continuation mean the same thing, judged 1 or 0 by a fixed reward model, and how close is it letter by letter (1 minus the edit distance divided by the longer length). One weight, γ, sets the mix.

The final objective balances the two with one more weight, α, the share given to distillation. Why both? According to the paper, with distillation alone the student had the knowledge but did not learn to use it, and only weakly copied the teacher. The ablation in the paper's Table 1 (Qwen3-8B, AIME 2024, averaged over 16 samples) puts a number on it: taking imitation out drops the score from 79.2 to 76.8. Taking out the new slots drops it to 78.2, and taking out the meaning check to 78.9.

Stage 2: dreaming#

The second stage is closer to the night before an exam, when you write your own practice questions and work through them. It builds on SEAL, an earlier method in which a model writes its own training data (Zweiger et al., 2025):

1. The model writes practice examples for the task, called dreams. While it writes, the router adds one random expert, so knowledge that looks unrelated gets mixed in.
2. Each dream gets an importance score from its gradient. The top-scoring dreams are kept, plus a few random ones for variety.
3. Each kept dream trains a copy of the model (with LoRA). If the copy's task score goes up, the dream earns a reward of 1; otherwise 0.
4. The model learns to write more dreams like the ones that earned a reward.

In the paper's ARC setup, the model writes 60 dreams per task and throws out 45 (page 26). The order of the stages matters: consolidation settles the new knowledge into a new slot first, and only then does dreaming change the model, which the paper says lowers the risk that repeated self-improvement writes over older skills.

Of the parts the paper removes on SQuAD, dreaming matters most. On SQuAD, where the model answers questions about a passage without seeing the passage, the full method scored 48.9. Without dreaming it fell to 35.7, close to the base model's 31.9 (Table 3).

What the paper reports#

All numbers in this section are the paper's own; we did not rerun any of them.

Math reasoning (Table 2, average over 16 samples):

Each column is one benchmark; each row is one way of training the same model. The strongest baseline in the table, OPSD, reaches 76.6; Sleep reaches 79.2. On Qwen3-1.7B, Sleep also scores highest on all three benchmarks (AIME 2024: 53.2 against 49.8 for the base model).

New facts (Table 3, SQuAD without the passage, same model and setup as SEAL): base 31.9, SEAL 46.7, Sleep 48.9 with one passage; with 200 passages at once, SEAL 43.2 and Sleep 46.2.

Few-shot puzzles (ARC, Llama-3.2-1B): Sleep 80% against SEAL's 72.5%. The score is the share of dreams that led to a correct answer: 5 dreams on each of 8 held-out tasks (page 26).

Two new languages, one after the other (Figure 5): the Manchu and Kalamang translation from the start of this post. Learned one at a time, every Sleep variant matches or beats in-context learning. Learned in sequence, the variant with three memory levels "nearly recovers its single-language performance" (page 12). Google's blog post describes the same result as a "near-zero performance drop", which is stronger than the paper's wording. Cartridges and plain fine-tuning forgot at least one of the languages and fell below in-context learning on at least one task.

Long context (BABILong, Figure 6): Sleep holds up to 10M tokens, where GPT-4 and GPT-4o-mini fail beyond 128K-256K tokens. The small models in that figure, Sleep included, were fine-tuned on the task, and the paper notes that without fine-tuning Sleep also falls sharply (page 26).

What it costs#

Per training step, plain fine-tuning (SFT) is about 4 times faster than Sleep. But to reach the same score, SFT took 4.3 times longer in wall-clock time on AIME 2024, 3.6 times on AIME 2025 and 4.8 times on HMMT 2025 (Appendix B.5). The training settings in Table 5 give 100 steps for both SFT and Sleep and 500 for GRPO. The paper does not say how often a deployed model should sleep, or what that would cost in a running service.

Before you lean on these numbers#

• No code. We found no code link in the paper, so none of the paper's numbers can be checked independently yet.
• Graph values. The translation scores are our reading of Figure 5; neither the translation nor the long-context result has a table.
• Small tests. The ARC result rests on 8 tasks.
• Our toy is a principle demo, not a reproduction. It has two tasks; with three, the first one was not protected.

Summary and the open question#

Models forget because new learning writes over the same weights. Sleep first copies what the fast block holds into a small new slot, then practices on examples it writes itself.

So if a model grows a new slot every night, how many nights can it sleep before it runs out of room? The paper's experiments use a fixed number of slots (five blocks of 64 in its experiments) and does not test what happens when they are used up.

References#

• Behrouz, Hashemi, Javanmard, Mirrokni, "Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories", arXiv 2606.03979 (v2, July 2026).
• Google Research blog, "The SLEEP framework: A new way for LLMs to retain knowledge in continual learning", October 27, 2026.
• Zweiger et al., "Self-Adapting Language Models" (SEAL), 2025.