Long-Term Memory for a 32K Local Agent Without More GPU Memory: Testing Mnemonic Proxy

Long-Term Memory for a 32K Local Agent Without More GPU Memory: Testing Mnemonic Proxy#

Mnemonic Proxy is a proxy that sits between a coding agent and a local model server and lets a long conversation work as long-term memory. Tools that turn your conversations with an AI agent into long-term memory are not new, and Mnemonic Proxy is one more of them. The difference is what it keeps. The common approach summarizes old conversation and keeps the summary. Mnemonic Proxy stores the original text as it was and leaves the model only one line that can fetch it back.

Few terms in AI are as confusing as long-term memory. The name suggests the AI remembers your conversation, but the model does not remember anything. It takes a request, writes an answer, and keeps nothing of what the request contained. So how do ChatGPT or Claude know what you said earlier? And what do the memory products and libraries actually do?

The answer is almost disappointingly simple: every request sends the earlier conversation to the model again, from the start. Memory that reaches beyond one chat works the same way, with saved notes added to the request. The model remembers nothing, but because it rereads everything each time, it can answer as if it did.

That has an obvious limit. A model can only read so much at once, and that amount is its context window, counted in tokens. The 32K window we used holds 32,768 tokens. When the conversation is longer than the window, the server refuses the request. A bigger window helps, but costs GPU memory: in our runs the same model's server used 6,580 MiB with a 32K window and 9,748 MiB with 128K. Coding agents also read whole files and take in command output, so even 128K fills up sooner than you would think.

So most long-term memory methods so far have been about shrinking a conversation that keeps growing. The typical one waits until the window is full, summarizes the older part and carries only the summary forward. opencode, the coding agent we used, has this built in as compaction. The catch is that whatever the summary leaves out cannot be found again.

Mnemonic Proxy puts things away instead of shrinking them. It replaces old tool outputs with a one-line receipt that says what was read, and stores the original text in a local archive. It gives the model a recall tool that searches the archive and brings the original back. When the window is nearly full, the model writes handoff notes and the proxy starts a new segment, so the conversation can go on while the window stays at 32K. With llama-server it also saves the server state to disk, so a restarted server does not have to read a long conversation from the start.

To see whether it really remembers, we ran a nine-turn coding task with Qwen3.5-9B and opencode. Turn 1 prints three values and turn 8 asks for them. We ran it three times each with a 32K window and opencode compaction, a 32K window with Mnemonic Proxy, and a 128K window without compaction.

Here are the results first. The three sessions with compaction did not get a single value right. The 128K window filled up around turn 6 in all three runs, and every request after that was refused. With Mnemonic Proxy, on the same GPU memory as plain 32K (6,568 to 6,580 MiB), all three runs reached turn 9 (B1 and B2 hit the 30-minute limit on one or two turns along the way), and in all three sessions the turn-1 values were still in a 9 to 12 MB archive, word for word. One of those sessions started a new segment 25 times before its turn-8 recall and still got all three values with a single recall.

Fetching them, though, was up to the model. Asked for the values in turn 8, one session in three went to fetch them with recall; after turn 9 said where the values came from, two sessions had them right. No session made up a value; in the compaction and proxy runs, every value the model could not find was written as unknown.

The project README says itself that this combination is still experimental: opencode with llama-server has not had long real sessions. On a subset of the benchmark it reports, the proxy at 32K scored 86, summarizing 47 and the full history in a larger window 69, and 85, 57 and 75 on a rerun (where the gap to the full history was not significant). Our task is different from that benchmark, so read the numbers below as what came out under our conditions.

The setup#

All runs used the same model file (Qwen3.5-9B, Q4_K_M), llama.cpp build 947, opencode 2.0.32 and Mnemonic Proxy 0.3.24, with temperature 0.6 and seed 42. The agent ran in a container with no network; only the model port was passed through.

C is the spend-more-memory option: a window four times larger. Compaction was off, so the whole conversation stayed in the window for as long as it fit.

We ran everything on an A100 and checked that the server process stayed within a 12GB memory limit: 6,568 to 6,580 MiB at 32K and 9,748 MiB at 128K.

The task was a copy of the Click library, nine turns in a fixed order. The exact prompts are in the nine turns, verbatim.

1. Run setup_env.py. It prints 35 lines of setup log, three of which hold a build ID, a p95 latency and a count of skipped migrations (example output). The values are random per session, and the script deletes itself after printing, so they are not left anywhere in the project folder.
2. to 5. Read Click modules in full and write a document for each (docs/core.md and three more).
3. and 7. Write a new table.py with a table formatter and a CSV writer and parser, plus tests.
4. Write REPORT.md with the three values. The prompt does not say "remember" or "earlier". It says to write unknown for any value it does not have.
5. A hint: the values were in the output of the setup_env.py run at the start. Write them to REPORT2.md.

We scored the code with 18 hidden tests the agent never saw, and turns 8 and 9 by exact match against the values we generated. Each turn had a 30-minute limit (10 minutes for turns 8 and 9) and each session 90 minutes, so a turn late in a slow session could be cut off sooner.

Nine sessions#

No session wrote a wrong value. In A and B every miss was unknown, as the prompt asked; C never got to write a report. One A session went looking elsewhere: in turn 9, A2 searched opencode's own state folder and log 12 times, found nothing it could use, and wrote unknown.

The 128K window ran out first. All three C sessions reached a prompt of about 129,000 to 131,000 tokens by turn 6. Partway through turn 6, after writing two files, C1 and C2 sent a request llama-server refused ("request (143,011 tokens) exceeds the available context size" in C1). C3 ended turn 6 on an output-limit error instead and was refused from turn 7. Every later request was refused, including turns 8 and 9, and the table code was never finished, hence 2 or 3 hidden tests. C1 and C2 also ended with the same number of requests (27) and the same share of prompt tokens served from llama-server's cache (87.85%), though their prompts differed slightly in size. All sessions used the same seed, so the repeats may be less independent than three separate runs suggest.

On code, A and B were close: 9.0 hidden tests on average for A and 10.0 for B, with A ranging from 6 to 11 and B from 9 to 11. Three runs per setup cannot separate a one-test difference from noise.

Inside the reread loop#

opencode's read tool, called without a range, returns at most the first 1,313 lines of click/core.py. To read the rest, the agent has to ask again with an offset. In B and C, it did: the second call asked for line 1,314 onward. All three C sessions and B1 and B2 then wrote docs/core.md; B3 wrote the same document to click/core.py.md instead.

In A, every one of the 507 reads in A1 was the same call without a range, and every one got lines 1 to 1,313. The session database shows the messages of that turn alternating exactly: one model step, one compaction, one model step, one compaction. This is the summary opencode wrote after one of those reads:

## Work State
### Completed
- Located click/core.py at /work/click/core.py
- Read full contents of click/core.py (1313 lines)

## Next Move
1. Analyze click/core.py to identify all public classes
2. Write docs/core.md with one section per public class listing purpose and main methods

The summary records that the file was read, and believes it has 1,313 lines, but it does not contain the file. The next step is to analyze a file the model no longer has, so it reads it again, which fills the window, which triggers the next compaction. None of the three A sessions called write in turn 2. The 1,313-line cut is not what starts it: in turn 5, A2 fell into the same loop with three shorter files it had read in full, each of them 311 times, and timed out again.

C, with no proxy and no compaction, read the file the same way B did, so the loop came with compaction. B did not keep the whole file either. The proxy replaced pages of core.py with one-line receipts during the turn, 13 times in B1 and 17 times in B2, starting with the second page right after it arrived. The model then read the file again in smaller pieces, which is where B1's 18 reads and B2's 41 come from. Unlike A, the reads moved through the file instead of repeating the first page, and the turn ended with a document. B3 took another route: the proxy switched segments twice inside turn 2, the model called recall nine times, and it then read the last 74 lines, 2,974 to 3,047, for the first time; lines 2,681 to 2,973 it saw only through a search for class definitions. What B never got was a summary saying the whole file had been read; which part of the receipts the model relied on, we did not test.

The same compaction also explains A's memory result. The very first summary in A1, written in turn 2 about turn 1, did keep two of the values: "build ID BLD-9W2LNZYG and p95 latency of 428ms" (the migration count was already gone). The next summary was about core.py and kept nothing from turn 1. In all three A sessions, only that first summary held any true value, and it was replaced hundreds of times before turn 8.

What decided memory: recall#

In B, the values never disappeared. The archive held the raw setup_env.py output, a log of 35 setup steps with the three values among them, in all three sessions. What differed was whether the model went to get it.

B2 called recall in turn 8 with the query "build_id p95_ms skipped_migrations" and got all three. B3 used recall 15 times during the session, mostly in turn 2, but in turn 8 it wrote unknown without looking. After the hint in turn 9 it called recall and got all three. B1 never called recall at all, not even after the hint.

The handoff notes did not carry the values either. None of the notes stored in the archive database (13, 24 and 19 of them) contained the build ID. So with this model and this prompt, the proxy kept the information safe, and getting it back depended on the model deciding to search. A prompt that tells the model to check the archive for earlier facts might change that; we did not test one.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

Restoring after a restart#

The proxy's other promise is speed after a restart: instead of reading a long conversation again, llama-server loads the saved state. We tested that separately; each run started only when no other job was using the GPU. Each run built a conversation of a given size through the proxy, waited for the proxy to save the state, restarted llama-server and asked a follow-up question through the proxy. Then we restarted again and sent the same follow-up straight to llama-server, which has to read everything again. Times are from sending the request to the end of a 32-token answer.

The 24K row is there to show a restore with no segment switch in the way.

At 24K and 100K, the restore worked as described: after loading the state, the server processed only 22 tokens, and the answer matched the one from a full re-read in every run. Loading the state itself took 0.36 to 0.39 seconds at 24K and about 1.08 seconds at 100K. The README reports 1.4 seconds against about 50 seconds at 110K, on different hardware, and the README says most of its measurements used Strata rather than llama-server; on our A100 with llama-server the gap at 100K was about tenfold.

The 28K row shows a boundary rather than a restore. The proxy estimated the first request at 28,768 tokens; with the 4,096 tokens it reserves for the answer, that is 32,864, past the 32,768-token window. So the first request already triggered a segment switch (writing the handoff notes took 11.8 seconds in the first run), and after the restart the follow-up triggered another one, which meant reading about 28,600 tokens anyway. The state loaded in under half a second, but the switch made the whole request slower than simply reading again. The 2.4 GB includes the state saved at the switch; the autosave itself was 1.2 GB.

Disk use adds up. Each B session left 24.5 to 34.0 GB of saved states, while the archive itself was only 9 to 12 MB.

If you try it#

• opencode setting. The README suggests chunkTimeout: false. In opencode 2.0.32 that value made opencode drop the whole provider without an error (the log only says provider.no-route). A large number, such as 3600000, worked.
• Look at the summaries when an agent rereads. In our runs the reread loop appeared only with compaction on. The summaries recorded that the file was read and dropped its content.
• The archive is the memory, and only if the model searches it. In turn 8, one session in three called recall for the values without a hint.
• Mind the switch point. A conversation just under the window size can trigger a segment switch on the next request, and then a restore is not faster.
• Plain text on disk. The archive stores tool outputs as they were, including any secret the agent printed, and the proxy has no authentication, so it should listen on 127.0.0.1 only (both points are in the README).

What this does and does not show#

• Three sessions per setup, one task, one model, one opencode version. A different model may use recall more, or less.
• Sessions ran on an A100; we only checked that the server stayed under 12GB. Session times are rough.
• All sessions used the same seed; C1 and C2 ended with the same request count and cache reuse.
• This is not a reproduction of the README's benchmark. The task and the scoring are different.
• The only bigger window we tried was 128K with compaction off. We did not test 128K with compaction, or sizes in between.

The next thing we want to know is whether one line in the system prompt, telling the model that earlier tool outputs are in an archive it can search, changes the 1 in 3. If you want that result in your inbox, the newsletter form is below.