Local AI Coding Agent, Part 3: When the Context Runs Out, the Agent Says It's Done

Local AI Coding Agent, Part 3: When the Context Runs Out, the Agent Says It's Done#

We cut the context to 4,096 tokens and ran the agent six times: three runs with the Part 2 code as it was, and three with the Part 2 code plus one addition that clears old material: tool output, thinking, and file contents already written (we call it "the clearing version" below). Here is one of the clearing-version runs. The model read three files, and on its third step began to think about how to change the todo app. It kept thinking. When its answer reached 2,495 tokens, answer plus the 1,600-token prompt came to 4,095, the end of the window, and the server stopped it there. The answer held no tool call yet. It never got as far as writing the file.

Where the series stands so far:

• Part 1: build a coding agent from local parts only. Flint writes the code, a Python harness, agent.py, runs the tool calls, and a small decision model, Jeff, comments on each shell command.
• Part 2: add three rules to agent.py. Safe commands run without asking, the tests already in the repo are read-only, and when the model stops, agent.py runs pytest itself.

Since Part 2, agent.py works like this: if the model calls no tool, it is done; agent.py runs pytest itself and finishes if the tests pass. So the run printed "3 passed" and ended after 25 seconds.

todo.py was untouched. The three tests already in the repo passed because they pass on the original code. Of the six tests we keep hidden from the agent, two passed.

Four of the six runs at 4K ended this way: one of the three Part 2 code runs, and all three clearing-version runs. The other two (both Part 2 code) were cut off in the middle of a tool call, which the server could not parse, and stopped. In all six the answer ran into the end of the window, and not one file changed.

The model never said it was done. It was cut off mid-answer. It was agent.py, the harness we built, that read the cut answer as "done". In this part we fix that and make the agent a bit more usable. Two changes:

1. Past half the window, clear the oldest material. Old tool output, old thinking, and file contents the model has already written to disk. The newest two tool results stay whole.
2. An answer cut off at the end of the window is not an answer. Drop it and tell the model it was cut off.

At 8K, the Part 2 code stopped at the server in all three runs; the Part 3 code finished all three by itself, with all six hidden tests passing. We measured the price too. A clearing threw away most of the computation the server had kept from the previous request (the prompt cache), and one run reprocessed more than four times as many tokens as the other two. At 4K, two of three runs cleared everything old and the request was still bigger than the window; the third hit the step limit.

Why 4K#

At the end of Part 2 we said Part 3 would deal with a long task filling 32K. Looking back at the six Part 2 runs, though, the largest single request in each run was between 4,834 and 9,932 tokens. This task never fills 32K. To see what happens when the window fills, we had to make the task longer or the window smaller. We kept the task and shrank the window, so the results compare directly with Part 2.

It was not to save memory. On the same A100, Flint used 13,644 MiB of GPU memory at 4K and 13,652 MiB at 8K; at 32K in Part 1 the peak was 14,386 MiB. For this model the window size barely moves memory. How much memory context costs depends on the model; we measured it for a 9B model in the 8GB card post.

The failures people notice late#

While writing this up we read Mittal et al., Babysitting AI Agents (2026). Out of 163,487 Reddit posts about using AI agents, the authors took the 4,367 that describe failures (6,623 failure instances) and sorted them into 23 categories in seven groups, a taxonomy they call MANDATE.

One result lines up with ours. They asked 73 agent users how severe each category is and how easy it is to notice. Across the 23 categories the two ratings ran strongly opposite (Spearman ρ = −0.61): the harder a failure was expected to be to notice, the more severe it was rated. These are users' ratings, not measured failures. Still, it matches what we saw at 4K. A run that crashes, you see at once. A run that prints "3 passed", you believe until you open the log.

Two of the paper's categories showed up:

• Execution misrepresentation: the agent reports something other than what it did. The fake "done" at 4K is close to this, except that here the harness, not the model, made the report.
• Unbounded or idle execution: the agent keeps running without making progress. One of the 4K runs below read the same two files nine times until it hit the step limit. It is among the most common in their data (15.4% of the 6,623 failure instances they coded from Reddit) and, the authors note, has the sparsest research base.

The code: clearing, and dropping cut-off answers#

agent.py grows from 216 lines in Part 2 to 274. It still uses only the standard library.

Clear what the model has already used#

The messages list sent to the model holds the whole conversation: file contents the agent read, command output, the model's thinking, the code it wrote to files. Most of the old part is never needed again. Code written to a file is in the file; a file read earlier can be read again.

KEEP = 2          # the newest tool results are never cleared

def clear_old(messages):
    """Clear everything older than the newest KEEP tool results. Returns how many messages changed."""
    tools = [i for i, m in enumerate(messages) if m["role"] == "tool"]
    if len(tools) <= KEEP:
        return 0
    changed = 0
    for m in messages[:tools[-KEEP]]:
        before = json.dumps(m)
        if m["role"] == "tool" and not m["content"].startswith("[cleared"):
            m["content"] = f"[cleared to save context: {len(m['content'])} characters. Call the tool again if you need it.]"
        if m["role"] == "assistant":
            m.pop("reasoning_content", None)  # old thinking
            for call in m.get("tool_calls") or []:  # file contents the model wrote; the file itself still has them
                try:
                    args = json.loads(call["function"]["arguments"] or "{}")
                except json.JSONDecodeError:
                    continue
                for k in ("content", "old", "new"):
                    if k in args and not args[k].startswith("[cleared"):
                        args[k] = f"[cleared: {len(args[k])} characters]"
                call["function"]["arguments"] = json.dumps(args)
        changed += json.dumps(m) != before
    return changed

Nothing is deleted outright; a placeholder takes its place. A cleared tool result says how many characters were there and to call the tool again if needed. The model knows something was there and can fetch it back. The order of the conversation and the pairing of tool calls with results stay intact, so the server has nothing to reject.

The server says when to clear. Every response carries usage.total_tokens, the tokens now in the window, and llama-server's /props gives the window size. If an answer leaves the window more than half full, agent.py calls clear_old. If a request is too big for the window and the server returns a 400, agent.py clears and asks once more; if there is nothing left to clear, it stops.

A cut-off answer is not an answer#

Our first version had only the clearing, and we ran it three times at 4K. It did no better than the Part 2 code: all three runs took the cut answer as done and finished with pytest passing on an untouched repo. Clearing happens after an answer comes back, and the window was filling while the answer was being written. The prompts of the cut requests were 1,583 to 1,708 tokens, under half the window; the answers had reached 2,387 to 2,512 tokens when they were cut. The trouble was not what was left to clear. It was accepting a cut answer.

So agent.py now reads why the server stopped (finish_reason):

        if finish in ("length", "unparsed"):
            # The answer ran into the end of the window. It is not an answer: drop it and say what happened.
            messages.append({"role": "user", "content": f"Your last answer was cut off or could not be read; it may "
                             f"have reached the end of the {size}-token context before you finished. Think briefly, and change files in small "
                             "steps with edit_file instead of rewriting them whole."})
            continue

length means the answer stopped at the end of the window or a length limit. unparsed is our own label: when an answer is cut in the middle of a tool call, the server cannot parse the call and returns a 500 "Failed to parse", and agent.py treats that as a cut answer too. That is how the two Part 2 runs at 4K stopped. The cut answer stays out of the conversation; instead the model is told it was cut off, to think briefly, and to change files in small steps with edit_file rather than rewriting them whole.

Running it again at 8K#

Same task, same repo, same model, three groups of three: the Part 2 code at 8K (A4 to A6), the Part 3 code at 8K (B4 to B6), and the Part 3 code at 4K (C1 to C3). A script answered y to every y/N.

The Part 2 code stopped at the server in all three runs at 8K: twice the window filled to 8,191 tokens mid-answer and cut a tool call in half, once the request was too big and was refused. Unlike at 4K there was no fake "done"; a run that stopped said so. A5 and A6 had already written a todo.py that passes all the hidden tests when they stopped. The code was there; the run never got to finish.

The Part 3 code finished all three runs by itself, with all six hidden tests passing. B4 had one answer cut off, dropped it, asked again and carried on.

What clearing costs: the prompt cache#

llama-server keeps the computation from the previous request. If the next request begins the same way, it skips that part and computes only what is new. An agent sends the same conversation plus a few lines each step, so normally there is little to compute. Clearing old messages changes the front of the conversation, and everything from the first changed message onward has to be computed again. What that cache holds is covered in our KV cache guide.

From the server log we took how many tokens each request processed anew, as a share of its prompt. In the three B runs at 8K there were 26 requests right after a clearing, and 24 of them processed more than half their prompt anew (lowest 45%, median 85%). Of the other 47 requests, 40 processed less than a quarter anew.

How often a run cleared varied a lot. B5 cleared 17 times and processed 71,820 tokens anew over the run; the run took 231 seconds. B4 cleared 5 times, 14,689 tokens, 124 seconds; B6 cleared 5 times, 15,832 tokens, 96 seconds. B5 was over half the window on most steps, so most of its requests came right after a clearing.

On the A100, reprocessing those 71,820 tokens took about 32 seconds in total; most of B5's 231 seconds went to writing answers. On a card that processes prompts more slowly, those 32 seconds grow in proportion. Half the window is also just the threshold we picked; we did not try others.

At 4K, clearing runs out#

We ran the same Part 3 code three times at 4K. No run claimed a fake "done", as expected now that cut answers are dropped. But no run finished either.

In C1 and C3, with everything old cleared, the system prompt, the task, the newest two tool results, the "your answer was cut off" notes that had piled up, and the new request still came to more than 4,096 tokens. There was nothing more to clear, so agent.py stopped. The todo.py C3 left behind passes all six hidden tests, but the run stopped without adding a single test of its own, so we do not count it as a task the agent completed.

C2 never stopped; it used all 30 steps. Five answers were cut off, and five requests were refused and went through after a clearing. Along the way it read todo.py and test_todo.py nine times between them. What it read was soon cleared, and it read the same files again. This is the paper's unbounded or idle execution. Earlier it had also rewritten todo.py whole twice, and the code it left was worse than the original: none of the hidden tests passed (the original code passes two).

Put plainly: clearing old material only delays the moment the window fills. It does not make the window any bigger. Once the things that must stay (the system prompt, the task, the latest results) fill the window by themselves, there is nothing left to clear. For this task, 8K had enough room to keep clearing all the way to the end; 4K was too small for that.

The predictions we wrote before running#

Both rounds had predictions committed before the runs.

Round one (the Part 2 code and the clearing-only code, three runs each at 4K): three of seven hit, two of them only as worded.

• The Part 2 code would not end with pytest passing in any run and would stop before step 8: miss. One run ended with pytest passing on an untouched repo.
• The clearing code would end by itself in at least two runs: it did in three. None of the three changed a file, so it is hard to call that a hit.
• Runs that ended by themselves would pass all six hidden tests: miss. All three passed two.
• Clearing would happen at least once in every run: hit, but after the cut answer, and once it cleared zero messages.
• At least one run would read a cleared file again: miss. No run got that far. The cache prediction for requests after a clearing could not be scored, because there were none.
• No run would stop on a 400: hit.

What we did not predict: the window filled while the answer was being written, not in the prompt; and a done check of "pytest passes" passes on an untouched repo.

Round two (the 8K and 4K runs above): three hits, one half hit, one miss.

• The Part 2 code at 8K would miss the hidden tests in at least one run and pass them in at least one: hit (3, 6, 6).
• The Part 3 code at 8K would end by itself with all hidden tests passing in at least two runs: hit (three).
• No Part 3 run, at 8K or 4K, would report done on an untouched repo: hit.
• No run at 4K would pass all the hidden tests, and every 4K run would have at least two cut answers: the first half missed (C3's todo.py passes six), the second hit (8, 5 and 6).
• Every request right after a clearing would process at least half its prompt anew: miss. It was 24 of 26 at 8K, the lowest 45%.

When we wrote the round-two predictions we put round one's cut answers at 2,400 to 3,400 tokens. Recomputed from the server log they were 2,387 to 2,512. No hit or miss changes.

What we did not check#

• Clearing before sending. agent.py clears after an answer comes back, so a request can go out already over half, as B4's did. Measuring a request before sending it and clearing first is untested.
• The half-window threshold. Clearing later would lose the cache less often but risk more cut answers. We did not vary it.
• Summarizing instead of clearing. Keeping a short summary of the old conversation rather than a placeholder is not covered here.
• The done check itself. The fix filters out cut answers; "done when pytest passes" is unchanged. It does not check that any file changed or that new tests exist.
• A long task at 32K. We shrank the window to make this happen. We expect a roomy window to make this rarer, but we did not run a task long enough to fill 32K.

Next#

Part 4 gives the decision model, Jeff, more to do. Today it adds a one-line opinion to each shell command; next we try letting the small model decide first and calling the large one only when needed. As before, the predictions go in before the runs. New posts go out through the newsletter at the bottom of the page.