autoharness: "Summarize the Chat for Next Time" Is One Line. Why Is the Code 3,104 Lines?

autoharness: "Summarize the Chat for Next Time" Is One Line. Why Is the Code 3,104 Lines?#

When we first read the autoharness README, our reaction was: isn't this just handing the conversation to another session, asking it to summarize, and loading the summary next time? That idea fits in one sentence. The plugin's Python code is 3,104 lines, plus 7,064 lines of tests.

So we followed one piece of conversation through the code, from the moment it happens to the moment a later session uses what was learned from it. The one-line idea is all there. Most of the code is about something else: keeping that idea from going bad when nobody is watching it.

Line numbers below refer to commit 4f2a358 of tigerless-labs/autoharness (v0.5.3, MIT), the main branch on October 29, 2026. Every claim in Steps 1-5 is something we read in the code or the README; where a number comes from our run, we say so. In the last part we install it into Claude Code connected to Qwen3.8 Flash Next on our own GPU and watch one full loop.

What it is, in one paragraph#

autoharness is a Claude Code plugin. Every 50 tool calls, it starts a second Claude session in the background, gives it the recent conversation, and asks: is there anything here a future session should know? The answer comes back as proposed skills, small instruction files under .claude/skills/. A separate piece of Python decides whether each proposal is written to disk. Later sessions see a list of those skills and can open the ones that fit the task.

Step 1. One hook catches everything#

hooks/hooks.json registers four Claude Code events: SessionStart, Stop (end of a turn), PreToolUse (just before any tool runs) and SessionEnd. All four call the same Python module, autoharness.hook.dispatch, which routes by event name.

The counting happens in PreToolUse. Every tool call in the main session adds one to a per-session counter (dispatch.py:153-157). At the end of each turn, on_stop checks that counter. Once it reaches 50 (REFLECT_EVERY_N, config.py:32), the counter resets and reflection is triggered (on_stop.py). When the session closes, on_session_end triggers one more time for whatever is left over, as long as at least one tool was used.

Step 2. Start the second session and don't wait for it#

The trigger does not block you. dispatch.py:87-98 launches the reflection job with subprocess.Popen(..., start_new_session=True) and sends its input and output to /dev/null. Your turn ends normally while the job runs on its own.

That job assembles a bundle (spawn.py:56-66) with four parts:

1. the conversation since the last reflection, cut from the transcript file by byte offset (capture.window)
2. a short digest of the 20 exchanges before that, text and tool names only, marked "never an evidence source"
3. a one-line index of every skill that already exists
4. the format spec the new skill has to satisfy

Before it leaves, the conversation slice is passed through lib/redact.py, which masks secrets and personal data using the rules in redaction_rules.toml (13 secret patterns and 4 PII patterns).

Then it runs a new Claude Code process (spawn.py:103-109):

claude -p --agent autoharness:reflector --dangerously-skip-permissions

with the bundle on stdin. The reflector agent is defined in agents/reflector.md. Its tools are Read, Grep, Glob and one more, stage_skill. It has no Write, no Edit and no Bash. The model is haiku. Permission prompts are skipped because nobody is there to answer them. The tool list, plus a hook that denies file writes, keeps it contained.

One detail matters for the measurement below: child_env (spawn.py:112-117) copies your whole environment into the child and adds three variables. If your Claude Code points at a different endpoint through ANTHROPIC_BASE_URL, the reflector goes there too.

Step 3. The model can only propose#

This is the first place where "summarize and reload" turns into something else.

stage_skill does not write a skill. It appends a proposal, called an intent, to a queue file (stage_skill/server.py). An intent says what to do (create, update, patch, remove_file or delete), the content, a reason, and an evidence field that the reflector is told to quote word for word from the conversation.

After the child session exits, promoter.drain reads the queue (spawn.py:179). The promoter is the only code that writes to the skill folders. Its docstring calls it "the sole deterministic writer of the host skill tree." For each intent it builds the final file in memory, runs the checks, and writes only if every check passes (promoter.py:161-237).

The checks in lib/validate.py (no LLM involved) include:

• safety: regex families in skills_guard.py for exfiltration, prompt injection and destructive commands
• secret: the same redaction rules again, on the final text
• structure: frontmatter that parses, with name and description
• completeness: a body that is not empty, with no TODO or placeholder text
• description: one sentence, at most 60 characters, because it becomes the index line
• global: a skill meant for every project may not contain absolute paths or this repo's name
• evidence: reason and evidence must be present
• self-produced: update, patch and delete only work on skills autoharness itself created, and create may not overwrite a skill that it didn't create (validate.py:336-339)

That last check is how the plugin avoids touching skills you wrote by hand. A new skill gets a sidecar file stamped created_by: agent after it passes. Anything without that stamp is outside its reach for edits, deletions and overwrites.

If any check fails, nothing is written. If everything passes, support files land first and SKILL.md is written last with an atomic rename, so a reader never sees a skill that points at a file that isn't there yet. The quoted evidence is saved next to the skill as references/evidence-<hash>.md, so you can always see which part of which conversation a rule came from.

Step 4. The next session sees an index, not a summary#

Here is the second difference from "reload the summary."

At SessionStart, recall_index (on_session_start.py:58-85) builds one line per skill, name [layer]: description, grouped by category, and injects it as extra context. Only skills autoharness created are listed. The header tells the model to consider loading a skill when the task matches.

The full instructions stay in each SKILL.md, and the model reads one only when it decides it's relevant. In our run (below), each index line was 92 to 95 characters: a name, the layer, and a description capped at 60. At that width, 50 skills add about 4,700 characters to the start of a session, however long each skill's body is. Loading the four SKILL.md files our run saved would cost about 750 characters each, eight times more, and a summary has no cap at all: it grows with everything that was ever learned.

The format spec pushes the content the same way. A skill body is capped at 25 lines (SKILL_BODY_MAX_LINES), and the reflector is told to write the rule, not the story: "do not retell this episode." Detail goes into references/ files that are read only on demand.

Step 5. Skills that nobody uses get moved out#

The third difference is the one we found most interesting, because it is the part we do by hand.

When autoharness writes a skill, it records how many turns the project has had so far. From then on, on_skill_call.py keeps two counters per skill: use, bumped when the model calls the skill through the Skill tool, and view, bumped when the main session reads any file inside the skill's folder. The docstring says that in the authors' own experiments, models almost always read SKILL.md directly instead of calling the Skill tool.

At every SessionStart, lifecycle.evaluate (lib/lifecycle.py:23-45) decides what to archive:

1. Probation. A project skill younger than 100 turns is left alone.
2. Graduation review. At 100 turns, a skill with zero use and zero view is archived. A skill that was read but never called is spared.
3. Capacity. Skills with at least one use compete for 50 slots per project. Above that, the lowest use per turn goes first. view never counts toward this rate (tests/test_lifecycle.py, test_view_never_feeds_the_rate).

Archiving moves the folder out of the live tree. It does not delete it.

On top of that, every 250 turns a second agent, the curator, looks only at autoharness-created skills and merges narrow ones into broader ones (agents/curator.md). Before it runs, the whole library is saved as a tarball, five kept per layer (spawn.py:184-197).

So, is it just "summarize and reload"?#

The idea is. What the 3,104 lines add are three answers to problems that only appear after weeks of running unattended:

• Size. The next session pays for one line per skill, not for everything ever learned.
• Who writes. The model proposes; deterministic code checks and writes, and won't edit what you wrote yourself.
• Decay. What isn't used gets moved out, so old rules don't pile up forever.

We keep our own project memory the same way, by hand: a one-line index, separate files for each rule, and deleting entries that turn out wrong. autoharness does that work every 50 tool calls, without being asked.

What the code doesn't tell us#

Reading the code shows how the loop is built. It does not show whether the loop produces good skills.

• The reflector is told to "capture liberally," because "retirement is cheap, forgetting is not." That could fill the library with weak rules that sit in probation for 100 turns.
• A skill that has been read but never called through the Skill tool is spared at review and never enters the capacity race, so the 50-slot cap does not apply to it. If models mostly read skills directly, as the authors' docstring says, many skills could end up in that group. We don't know yet how many.
• Each reflection is a full extra Claude session, and the README does not report what one costs.

So we ran it.

Running it on a local model#

We did not use the Anthropic API for any of this. Claude Code, and the reflector it starts, both talked to Qwen3.8 Flash Next (IQ2_XS) served by Strata on one 80 GB GPU in our lab, with a 65,536-token context. Strata speaks the same /v1/messages API as Anthropic, so connecting Claude Code is a matter of environment variables. The full walkthrough, including a 12 GB card, is in our earlier post on running Claude Code on a local model. The short version:

export ANTHROPIC_BASE_URL=http://127.0.0.1:8090
export ANTHROPIC_AUTH_TOKEN=local
export ANTHROPIC_MODEL=claude-sonnet-4-5
export ANTHROPIC_SMALL_FAST_MODEL=claude-haiku-4-5
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export API_TIMEOUT_MS=1800000
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192

The model names are labels only; Strata answers every request with the one model it has loaded. The last line matters most. Claude Code asks for 32,000 output tokens by default, and Strata refuses a request whose prompt plus output budget doesn't fit in the context, so a long session dies with a 400 error. With 8,192 it fits.

Two things were specific to autoharness.

• Where to install the plugin. Claude Code has a --plugin-dir flag, but the reflector is a fresh claude -p process that never sees that flag. So we installed the plugin with claude plugin install into a separate config folder (CLAUDE_CONFIG_DIR), which the child inherits along with everything else. This also kept our own Claude Code setup untouched.
• The reflector's model. reflector.md asks for haiku, but because the child copies the parent's environment (Step 2), its requests went to the same local server. Every response in all four transcripts came from qwen3.8-flash-next-iq2_xs.

The setup, in a scratch git repo outside our blog code:

• autoharness at commit 697cf5e (October 28). The 19 commits between it and 4f2a358 (4 of them merges) add redaction rules and the empty-body and overwrite checks, and fix locking, queue recovery and the cleanup of leftover files. Against our run: the reflector used only Read, Glob and stage_skill, every queued proposal landed, none of the new redaction rules matches anything in our saved skills, our skills were all new and had bodies, and no hand-written skill existed in the repo. None of these changes would have changed the result.
• Reflection every 10 tool calls instead of 50 (AUTOHARNESS_REFLECT_EVERY_N=10), so a short session triggers one round.
• Global skills off (AUTOHARNESS_DISABLE_GLOBAL=1), so everything stays in the scratch repo.
• Session A: write wordcount.py with count_words(path) and tests, plus a house rule stated once: tests go in checks/ as check_*.py, not tests/.
• Session B: a new session in the same repo, adding count_lines(path) with tests. The rule is not repeated.

We wrote down five predictions and committed them before the first run.

What we predicted, and what happened#

P3 was wrong in an instructive way. The long descriptions did happen. Every new skill's first description was 126 to 151 characters long. But none of them reached the promoter. stage_skill already runs the structure and description checks the moment a proposal is staged (stage_skill/server.py:123-145), and sends the failure back as the tool result. Across both rounds, 5 of 11 stage_skill calls were refused this way: two for length, and three because they were aimed at the global layer we had turned off (that check comes before length). The reflector read each error and staged again. Every new skill ended up with a description of 59 or 60 characters. By the time the promoter ran, every proposal was already valid.

So the 60-character rule works less like a filter and more like an editor. Here is what landed after Session A, in full:

---
name: test-location-checks
description: "Use when writing tests here: checks/check_*.py, not tests/."
category: testing
---

Tests in this repo go in `checks/` as `check_*.py` — never a `tests/` directory, never `test_*.py` files. Run them with `python -m pytest -q` from the project root and confirm they pass before reporting the work done.

Collection depends on the pytest config in `pyproject.toml`; if that file is missing, start from `templates/pyproject-pytest.toml`. Use pytest's `tmp_path` fixture for test data rather than writing files into the repo.

Two other skills landed from the same 38-second session. pytest-import-path records a rule the session never tested: that bare pytest would fail to import modules at the repo root. Session A only ever ran python -m pytest, as the task said. The third one surprised us: memory-file-placement. In Session A, the main model first wrote its own memory note into the repo, then noticed, rewrote it in the right place and deleted the stray copy. The reflector turned that slip into a rule. After Session B, a fourth skill (text-unit-counting) landed and test-location-checks was patched twice. That is six writes from about one minute of actual coding, which is what "capture liberally" looks like in practice.

What one round costs. The Session A reflection made 3 requests to the server, with 154,032 input tokens in total and 8,732 output tokens. The coding session it reflected on generated 2,732 output tokens. The largest single reflector prompt was 56,463 tokens. With the 8,192-token output budget, that is 64,655 of the 65,536 available. A slightly longer window and that round would have failed with the same 400 error as above.

Session B and the index. The three-line index was injected at the start of Session B (we can see it in the transcript). Session B never opened a skill. It did put its new tests in checks/, but we can't credit autoharness for that: the existing test file was already there, the model listed the folder first, and Claude Code's own memory had saved the same rule in Session A. The rule ended up stored twice, once by Claude Code and once by autoharness.

This was one run of each session, with reflection every 10 tool calls rather than 50, on one local model. It shows the loop works end to end on a local model, and where it's tight. It doesn't show whether the skills help over weeks.

If you install it#

Open .claude/skills/ after the first reflection round and read every SKILL.md it wrote. Each one is under 25 lines and has the conversation it came from saved next to it in references/, so this takes a few minutes. In our run, about one minute of coding produced four skills, and one of them came from a mistake the model fixed on its own. Deciding which of those you want to keep is still your job; the 100-turn probation means nothing is archived for a while.

We'll keep opening tools like this one and checking what the code does against what it says. If you want the next one in your inbox, the newsletter is below.