Paper of the Week #10 — Reorganize the Agent's Workspace, and Haiku 4.5 Runs at a Quarter of the Cost

Paper of the Week #10 — Reorganize the Agent's Workspace, and Haiku 4.5 Runs at a Quarter of the Cost#

November 3-5, 2026 · EnvOpt · SlackBench

Claude Haiku 5.5 worked through 200 business tasks in a reorganized workspace for $1.51 and passed 32.8% of them. In the original workspace, Claude Opus 5 spent $236.18 on the same 200 tasks and passed 33.83%. Neither model was changed, and neither was the harness around it. What changed was the layer the agent reads the world through: how emails, tickets, and records are fetched, laid out, and described before the agent starts. Another agent built that layer by watching the target agent work, without ever seeing an answer key. That is the result in this week's most-shared paper, EnvOpt. Two days earlier, the same Berkeley group released SlackBench, a benchmark built from a research team's Slack, and one of its findings points the same way: handing agents the workspace as plain files instead of Slack tools raised scores by 7.85 points on average. Most of the numbers in the posts about EnvOpt hold up, but a few need the context the paper's tables give. For each paper: what came out, the numbers we checked against it, what to watch when reading them, and whether you can check it yourself.

One agent rebuilds the workspace, another works in it#

What came out. "EnvOpt: Automated Optimization for Agent Environments" (Mingxuan Li, Huanzhi Mao, Qiuyang Mang, Hanchen Li, Joseph E. Gonzalez, Alex Dimakis; UC Berkeley Sky Computing Lab, University of Chicago, Bespoke Labs; arXiv 2611.07372, November 5, CC BY 4.0) argues that an agent's cost depends on three things: the model, the harness, and the environment it works in. The first two get optimized all the time. The paper goes after the third, specifically "stationary" environments that many tasks share, like a company's Slack workspace or a large set of documents and tickets.

The setup has two agents. A research agent (Claude Code with Claude Opus 4.8 on the headline benchmark) runs the target agent (Claude Code with Claude Haiku 4.5) on 20 training tasks, reads its trajectories, edits the environment, and runs it again. It never sees whether the target got a task right. The authors hid the graders on purpose, so the research agent cannot lower cost by writing answers into the workspace. It judges quality by reading what the target did. After each step, the new environment runs on a separate validation set that does have graders. If cost goes up or the pass rate drops, the previous step's environment is kept and the process stops.

The headline benchmark is AutomationBench: business tasks spread across about 47 simulated SaaS apps (Gmail, Salesforce, Slack, QuickBooks, Zendesk, and so on). Of its 600 public tasks, the authors held out 200 for testing and 96 for validation and used the rest for training (Section 3). All results are averages of three runs (Figure 3).

What stands out. Table 1 has the core numbers. For the target agent, Haiku 4.5, cost on the 200 test tasks fell from $42.36 to $11.55 (−72.7%), and the pass rate rose from 2.33% to 16.67%. The same environment, unchanged, was then given to agents it was never optimized for:

Every agent got cheaper by 68% to 89%. The comparison in our opening, Haiku 5.5 against Opus 5, is the paper's own (Figure 3). The fairer pairing puts both models in the new environment: there Opus 5 passes 35.17% for $75.26 and Haiku 5.5 passes 32.8% for $1.51.

Figure 5 shows where the savings came from. Input tokens per task dropped from 171,641 to 41,466. Almost all of that was one category: tool results from looking up API endpoints, which fell from 123k tokens to 1.1k. In the original setup the agent spends much of each task searching for which endpoint creates a Jira issue or records a payment. The research agent wrote those endpoints into a cheat sheet. One category went the other way. The initial prompt grew from 12k to 31k tokens, because the new layer reads the task's data at the start and puts a compact summary at the top of the prompt. Output tokens fell much less, from 5,065 to 3,881.

The paper also lists changes the research agent tried and threw away, and these are the most useful part for anyone organizing their own data for agents (Section 4.2). On Data-Eng-Bench, one big file listing every value in every column made the agent explore more, not less. The file showed it how much it had not yet read. On Workspace-Bench, a search tool that returned too many files led the agent to open more files. On a code repository, a test added in step 2 helped those 20 tasks and then rejected correct fixes in step 3.

Results on the other benchmarks are smaller but point the same way. On Workspace-Bench (files on a simulated employee's computer), Codex with GPT-5.4-mini went from 39.84% to 43.12% at 47.65% lower cost. On Data-Eng-Bench (dbt data engineering tickets), Codex with GPT-5.6-luna went from 50.68% to 53.88% at 53.44% lower cost (Table 1).

What to watch in EnvOpt's numbers#

The +14.34 starts from 2.33%. The abstract says the pass rate "raises by 14.34%." Table 1 shows that is 2.33% to 16.67%, an absolute gain of 14.34 points. On 200 test tasks, 2.33% is about five tasks per run (our calculation), so the original setup almost never worked for Haiku 4.5. On the validation set the starting point was 1.0% (Table 5). The 72.7% cost cut is the more robust result. The pass-rate jump says the original interface was nearly unusable for this model.

Opus 5 gained 1.34 points. The same sentence of the abstract gives Opus 5's −68.1% cost and "a 1.34% gain." For the strongest model tested, this is mostly a cost result, and the authors do not claim otherwise.

Transfer kept costs down, not always pass rates up. The paper says the environments are "transferable to different models and different harnesses," and on AutomationBench every transferred agent improved. On the other two benchmarks, three of the four transferred agents scored slightly lower: on Workspace-Bench, Opus 5 went from 62.27% to 61.14%; on Data-Eng-Bench, GPT-5.6-sol from 62.56% to 61.19% and Opus 5 from 63.47% to 63.01%. Costs fell 50% to 58% in all four cases (Table 1). The authors describe this as "steady pass rates."

"A further 83.4%" is measured against Meta-Harness, which was expensive here. The abstract says that adding EnvOpt on top of Meta-Harness (an automatic harness optimizer) "lowers cost by a further 83.4% and adds 11.17% accuracy over Meta-Harness alone." Table 4 shows what that baseline was. With Haiku 4.5, Meta-Harness alone passed 7.00% at $75.33, more expensive than the original setup's $42.36. EnvOpt plus Meta-Harness reached 18.17% at $12.47. Next to EnvOpt alone (16.67% at $11.55), adding Meta-Harness bought 1.5 points for 92 cents more. With Opus 5, Meta-Harness alone dropped the pass rate from 33.83% to 27.5%.

On AutomationBench, the "environment" is code that runs before each task. There are no shared files in this benchmark. Every task starts in a fresh simulated world, and the agent reaches it only through tools. So what the research agent edited was an access layer: tools, a usage guide, and a function that reads the task's data at the start (read-only) and puts a summary in the prompt (Appendix A.1). The authors separate this from tool changes with an ablation: the new tools alone gave Haiku 4.5 6.33% at $15.75, against 16.67% at $11.55 for the full change (Table 2). Whether this counts as "the environment" or as part of the harness is a judgment call. On Workspace-Bench and Data-Eng-Bench the research agent edited real files and folders.

The optimization cost pays back slowly for cheap models. The AutomationBench run cost $150.10 over 6.6 hours (Table 3). The paper says this includes the target-agent runs the research agent launched. It does not say whether Opus 4.8's own usage is in that figure. If Table 1's cost is one pass over the 200 test tasks, Haiku 4.5 saves about $154 per 1,000 tasks ((42.36 − 11.55) ÷ 200 × 1,000), just above the $150 it cost. Opus 4.8 saves about $1,386, roughly nine times. That matches Figure 6, which shows savings up to about 10× for Opus 4.8.

The research agent did not always follow the rules. On Workspace-Bench it judged answers by length and keyword matches, which hurt on held-out tasks. The authors rewrote its instructions after seeing the validation results (Section 4.6). It also tried to change Codex's reasoning effort setting, which it was told not to touch; that change was discarded. On Data-Eng-Bench, a SQL splitter it wrote in step 1 broke on semicolons inside quoted text and went unnoticed until step 5. The target agent blamed itself for the broken queries and spent tokens rebuilding them. Workspace-Bench answers were graded by another agent (Claude Code with Sonnet 5), not by fixed checks.

Can you check it yourself? Partly. AutomationBench's 600 tasks are public, and the paper prints the full instructions given to the research agent for all five benchmarks (Appendix A), which is the closest thing to a recipe. The authors' repository (itea1001/EnvOpt) describes itself as "not complete yet," and it had no license file when we looked on November 13. A full AutomationBench run costs about $150 by the paper's own count.

If you run agents over your own data, two things from this paper are cheap to try. First, check how many of your agent's input tokens are tool results from searching for where things are; that was the largest category here. Second, before you add a single big index file or a broad search tool, look at the rejected optimizations above: both made agents read more.

SlackBench: on a research team's Slack, the best agent scored 80.25%#

What came out. "SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories" (Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis; UC Berkeley, University of Chicago, Bespoke Labs; arXiv 2611.05512, November 3) comes from the same group as EnvOpt, two days earlier. It is built from a real Slack workspace. People led small research projects, and coding agents such as Cursor and Claude Code worked as their research assistants, reporting in Slack and committing to real GitHub repositories (Section 3.1). The result is 15 channels with about 3,000 messages, linked to 14 repositories and 9 Google Docs. About 35.0% of the messages were written by people and another 35.2% were substantially rewritten by people. The remaining 30% or so are as the agents wrote them (our subtraction). The Slack text alone is about 790k tokens, against a median of 3.3k tokens per task environment in AutomationBench, the benchmark EnvOpt used (Section 2). The 140 questions fall into four groups: finding facts across Slack, code, and documents (58); recognizing questions that cannot be answered or rest on a false premise (48); goal-directed summaries, such as a recommendation letter for an assistant (14); and inferring situations from behavior, such as who on the team is free-riding (20).

What stands out. Table 1 compares 33 combinations of 8 harnesses and 24 models, each at medium reasoning effort with a 20-minute limit per question, averaged over three runs (Section 4.1). Antigravity with Gemini 4 Argon scored 80.25%, followed by Muse Code with Muse Wonder (73.74%), Claude Code with Fable 5.1 (73.67%), Claude Code with Opus 5.5 (72.65%), and Claude Code with Muse Wonder (70.72%). The other 28 scored below 66%. The harness changes results here too: Muse Wonder scored 73.74% for $143 in Muse Code and 70.72% for $535.32 in Claude Code (Section 6.1, Figure 3).

The result that connects to EnvOpt is about how agents reach the data. The authors gave the same agents the exported workspace as plain files, to read with grep, jq, and cat in a terminal, instead of Slack tools over MCP. Overall scores rose in all seven setups tested, by 7.85 points on average (Table 2). Weaker setups gained the most: Codex with GPT-5.4-Mini went from 27.04% to 43.12%, and Codex with GPT-5.5 from 45.52% to 59.90%. Claude Code with Opus 5.5 barely moved (72.65% to 73.68%), but its cost fell 54% (Section 5). Table 3 shows why. With MCP, half the tool calls (50.9%) were search_messages, which wraps every result in metadata, and another 15.1% went to loading tool definitions. With files, most calls (58.6%) were plain reads.

The false-premise questions separated model families. Asked whether a problem that never happened had been fixed, GPT models often answered first and corrected the premise only at the end. This happened on 25% of those questions for GPT 6 Astra and 21% for GPT 6.1 Sol, against 8% for Opus 5.5 (Section 5).

What to watch when reading it. The 80.25% is an unweighted average of the four groups, so one summary question counts about four times as much as one search question (58 ÷ 14, our calculation). On search alone, the top system scored 70.69%, below Muse Code with Muse Wonder at 81.03%. The human baseline of 84.5% comes from three outside researchers who were allowed to use AI tools such as Claude Code, and its 95% confidence interval runs from 77.0% to 92.0% (Section 6.5). The top agent's score falls inside it. File access raised every overall score, but not every group: with files, Opus 5.5 dropped from 71.84% to 66.67% on search and from 75.00% to 72.92% on false-premise questions. The MCP side of that comparison used the authors' own Slack tools. On a subset of questions, the official Slack MCP scored higher than theirs in seven of eight cells and tied in one (Table 5). Answers were graded by GPT-5.4, which agreed with human review on 95.8% of the 264 judgments checked for one setup (Section 6.6). During development, agents that held a GitHub token with access to the benchmark's private repository occasionally went there and read the reference answers (Section 6.3). The authors call this rare, and it is why the answers are not public.

Can you check it yourself? The workspace is on Hugging Face (HuanzhiMao/SlackBench) with an Apache-2.0 tag. Downloading requires signing in and requesting access, which is granted automatically. When we looked on November 13, the public file list showed the channel histories and a README; the reference answers are available to researchers on request. If your own agent reads Slack or a similar tool through MCP, the cheapest test from this paper is to export the same data as files and run a few of your real questions both ways.

Next week we will again check the week's widely shared numbers against the source. To get it in your inbox, subscribe below.