Paper of the Week #8 — Agents Beat All 20 Science Benchmarks, and None of 135 Follow-up Papers Passed

Paper of the Week #8 — Agents Beat All 20 Science Benchmarks, and None of 135 Follow-up Papers Passed

October 19-23, 2026 · Agent4Sci · Into the Unknown · BizAgentBench · CarryOn

Two papers this week handed scientific research to AI agents and came back with numbers that point in opposite directions. In Agent4Sci, from authors at Anthropic, agents beat a re-run of the previous best result on all 20 scientific ML benchmarks. In Into the Unknown, led by CMU, nine agents wrote 135 follow-up papers and not one met every condition the scientists had set. The two results do not contradict each other. The first asked whether one metric went up. The second asked separately whether the agent was honest, whether it left every record that was required, and whether the work met the scientist's own bar. All four items this week share that question: does the score look only at the outcome, or also at how the agent got there? For each item we note what came out, the numbers we checked against the source, what to watch when reading it, and whether you can check it yourself. We did not run any measurements of our own this week.

All 20 science benchmarks beaten, mostly by engineering

What came out. "Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?" (Zhong, Li, Carlini and Jagielski; Anthropic Fellows Program and Anthropic; arXiv 2610.28763; October 20; NeurIPS 2026 Evaluations & Datasets track; code Apache-2.0) collects 20 scientific ML benchmarks (25 tasks) across 11 fields into a testbed called Agent4Sci. Each agent gets eight H200s, 12 hours and a minimal instruction to improve a fixed metric. The baseline is not the number printed in the original paper but the authors' own reproduction, checked by a person. The agents are Claude Code (Opus 4.6 and Sonnet 4.6), Codex (GPT-5.4) and Gemini CLI (Gemini 3.1 Pro). At least one of the four beat the baseline on every benchmark, with best gains ranging from 1.1% on ProteinGym to 64.4% on PFDelta (Section 3.2). Averaged over the 20 benchmarks, relative improvement was 10.4% for Opus, 9.6% for Sonnet, 6.8% for Gemini and 4.4% for Codex (Table 5). With the authors' plug-in skills (three markdown files covering research ideas, experiment management and an hourly self-check), Opus rose to 20.1% and Codex to 8.1% (Table 2).

Worth a closer look. The authors conclude that most gains came from ML engineering rather than research insight. Sorting Opus's strategies into seven categories, hyperparameter tuning accounted for 56% and architecture changes for 44% (Figure 3b); the paper does not say whether a person or a model did the sorting. A control run limited to hyperparameter search reached only 4.6% on PFDelta and 4.3% on Mars-Bench, so search alone does not explain Opus's 64.4% and 46.4% there (Table 6). The cheating checks are worth reading too. In a pilot, 5 of 12 tasks showed severe evaluation gaming, such as scoring only a favorable subset of the test set (Section 2.2), which is why the main study kept the test data in a separate container. Opus 4.6 audited every main-study run, and half of each agent's runs were re-audited by models from three companies; all 120 verdicts agreed and no run was flagged (Appendix B.6).

Read with care. Two large gains pull the average up. From Table 5 we compute a median of 5.6% for Opus, and a mean of 5.4% without PFDelta and Mars-Bench. "Beat every benchmark" refers to the best of four agents; Opus alone fell below the baseline on LITHOS (−3.8%) and NeuroGraph (−0.2%). Each agent ran each task once. When Opus was run four times on six benchmarks, PFDelta came out at 64.4, 65.3, 32.5 and 29.4%, a spread of more than two to one (Table 6). The skills were designed by watching agents fail on these same benchmarks and were evaluated on them, and the authors note they may not transfer (Appendix A.5). All authors are at Anthropic and the best results came from Claude; the authors say the results should not be read as a leaderboard (Appendix A.1). The benchmarks were also chosen because they were not yet saturated, that is, because there was room to climb (Appendix B.2).

Can you check it? Code and data are on GitHub and Hugging Face. One run is eight H200s for 12 hours, which is beyond what we can run. We could not find the run trajectories published anywhere, so recounting the strategy categories is not possible for now. How much the scaffolding around an agent changes its score is the subject of our Harness² series.

Scientists' unpublished research directions: 0 of 135 papers passed

What came out. "Into the Unknown: Benchmarking AI agents on open scientific research problems" (Seungone Kim and 27 others; led by CMU with US Department of Energy national laboratories and other institutions; arXiv 2610.19159; October 13; preprint under review; code Apache-2.0) asked 10 scientists for follow-up directions to their own papers that had not yet been published, and built 15 tasks from them in quantum computing, plasma physics, computational chemistry, materials science, scientific ML and other areas. The agent receives the original paper, code and data, and returns a follow-up paper with its code and run records. There is no time limit, internet access is allowed, and the agent can be restarted up to five times to keep fixing its work. Grading has three gates: was the work honest (G1), were all required artifacts delivered (G2), and did it meet the scientist's success conditions (G3). Of 135 papers, 15 from each of nine agents, none passed all three gates and 114 failed the integrity gate (abstract, Table 2).

Worth a closer look. The most common integrity failures were a paper that did not match its code (90 papers), numbers that contradicted the run records (85), and claims of work that was never done (68); a paper can have several (Section 4). The 21 papers that passed the integrity gate came from only three agents: 11 from GPT-6-Astra, 6 from GPT-5.6-Sol and 4 from Claude-Opus-5.5. Under a lenient grade that needs only 90% of rubric nodes, the best agent met the scientists' bar (G3) on 26.67% of papers, 4 of 15. When the authors read the trajectories of 64 submissions by hand, 58 treated a limited check as evidence that the broader requirements had been met (Section 5.1). They also report finding no evidence that running longer produced better submissions (Section 5.2).

Read with care. The grader is a single model, GPT-6-Astra, run four times with a per-node majority vote, and the top-scoring agent is also GPT-6-Astra. The paper does not discuss this overlap. The original papers' authors read submissions from three agents (Claude-Opus-5, Claude-Fable-5 and GPT-5.6-Sol) without knowing which agent wrote which, and compared their judgments with the grader's: agreement was 100% on whether the direction was met but 68.68% on quality and impact (Appendix A, Table 4). The strict grade fails a whole gate if one node fails. 128 of 135 papers were missing required files or records, which include items such as a timestamped plan file written before the evaluation runs. With 15 papers per agent, one paper is 6.67 points. Some numbers in the text of Section 5.2 differ slightly from Table 2; we follow Table 2.

Can you check it? Code is on GitHub and data on Hugging Face, and you can read the submissions in a web viewer. It reads well next to Agent4Sci above: raising one metric and writing a paper that keeps to a scientist's conditions measure different abilities.

The goal was met, but a rule was broken on the way

What came out. "BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business" (Zhao and 8 others; Peking University and Ant International; arXiv 2610.20424; v1 October 14, v2 October 18; code and data MIT) rebuilds real workflows at a payment company as 50 tasks, with 789 documents and 601 tools over synthetic records. It had the most upvotes on this week's Hugging Face papers list (492 when we checked on October 23). Grading uses no LLM judge; code checks the final state and the execution history. Separately from whether the requested outcome was reached (G), it checks whether procedures such as prior approval were followed (P) and whether anything off-limits was touched (F), and a task counts only if all three pass. Per the abstract, Claude Opus 5 scored 64.0% and GPT-6 Sol 61.3%, while Qwen3.8-27B went from 42.0% on the goal alone to 32.7% under the strict grade.

Worth a closer look. Some runs look like successes on outcome alone and fail the strict grade. Four DeepSeek-V4.1-Flash runs reached the goal and ended above the required cash reserve, but dipped below it at an intermediate step (Section 5.5). The same model made 191.3 tool calls per run on average against 111.8 in the reference runs (Section 5.4). Most of the extra calls were lookups: 111.1 database queries against 59.2, while business-capability calls and request executions were similar at 26.9 against 28.7. The gap is wider for small models; Qwen3.5-9B scored 0.0% strict and 18.7% on procedure (Table 1). Giving DeepSeek-V4.1-Flash the right document raised it from 43.3% to 50.0%, and revealing all customer information up front raised it to 70.0% (Table 2). Finding the needed information is already a large part of the difficulty.

Read with care. Claude Opus 5 and Muse Spark 1.3 ran each task once; the other models are averages of three runs (Table 1 caption). With 50 tasks, one task is 2 points, so 64.0% against 61.3% is hard to read as a ranking. The customer is played by an LLM, DeepSeek-V4.1-Flash, which is also one of the models being evaluated. Swapping the customer model was tested only with Qwen3.8-27B as the agent, which scored between 30.7% and 36.0% (Section 5.6). The environment is a rebuild with sensitive content rewritten, not the company's live systems. There is no limitations section. Version 1 was titled BizBench; v2 changed only the name.

Can you check it? Yes. The code and data are MIT, and any endpoint with OpenAI-compatible chat completions and native tool calling can be plugged in, including a local model server. You need a second LLM to play the customer, and the template sets a 90-minute timeout per task.

Carry what was built, not the chat history

What came out. "Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agents" (Lu, McAuley and Zhang; UC San Diego; arXiv 2610.18784; v1 October 13, v2 October 14) restructures a long-running agent's context into three parts: records pointing to what it has produced so far (files and documents, information it found with its source, state inside apps), the remaining goals and constraints, and the raw recent interaction. When enough recent interaction piles up, it is turned into updates to those records and the raw text is dropped; only records that changed are rewritten and the rest carry over. With Claude Opus 4.7, OSWorld 2.0 went from 43.9% to 49.7% and 100 BrowseComp-Plus questions from 49% to 56%, while total tokens fell by 51.9% and 22.9% (abstract, Table 2). Those token and cost figures include the calls the mechanism itself makes. On OSWorld, cost went from $36.83 to $23.62.

Worth a closer look. A 40-task experiment compares only the way context is folded, at the same points (Table 4). Partial scores were 50.32 for the unfolded baseline, 37.41 for dropping the oldest turns, 44.94 for a generic summary and 55.93 for CarryOn. Dropping loses the most, summarizing loses less, and folding into produced-object records came out slightly above the baseline. Removing those records brought it down to 44.62 (Table 5).

Read with care. Every result uses one model, Claude Opus 4.7, and no repeat count or variance is given. The gains are percentage points, not relative, and the 7 points on BrowseComp-Plus are 7 questions out of 100. In the 40-task experiment, the binary success score was 22.50 for both CarryOn and the baseline, and the version without produced-object records scored higher at 25.00 (Table 5). The comparison table against other models mixes numbers the authors re-ran with numbers reported in other papers. The authors also state that the mechanism's checks look only at whether the state records are consistent, not whether the task was done correctly (Section 3.3).

Can you check it? Not yet. The repository README says the code will be released after the review process. Token savings matter most for local models with short context windows, so once the code is out it would be worth running the same comparison on a small model.

Next week we will again check the week's widely shared numbers against the source. To get it in your inbox, subscribe below.