Paper of the Week #9 — The Best Agent Passed 46 of 168 Physics Research Runs

Paper of the Week #9 — The Best Agent Passed 46 of 168 Physics Research Runs#

November 2-6, 2026 · FrontierPhysics · IdeaTaste · Route Anything · CoworkOS · LoCo

Forty-two physicists took problems from their own research, wrote them up as 56 tasks, and handed them to AI agents with up to 10 hours per attempt. The best configuration, Claude Opus 5.5 in Claude Code, passed 46 of its 168 attempts. When the physicists read the failed runs of another agent, the most common problem was not missing physics knowledge. It was reasoning that was not rigorous enough, such as an agent checking its answer with a method that relied on the same wrong assumption. A second paper this week found something similar from another direction. Agents asked to propose next year's research often picked the same problems researchers went on to work on, but rarely the approach the researchers actually took. Last week's issue (#8) asked whether a score looks at how an agent got its result. This week's first two items look at where agents doing research fall short. The other three are practical: choosing a model per request, a harness that fine-tuned another model, and a way to shorten reasoning. For each item we note what came out, the numbers we checked against the source, what to watch when reading it, and whether you can check it yourself.

Physicists wrote 56 tasks from their own research, and the best agent passed 27.4% of attempts#

What came out. "FrontierPhysics: Evaluating Agents for End-to-End Physics Research" (You et al., 73 authors; led by BenchFlow and UC Berkeley with Caltech, Harvard, Stanford and others; arXiv 2611.06910; November 4; CC BY 4.0) has 42 physicists write 56 tasks across nine areas of theoretical, computational and experimental physics. Each task gives the agent a short brief and a proposal, the way a lab head would hand work to a student, plus input files. The agent must return numerical results and a written paper. A run passes only if it clears every deterministic check and every "blocker" item in the expert rubric (Section 2.3). There are 22 models in 26 configurations, each run three times on every task, which is where 168 comes from (56 × 3). The best configuration, Claude Opus 5.5 in Claude Code, passed 46 of 168 runs, or 27.4% (95% bootstrap interval 17.9 to 36.9, Table 8). Counting a task as solved if any of its three runs passed, that rises to 44.6%. GPT-6 Astra (max) came next at 23.2%, and the paper marks it as not statistically separable from Opus.

Worth a closer look. The task authors read the three failed runs of GPT-6 Astra (xhigh) on their own task and wrote failure reports, which the organizers merged into 14 categories (Section 3.3). Across 143 failed runs, "not rigorous enough" appeared in 68 (48%), "lacked expert intuition" in 48 (34%), and literature problems in 39 (27%). Wrong physics appeared in 23 (16%) and execution errors in 6 (4%). One run can carry several labels. In one example, agents analyzing the error sources of a Rydberg atom experiment never considered Rydberg decay, which the authors say "an AMO physicist would naturally consider" (Table 4). The authors also describe a pattern of circular self-checks: "the agent's own validation method depends on its wrong assumption, so it can never find out its mistakes" (Section 3.4). Cost did not track results within one family. Among Claude models, pass rate went Opus 5.5 > Fable 5.1 > Opus 5 while cost per run went the other way (Section 3.2). Opus 5.5 averaged $26.95 per run (Table 10).

Read with care. Every run was graded by one reviewer agent, GPT-5.6 Sol (xhigh), and that configuration is also on the leaderboard. The authors regraded first-round runs with Claude Opus 5.5 as a second reviewer (Table 6). On GPT-6 Astra runs the two reviewers nearly agreed (25 and 26 passes). On Opus 5.5 runs, the Opus reviewer passed 44 where the default reviewer passed 37, and the authors say the Opus reviewer "is more lenient on rollouts from its own model family". The headline 27.4% uses the default reviewer. The failure categories come from one agent, GPT-6 Astra, and the authors say other models may fail in different ways. The reviewer also wrongly failed 9 of the 168 GPT-6 Astra runs; that correction was made only for GPT-6 Astra, and other models' numbers were not corrected the same way (Section 3.4). The bar is high even for people. Under the same rubrics, the authors' own reference materials averaged 96.2%, not 100% (Table 11).

Can you check it? Not easily. The paper cites BenchFlow's run tool (benchflow-ai/benchflow), but we found no link to the tasks or the run records it says were collected. One run can take up to 10 hours and some tasks need GPU machines.

Agents chose the right problems, but rarely the approach researchers actually took#

What came out. "IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?" (Zhong, Xu, Ning, Kumar and Xiong of Carnegie Mellon University with Liu of MBZUAI; arXiv 2611.03424; November 2; CC BY 4.0) gives agents the literature of a field up to 2025 and asks them to write research proposals. Those proposals are then compared with papers the same communities published from January to August 2026. Six models, four research areas and more than 90,000 proposals are covered. Two scores are used. Citation matching measures how much an agent's reference list overlaps with real papers' reference lists. Idea matching splits each abstract into parts such as problem, motivation and approach, and a judge model (GPT-5.6 Luna) checks each part against the closest real papers (Section 3.2). The abstract's result is that agents reach "about 26% of the human level on citation matching and only 18% on idea matching."

Worth a closer look. "Human level" needs unpacking. It is how much the 2026 papers overlap with each other, with each paper compared only against the other human papers (Section 3.2). That reference is itself low: 16.38 to 20.84 on citations and 10.53 to 13.87 on ideas (Table 2). So 18% does not mean the proposals are 18% as good. It means agent proposals overlap with real 2026 work at about 18% of the rate real papers overlap with each other. One case in Figure 4 shows the gap. The problem is the I/O cost of fetching mixture-of-experts weights that are not in cache. The real 2026 paper, ReMoE, fine-tunes the router to prefer recently used experts. One Qwen3.8-27B proposal attacked the same problem by predicting demand and prefetching experts without changing the model, which did not count as a match; another proposal from the same model trained only the router, and it did. The paper's summary: "a shared problem does not imply a shared approach." In a blinded review, 11 reviewers read 96 proposals and 96 real papers. Expected success scored 70.70 for papers and 50.39 for proposals, and reviewers judged 86.46% of the proposals to be AI-written (Table 3). Writing many more proposals did not close the gap. The curves "level off" well below the human level (Section 5).

Read with care. The authors themselves write that "high matching does not by itself imply higher research quality" (Appendix A.5). Llama 4 Scout had the highest idea match and the lowest review ratings (Section 4). The idea judge was validated on 40 pairs, all from one area (automated research), and agreement with three annotators ranged from 70.00% on motivation to 87.50% on problem (Appendix B.4). Each model's review score rests on 16 proposals. Some pipeline steps use DeepSeek-V4-Flash, which is also one of the six models evaluated (Table 17).

Can you check it? Partly. The leaderboard is public and will be updated, but we found no code or data repository linked from the paper.

A perfect model picker would gain 5.12 points; the trained pickers gained 0.30#

What came out. "Route Anything: A Study of Model and Policy Routing for General-Purpose AI Systems" (Zhou et al.; NUS HPC-AI Lab with Cardinal AI Lab, UC Berkeley and HKUST; arXiv 2611.05275; November 3; CC BY-NC-SA 4.0) asks how much you gain by choosing a different model, tool or plan for each request instead of always using the same one. Candidates include LLMs, image generators, robot policies, and plans like "try a cheap model, verify, and escalate if rejected." The authors ran 452 candidate configurations over 25 benchmarks for 374,401 scored runs. On 24 of them (19,814 test inputs), always using the best candidate for each benchmark scored 82.60, while an oracle that picks the best candidate for every input after seeing the results scored 87.98, a 5.38-point gap (Table 1).

Worth a closer look. The routers the authors trained got very little of that gap. On the 16 benchmarks where they trained a router, the best one scored 79.98, the best fixed candidate 79.68, and the oracle 84.80 (Section 4.1). The oracle is 5.12 points ahead; the router is 0.30 ahead. The abstract's 5.4 and 0.3 come from different benchmark sets, so the like-for-like comparison is 5.12 against 0.30. Other designs did no better. The best of 12 recipes that learn from a "winner" label on each input scored 78.87 against 79.44 for the fixed candidate, and on image generation none of 31 router designs beat the fixed generator with an interval above zero (Section 4.1). The authors give three reasons. Part of the per-input variation is noise "that no router can predict." Within a benchmark, one candidate tends to win almost every input type. And when they labeled the winner on each input, "a latency tie-break decides 94% of the winner labels" (Section 4.3), so the router was mostly learning which candidate was fastest.

Read with care. The abstract also reports that routing raises efficiency by 8.82× "at a quality loss below 0.5 points." That number is quality per unit of cost relative to a reference model, averaged over four testbeds: 25.5× on code, 7.1× on text question answering, 1.35× on image generation and 1.33× on robots (Table 5). It is not efficiency at equal quality. On image generation, quality fell by 0.47 points with an interval of −0.87 to −0.06, which does not include zero. Costs are in dollars except for robots, which use seconds of policy time. The oracle is a ceiling, not something you could deploy, because it uses the outcome of every candidate on every input.

Can you check it? Not yet. We found no link to the code or to the 374,401-run matrix in the paper. If you route requests between a local model and a hosted one, the cheap check this paper suggests is to measure how well always using your best fixed choice does before building a router.

A harness ran five rounds of fine-tuning on another model#

What came out. "CoworkOS: Scaling Agent Coworkers for Real-world Automation" (Jin et al., 40 authors; led by Renmin University of China with Fudan and Peking University, plus SJTU, NUS and ECNU; arXiv 2611.03178; November 2) is a harness for long tasks in which the model decides the strategy and the runtime manages execution. The agent can work directly or hand parts to subagents, keeps a workspace and memory so work can resume, and lets a person and the agent edit the same documents, slides, CAD models and Blender scenes. The authors compared it with three other harnesses (OpenHands, Codex and DeepSeek Harness), all using GLM-5.3, on 12 benchmarks in four areas (Table 1). The case study that drew attention is in Section 4.3. With GLM-5.3 in charge, CoworkOS fine-tuned Qwen3.6-35B-A3B for five rounds, and the average over eight benchmarks went from 33.19 to 42.95.

Worth a closer look. In the harness comparison the margins are small. CoworkOS led on two of four area averages (productivity 57.54 against 56.13, ML coding 56.26 against 56.14) and trailed on two (tool use 55.22 against 55.31, search 82.14 against 82.22). In the fine-tuning case, each round the agent evaluated the model, diagnosed weak spots, rebuilt the data pipeline, generated and filtered data, ran supervised fine-tuning, and evaluated again. The round-by-round averages were 33.19, 36.94, 35.90, 39.22, 40.71 and 42.95 (Table 2). After the dip in round 2, the review found that 42 trajectories made up 84.2% of the supervised tokens, so a few long examples dominated the training. AutomationBench rose the most, from 11.00 to 44.15. JobBench fell from 22.66 to 20.53.

Read with care. The starting score of 33.19 is not the untouched base model. "R0 is the baseline obtained through seed SFT from the base model" (Appendix A.3), using 1,345 trajectories prepared before the agent took over, and the base model's own score is not reported. The agent diagnosed gaps on the same eight benchmarks it was scored on, so the rise measures how well it tuned toward those benchmarks. The only filter against near-duplicate training data was BM25 similarity with a 0.8 threshold. There is one run, no comparison with a human-written fine-tuning recipe, and no compute figure. The authors note that the changes in each round "describe concurrent interventions rather than isolated causal effects."

Can you check it? The code is on GitHub under MIT. Repeating the fine-tuning case would need a model the size of GLM-5.3 as controller and GPUs to train a 35B model, so we did not try it.

A 7B reasoning model learned to stop about 44% sooner on math#

What came out. "Think Less, Locally: Local Preference Learning for Reasoning Compression" (Yuan et al., Xi'an Jiaotong University and WeChat AI, Tencent; arXiv 2611.04814; November 3) proposes LoCo, a way to shorten the answers of reasoning models that keep going after they already have the answer. The model reads its own reasoning step by step and flags steps where its confidence is low and swinging. A second model (Qwen3-30B) confirms which flagged steps are overthinking. From each confirmed point, the model writes eight new continuations, and the shortest one that still reaches the correct answer becomes the preferred example over the original, longer one. DPO training on these pairs uses only math problems the model solved itself. The paper trains one model, DeepSeek-R1-Distill-Qwen-7B. Answers got 41.7% shorter on AMC23 and 45.4% shorter on AIME24, with the other three math benchmarks in between (Table 1).

Worth a closer look. Accuracy held or rose on four of the five math benchmarks: AIME24 went from 50.83 to 54.17 and AIME25 from 37.92 to 40.42. MATH-500 dropped slightly, from 91.88 to 91.80. The training was math only, but answers also got shorter on code: 36.3% on MBPP with the same accuracy (84.05), 24.7% on LiveCodeBench v5 and 23.4% on CodeContests, with accuracy slightly up on both (Table 2). Against two published methods evaluated from their released weights, DAST shortened answers by 2.0% to 7.4%, and TokenSqueeze by 22.5% to 39.2% while losing accuracy on every math benchmark (Table 1). Where the cut falls matters. Cutting at random points shortened answers about as much but lost accuracy on AIME24, while LoCo's cut points fell late in the reasoning, on average 57% of the way through (Table 3, Appendix A.3).

Read with care. It is one 7B model. Each problem was sampled eight times at temperature 0.6, and the tables show ± values without saying what they are. AIME has 30 problems per year, and the ± on AIME is about 4 points, so the +3.34 and +2.50 gains are within that spread. The paper has no limitations section, and the number of preference pairs and the GPU hours are not given. Training used LoRA on four H20 GPUs.

Can you check it? Yes. The code and the merged 7B checkpoint are MIT, and the repository includes the pair-building code, the source math problems and the five math test sets, though not the finished preference pairs. The released evaluation scripts cover math only. A 7B model fits on one GPU, so comparing answer length against the base model on AIME is within reach of anyone running reasoning models locally.

Next week we will again check the week's widely shared numbers against the source. To get it in your inbox, subscribe below.