AI Research•SOTAAZ Lab••KR

We connected PaperOrchestra to a local model. The PDF compiled; its evidence still needed checking.

A constrained run of official PaperOrchestra components produced a PDF with one changed numeric cell and two unresolved citation keys. A second seed failed at outline parsing. We audit the logs, manuscript and simulated review.

We connected PaperOrchestra to a local model. The PDF compiled; its evidence still needed checking.

The first generated paper compiled successfully. Its table also changed a measured 59.99% into 60.00%, and it introduced two citation keys absent from the supplied bibliography. The automated reviewer criticized a verified 2026 reference for being from the future. We ran the experiment on November 16, 2026.

We used the official PaperOrchestra outline, section-writing and AgentReview components with a local Qwen3.5 9B backend. This was a constrained integration test, not an untouched reproduction of the full search, plotting and refinement pipeline. The model substitution matters: failures from this small local configuration are not estimates of the original paper's system-wide error rate.

Our question was concrete: when a writing system receives a measured experiment with known numeric answers and explicit limitations, which facts survive into the manuscript? We kept the input logs, generated LaTeX, PDF, compiler output and review so those boundaries could be inspected.

The unit of work is larger than a writing prompt

PaperOrchestra takes an idea, experimental logs, a paper template and guidelines. An outline component plans sections, literature needs and figures. Other components gather sources and prepare visual material; a writer assembles the manuscript; simulated review informs refinement. The outputs include LaTeX and a PDF.

Much of the scientific substance already exists in the input. The user supplies what was proposed, which comparisons were run and what happened. This distinguishes manuscript production from discovering an idea or executing a research program. It also separates transcribing a measured value from judging what that value supports.

Intermediate artifacts make the system inspectable. An outline can be revised before prose is written; a citation collection can be checked independently; a table can be compared with source measurements. Those same boundaries are places where errors propagate. A plan can ask for an experiment that was never run. A writer can add a citation key that the bibliography does not contain. A PDF can exist despite either problem.

We therefore chose a small input with a tractable truth set, rather than asking whether an impressive-looking document felt publishable.

Our input and execution conditions

The input was our recorded synthetic selection-bias simulation. Every candidate has true accuracy 60%; the candidate with the highest score on a selection set is chosen; its score is also measured on an independent test. This illustrates a known statistical effect. It is not a novel optimizer or an experiment on real language-model harnesses.

We supplied the actual JSON results for candidate counts 1, 8, 40 and 328; 5,000 repetitions; 100 selection questions; and 1,000 independent test questions. The log included Python and NumPy versions, seed 20261116, first-index tie breaking and the independence assumptions. It explicitly stated that no LLMs, real harness changes, deployment throughput, financial costs or comparisons with published algorithms had been measured in that source experiment.

That last distinction can be confusing: an LLM wrote the manuscript in the present test, but the experiment described inside that manuscript was a synthetic simulation. We did not let the writing run turn the source simulation into a claim about real agent performance.

We supplied one verified reference, Self-Harness, and an empty figure list. We did not run literature search or plotting, so this tests compliance with a fixed citation inventory rather than automatic reference discovery.

ItemLocal condition
Official codeCommit d9bae997e793a015ed38bb1605be9eb58c00284f
Executed componentsOutlineAgent, SectionWritingAgent, AgentReview
ModelOllama qwen3.5:9b, Q4_K_M, approximately 9.7B in metadata
ServerOllama 0.40.2 on local A100 hardware
Context24,576 tokens
SamplingTemperature 0.7; seeds 101 and 202
ReferencesOne supplied and verified entry
Review ensembleOne

Every local request and response was retained. This table describes our integration, not the original paper's evaluation configuration.

Changing the writer model does not reroute every provider call

The official code exposes model options, but provider routing is not uniform across all components. The outline agent calls a Gemini utility directly. Literature search also specifies a Gemini model and Google search tooling. Replacing a writer model name does not by itself make the entire pipeline local.

Our native Gemini-client preflight failed because credentials were unavailable. We used the official components' prompt construction and data assembly, then connected their model-call functions to the existing local server. The adapter is included with the evidence. We did not substitute fake search results and report that search had run.

Dependencies required attention too. The initially installed pymupdf4llm 2.1 package did not supply the module expected by this checkout. Pinning 0.0.27 supplied the needed implementation. This is a record of our environment, not a recommendation that every reader install that version.

These details matter because successful orchestration depends on more than the intellectual division of labor between agents. Provider features, parsers, package versions and intermediate schemas form executable contracts between those roles.

One of two attempts reached a PDF

Seed 101 completed outlining, writing, compilation and review. Seed 202 returned an outline that the official JSON parser could not read. Its raw response contained invalid backslash escaping, and the parser's cleanup path did not recover it. The output was not truncated.

SeedOutlineManuscript and PDFReview
101ParsedPDF produced; all four compile commands returned 0Valid on the first response
202Response received, parsing failedNot reachedNot reached

This is not a meaningful estimate of a production success rate. It is two attempts with a modified backend. Our adapter also did not copy the native Gemini wrapper's up-to-five-attempt syntax-failure retry behavior. The adapter allows one retry for output truncation, not arbitrary malformed JSON; none of the four recorded paper calls used that retry. The second failure is conditional on that adapter policy; we do not claim the untouched pipeline necessarily stops there.

The content audit below applies to the one completed manuscript. The failed second response remains in the package. Reporting only the completed PDF would conceal a different kind of outcome: no manuscript to audit.

The first page of the actual generated PDF, not a designed mockup.

Twelve numeric cells, eleven matches

We audited the four-row result table's selection mean, independent-test mean and gap: twelve numeric cells. The comparison converted the recorded values to percentages and rounded to two decimal places. Eleven matched the input-derived values.

Candidate countInput test mean, roundedGenerated manuscript
159.97%59.97%
860.00%60.00%
4059.98%59.98%
32859.99%60.00%
An audit of the independent-test column against the input log.

For 328 candidates, the raw mean is 0.5999444. As a percentage that is 59.99444%, which rounds to 59.99%, not 60.00%, at two decimals. The manuscript did not apply the same rounding rule to that cell.

This tiny discrepancy does not reverse the simulation's conclusion: independent-test accuracy stays near 60%. It does expose an evidence-preservation issue. A measured value should not be silently replaced by its theoretical expectation, even when the two are close. In a smaller-effect comparison, the same behavior could change a rounded tie or a ranking.

We cannot infer the model's internal reason for the substitution. The audit establishes an output mismatch, not whether it deliberately “cleaned up” a number. A practical remedy is to render numeric tables from structured data, then let the model explain those tables. Units and rounding become code rather than a prose-generation decision.

Omissions need their own check. The manuscript discussed selection and test sizes symbolically but omitted the concrete 100 and 1,000 values from the input. It retained details such as the seed and software version while losing sample sizes needed to interpret uncertainty. A table-value audit alone would miss that.

Compiler success and citation integrity diverged

The writer added Paszke2019PyTorch and VanDerWalt2011NumPy. Neither key appeared in our supplied BibTeX file. BibTeX warned that it could not find the entries, and the final LaTeX log retained unresolved-citation warnings. The commands still returned zero and a PDF existed.

This does not establish that the underlying papers are imaginary. The observed failure is a broken link between manuscript citation keys and the provided bibliography. PyTorch was also not a tool used in the source simulation, which used Python and NumPy. A familiar citation can be inappropriate even when it refers to a real publication.

The writer changed the case of the Self-Harness key, but BibTeX resolved it in this run. We did not count that cosmetic difference as another unresolved reference. We checked actual compiler behavior rather than treating every string difference as equivalent.

A simple preflight can compare normalized citation keys against the supplied bibliography:

python
allowed = {key.lower() for key in supplied_bib_keys}
unknown = sorted({
    key for key in manuscript_cite_keys
    if key.lower() not in allowed
})

Passing this check does not prove that a source supports the sentence citing it. It handles a narrower deterministic problem before asking a reviewer to evaluate meaning.

The reviewer did not receive the source log

The review generated 1,990 tokens within its 2,500-token limit and stopped normally. The official parser read the completed review as Overall 5 and Decision Reject. These are simulated ratings, not an actual conference decision.

Some criticism was reasonable: selection bias is already known, and a synthetic illustration does not establish a new research method or capture every property of real model selection. Those limits were explicitly present in our input.

Other criticism relied on a false date assumption. The reviewer repeatedly described the 2026 reference as future work, although the actual run was in November 2026 and the referenced paper had been verified. The prompt did not state the current date, leaving the reviewer to infer one. Critical-sounding prose can contain unsupported premises too.

More fundamentally, this review call received the generated manuscript text, not the original experiment log. The official content-refinement path's peer-review function similarly reads PDF text into the review call. A later refiner's access to logs is a different boundary. A reviewer lacking the original 59.99% cannot detect its replacement by 60.00% through source comparison.

This suggests a concrete separation of checks. Simulated review can challenge reasoning, missing comparisons and overbroad claims. Numeric provenance, bibliography membership and required-field preservation should have their own access to source artifacts. A longer review prompt cannot compensate for evidence the reviewer never sees.

What the original evaluation does and does not establish

The original PaperWritingBench evaluation uses 100 accepted CVPR 2025 papers and 100 accepted ICLR 2025 papers. Ideas and experimental logs are extracted backward from completed papers. That is different from receiving unstructured notes from an unfinished research project.

The main sparse condition supplies a short idea description. The principal PlotOff comparison provides existing paper figures to systems, including baselines without figure-generation support; PlotOn is evaluated separately. A system's ability to generate figures and the figure condition behind a reported score should not be conflated.

Table 2's acceptance rates are automated judgments. Under the same sparse, PlotOff condition, AI Scientist-v2 Reviewer gives PaperOrchestra 48% on CVPR and 22% on ICLR, while ScholarPeer gives 84% and 81%. Those are not real publication acceptance rates. Their dependence on the evaluator is itself relevant to interpreting the claim.

Our small run cannot explain the difference between those reviewers. It illustrates why a review score and source fidelity should be measured separately: the reviewer is another generated artifact with assumptions that can be checked.

A more inspectable writing workflow

We would attach an experiment identifier, unit, denominator and rounding rule to each numeric field. Tables and plots would be rendered from those fields. Missing measurements would remain explicitly missing, with no invitation to fill plausible values.

Completion would have several recorded states: source generated, PDF compiled, citations resolved, numeric audit passed, required conditions preserved, and substantive claims reviewed. Our first attempt passed the first two while failing later checks. One undifferentiated “done” flag would erase that information.

We would also verify factual premises in review comments before applying them. Dates, reference existence and experiment configurations are checkable. A confident reviewer can invent an error while overlooking a real one.

The result of this limited experiment is not a verdict on PaperOrchestra as a whole. It is an inspectable example of why manuscript quality and evidence preservation need distinct mechanisms, even when the writing pipeline produces a convincing PDF.

Execution materials

The download contains the adapter, input logs, generated LaTeX and PDF, compiler records, numeric and citation audit, raw model calls, and the pinned official code revision. The generated PDF is an experimental output to inspect, not a manuscript we recommend submitting.

Updated November 16, 2026: added execution of official components through a local backend, artifact auditing, failed-attempt reporting and new visuals.

Subscribe below for research notes that include the outputs and failures behind the claims.

Files for this post

Reproduction package

The harness, raw logs and result CSVs behind this post. No account needed; the README has a command that recomputes the tables without a GPU.

paperorchestra-repro.zip · 4,655 KB

Download

Stay Updated

Follow us for the latest posts and tutorials

Subscribe to Newsletter