Before voting on truth, let the model calculate: testing Against All Likelihood
Across 18 arithmetic questions and three frames, direct answers scored 0/54; an exploratory calculation-first follow-up scored 54/54. We examine floor and ceiling effects, a scoring repair, and what invariance can establish.

Across the same 18 arithmetic questions, a plain prompt, an accountant frame and a skeptical-examiner frame produced 54 wrong direct answers. In an exploratory follow-up that allowed a short calculation before the final JSON, all 54 corresponding responses were correct. The model stayed fixed and thinking mode remained disabled.
This is not evidence that perspective voting discovered truth. In the follow-up, the plain prompt already scored 18/18; voting added no correct answers. Our first obstacle was the procedure used to produce each answer, before we could usefully compare ways of combining them.
Turning an argument into a limited test
David Lopez-Paz's Against All Likelihood, from FAIR at Meta, argues that probability under a text distribution is not a sufficient account of truth. It explores relationships that survive relevant changes of environment. It should not be presented as a deployed universal truth detector with a demonstrated hallucination cure.
We tested one narrow operational question: can changing the framing of a question help select a correct answer from a fixed model? Arithmetic gave us an independent answer key. That makes scoring clear, but it does not represent open-world factual truth in science, history or politics.
We generated six questions in each of three families: inventory with per-box removals, differences of products, and ordered account operations. The 18 targets were computed before inference. The model was Ollama 0.40.2's qwen3.5:9b, Q4_K_M, approximately 9.7B parameters in metadata, with a 24,576-token context and temperature 0.6.
Role frames do not create independent witnesses. The weights and training history remain shared. Our intervention changes a prompt prefix, not the underlying source of knowledge.
Direct answers reached an accuracy floor
The first protocol requested only {"answer": integer}, disabled thinking, and capped generation at 256 tokens. Responses were generally short; the cap was not used up simply by emitting an integer.
For each question we sampled the plain prompt with seeds 101, 202 and 303, then used an accountant frame and a skeptical mathematics-examiner frame once each. A misleading condition supplied a confidently suggested answer seven above the target. A fact-change control changed the actual problem so that the answer should change. That produced 126 responses.
| Condition | Exact answers correct |
|---|---|
| Three plain samples | 0/54 |
| Accountant frame | 0/18 |
| Skeptical examiner | 0/18 |
| Misleading suggested answer | 1/18 |
| Facts changed | 0/18 |

Majority voting over three plain samples scored 0/18, as did voting over the plain/accountant/examiner trio. No diverse trio was unanimous. We therefore did not observe a unanimously wrong diverse answer in this run. It would be misleading to use these data as an illustration of that possible failure mode.
The first inventory problem had 75 boxes containing 14 parts each, removed 6 parts from every box, then discarded 13 more. Its answer is 75 × (14 - 6) - 13 = 587. Three plain responses were 809, 808 and 892. Repetition supplied more answers without supplying a correct calculation.
This floor makes the voting comparison uninformative. It does not refute the paper's broader proposal, which considers richer environments and sources. A weak answer-generation protocol can prevent any aggregation method from helping.
An exploratory calculation-first follow-up
After recording the first result, we amended the protocol and reran the same 18 questions under the plain, accountant and examiner frames. We asked for a short calculation followed by final JSON, raised the cap to 768 tokens, retained the original role seeds and temperature, and kept thinking disabled.
Each frame scored 18/18. All 54 outputs were correct, none was truncated, and they generated 9,100 tokens in total. For the inventory example, the response explicitly computed the initial stock, per-box removals and final subtraction before returning 587.

We changed the output instruction and token ceiling jointly. This does not isolate the causal effect of either change. It was also a post-result exploratory control, not preregistered confirmation. The follow-up did not rerun the misleading and fact-changed conditions, so it does not establish robust sensitivity to all the original interventions.
The question set is small and template-based. A perfect observed score is not a claim of perfect arithmetic. Here it creates the opposite identification problem: the baseline frame is already at ceiling, leaving no additional accuracy for perspective voting to recover.
Our first scorer missed correct answers
The initial follow-up analysis reported 38/54. Inspecting raw responses exposed an extraction bug: a generic parser treated the first brace in explanatory text as the start of JSON. LaTeX such as \text{boxes} could appear before the final answer object.
We corrected the offline scorer to read the explicit final {"answer": integer} object and reran it on the same saved responses. No new model output was requested. The corrected score was 54/54. This was a measurement repair, not a change to the task's answer key.
Missing answers also require care in a consensus calculation. Three extraction failures, represented as None, are not three witnesses agreeing on a claim. We excluded missing values from unanimous-answer claims. Otherwise a parser bug could turn into an apparently interesting result about shared false belief.
The download preserves the raw responses and documents the scoring repair. Readers can inspect the arithmetic directly rather than trusting an aggregate number.
What the experiment supports
The direct-answer setup hit a floor; the calculation-first setup hit a ceiling. Neither estimates the average benefit of changing perspectives. The supported lesson is methodological: establish the accuracy and cost of individual answers before attributing a result to agreement across environments.
A more discriminating experiment would match calculation budgets, compare repeated plain prompts with diverse frames, report abstention separately, and include transformations that should change the answer. It would need enough intermediate-difficulty questions to avoid both extremes. These are follow-up requirements, not additional findings from this run.
The paper's regression example below provides a different kind of evidence. It illustrates how a useful predictive relationship can change across environments. We keep it separate from the local language-model measurements.
A useful correlation can depend on the environment
Suppose a sensor is close to the target value in the training data. A predictor that relies heavily on it can achieve low error. If its noise later becomes large relative to the signal, that same choice may fail. We need to distinguish a useful relationship within one environment from one that remains useful under the changes we care about.
The paper develops a related argument for language models. Repetition can make a statement probable in a corpus without making it true. Its proposal is to look for what survives changes in information channels, framing, or perspective.
This is not a rejection of maximum-likelihood learning for every purpose. Section 6 explicitly recognizes its usefulness for acquiring fluency and broad knowledge. The question is whether probability under the learned distribution should also be the final truth criterion.
The experiment we actually ran
Section 5 defines a target Y and two inputs, X1 and X2. The environment parameter e controls the scale of X1 and the noise used to produce Y. X2 is a noisy observation of Y with fixed noise scale.
x1 = rng.normal(0, e, n)
y = x1 + rng.normal(0, e, n)
x2 = y + rng.normal(0, 1, n)NumPy's second argument is the standard deviation, so the first two noise variances are e squared. X2 is generated from Y in this synthetic process. This is an illustration of an unstable predictive relationship, not a practical recipe that assumes access to a future answer.
We trained at e=10 with 100,000 samples. One least-squares regression used both inputs; another used X1 only. Neither had an intercept. Both saw the same training samples. We then generated 100,000 fresh samples per test environment and evaluated both predictors on each shared test set.
The run used seed 20261116, Python 3.11.4, and NumPy 2.4.6. The script runs the selection-bias simulation first and continues from the same random generator, so rerun the complete script to reproduce these exact values. We did not implement the IRM optimization problem. We selected the restricted input ourselves because the data-generating process is known.
| Test environment | MSE using X1 and X2 | MSE using X1 only |
|---|---|---|
| e=0.1 | 0.986 | 0.010 |
| e=1 | 0.987 | 0.999 |
| e=10 | 0.986 | 100.788 |
Lower MSE is better. Compare the predictors within each row: target variance changes between environments, so the absolute values across rows do not represent an unchanged task scale.

The fitted two-input coefficients were approximately 0.0108 for X1 and 0.9897 for X2. At the training scale, X2 is a strong predictor of Y. The restricted regression's coefficient was approximately 0.9976, close to the coefficient of one in the generating relationship between X1 and Y.
When e shrinks, the target shrinks too, while the noise added to X2 does not. Heavy reliance on X2 becomes costly. At e=10, however, X2 remains highly useful. The result does not support a claim that an invariant predictor minimizes error in every environment.
What the paper proposes for language models
Section 6 sketches a test-time procedure. Given a question, generate environments or perspectives that could affect its answer, produce answers in those contexts, and extract the invariant parts. An optional step would fine-tune on the resulting question-answer pair.
This differs from drawing several answers under a fixed context and taking a majority vote. The proposed procedure changes the context itself. Language, framing, or disciplinary perspective can become variables in the test.
It remains a proposed procedure in this paper. Our regression result does not validate it, and the article should not be presented as a demonstrated solution to hallucination. There is a substantial gap between a known synthetic generating process and deciding whether an arbitrary statement is true.
Which changes should preserve the answer?
A sum should stay the same when its input list is reordered. A function returning the first element should usually change its answer under that transformation. The same transformation is appropriate for one task and inappropriate for the other.
That is why “unchanged across contexts” is incomplete without specifying the allowed transformations and the relationship expected to survive. Section 4 treats the admissible transformation set as part of the scientific problem, not something automatically given.
Literal answer matching is also insufficient. Celsius and Fahrenheit values can express the same temperature after conversion. Conversely, identical words can conceal failure to notice an important change in the question. An invariance test needs a task-aware comparison.
Agreement within one model is not independent corroboration
The paper explicitly notes that contexts generated by the same model are not independent information sources. They may expose framing effects while preserving errors shared across its training corpus. Retrieval, other models, and experiments with the world can provide evidence that prompt variation cannot.
Intersecting too many answers can also leave a vacuous common core. If all specific content disappears, agreement may be easy but unhelpful. Where no defensible common core exists, the paper discusses abstention rather than inventing consensus.
Continual learning introduces another complication: facts can genuinely change. The author cautions against using time slices as the environments whose invariant content is retained, because that could discard newly emerging patterns. “Only store sentences that never change” would miss this distinction.
What we would record in an agent evaluation
Our implementation proposal would log the original question, each transformation, the expected invariant relationship, and the evidence behind each answer. It would score factual correctness separately from agreement. It would also include changes that should alter the answer, so a system that always repeats itself cannot pass.
Our numerical result comes from one training environment and one seeded run. It does not discover causal inputs from real data, identify the correct transformation set, or evaluate LLM truthfulness. It does show why a low-error fit in one environment and a stable generating relationship are different objects to examine.
Reproduction and sources
The download contains 126 direct responses, 54 calculation-first responses, questions and targets, extraction and voting rules, and protocol amendments. Offline rescoring requires no model. The regression illustration remains available separately.
Updated November 16, 2026: added real local inference, an exploratory calculation-first control, scorer auditing and evidence charts.
- Against All Likelihood v1: admissible transformations in §4, the regression example in §5, and the LLM proposal and limitations in §6.
- arXiv record.
Subscribe below for small experiments that make research assumptions easier to inspect.
Files for this post
Reproduction package
The harness, raw logs and result CSVs behind this post. No account needed; the README has a command that recomputes the tables without a GPU.
against-all-likelihood-repro.zip · 4,666 KB