Can You Trust a Decision Model's Confidence? Repeated Answers and Probabilities, Measured

Can You Trust a Decision Model's Confidence? Repeated Answers and Probabilities, Measured

The pitch for fast decision models like Jev is not only speed. Every answer comes with a probability, so you can act on the answers the model is sure of and send only the unsure ones to a larger model or a person. That only works if "sure" means something.

There are two common ways to judge it. You can ask the same question several times and trust the answer when the repeats agree, or you can read the probability and set a threshold. I tested both on the tasks from my earlier comparison, scored against human labels, and added CLM, a decision model released this week.

Repeated draws: most wrong answers were unanimous

openjev reads its answer from DiffusionGemma after one denoising step, starting from random noise. Different noise can give a different answer, so the server can read several times and average. I asked for each message 8 times with 8 different noise seeds, the way openjev itself varies them, and kept every answer.

Disagreement is a real warning. When the draws split, the answer was right less often: 2 of 13 on BANKING77 and 37 of 58 on TREC, against 70% and 84% when all 8 agreed. But the draws rarely split. When openjev was wrong, it was mostly wrong the same way every time, so agreement alone could not pick out a set of answers that was 95% correct (table below). As a predictor of correctness, agreement scored an AUROC of 0.59 on BANKING77 and 0.57 on TREC, where 0.5 is a coin flip. The probability of the first draw did better, at 0.72 and 0.73.

The effect of a majority vote depended on the task. Taking the majority of 8 instead of the first draw changed nothing on BANKING77 (100 against 102 correct, p = 0.5) and helped slightly on TREC (408 against 400; 9 against 1 on the questions only one got right, p = 0.02).

Agreement is still useful for something else: deciding when to stop. Reading two draws and stopping when they agree, otherwise reading all eight, needed 2.27 reads per message on average on BANKING77 and 2.24 on TREC. In these samples, skipping the extra draws when the first two agreed gave almost the same number of correct answers as the majority of 8 (100 against 100, and 406 against 408). It saves draws. It does not tell you which answers to trust.

A recent paper, Stable Answers, Unfinished Reasoning, reaches a similar conclusion in a different setting. It tests stopping a reasoning model once repeated probes agree, and finds that agreement "establishes that the current answer persists under a fixed probing procedure", not that the answer is finished. Here the equivalent is that agreement shows an answer survives different noise, not that it is right. The settings differ, though. The paper is about cutting off reasoning that is still in progress; here the read is already finished and the question is only how many more times to repeat it. Stopping early is fine in the second case, and I only showed it for these samples.

Probabilities: how useful they were varied sharply by task

The practical question for routing is how much you can hand off automatically without letting too many errors through. So for each model I found the largest share of answers that clears some probability threshold while the accepted answers stay at least 95% correct. These are the best shares found by searching thresholds on the same samples they are scored on. Whether 95% holds on new data needs a separate check.

On TREC, laya's probabilities were almost exactly right: of the 314 answers it gave at 90% or more, 305 were correct, and its calibration error was 0.02. That is what the routing pitch promises, and here it held.

On BANKING77 the same model stayed just as confident while being wrong most of the time. It gave 108 answers at 90% or more and only 48 of them were correct. At no threshold did the accepted answers reach 95%. (That is laya at its defaults; at the 512-token option budget its documentation recommends for many options, 60 of 107 were correct, and still nothing could be auto-accepted.)

openjev sat in between on both tasks. The "mean over 8" row averages the top probability of each draw and scores the most frequent answer; when draws pick different options, it averages probabilities of different options, which is not the same as averaging the distributions and then choosing. That way the auto-accept share was lower than with the first draw (28.6% to 14.9% on BANKING77, 26.4% to 0% on TREC), because answers that were unanimously wrong also had high probabilities. The two rows also choose answers differently, so this does not show that averaging probabilities makes things worse in general.

CLM: fast, but near chance with label names

CLM (Contrastive Language Model) was announced this week as a System One model with "up to 9× lower latency" than Jev at similar quality on computer-use, gaming and tool-calling tasks. It embeds the state and each option with Qwen3-8B, maps both through a 20M-parameter head trained with a contrastive objective, and picks the option closest to the state. I ran the published head (CLM-v0.1-8B) on the same three datasets, labels and instructions as the other models.

With label names only, that is close to random choice. On AG News it answered "Sports" for 188 of 200 articles. On TREC with descriptions it answered "description" for 420 of 500 questions. Descriptions help, because CLM compares the state with the option's text and was trained on answer sentences, not one-word labels, but even 60.5% on AG News is well below laya (94.5%, although AG News is in laya's training data) and openjev (85.0%).

I checked my setup before trusting this. The Qwen3-8B embeddings from my vLLM server nearly match a direct transformers run (cosine 0.9999 on the test sentences), and CLM's own ranking example comes out close to its README (0.993 against 0.997). But its README's typed-question example does not reproduce exactly: I get 0.84 for "Is this urgent?" where the README prints 0.41. The repository is new and had no issue reports when I checked, so a head or version mismatch on my side cannot be ruled out. What I can say is that the published head, served as documented, did not do text classification well. Its own claims are about agent actions, which I did not test.

What to do with this

• Measure the auto-accept share on your own data before you route. The same model went from 88% to 0% between two public datasets.
• Use agreement between repeats to save compute, not to decide trust. Two draws that agree are a fine reason to stop drawing. They are not a reason to believe the answer.
• A high probability does not guarantee a correct answer. laya's documentation says it ships over-confident and suggests fitting a temperature on your data. Here it was well calibrated on one task and badly over-confident on the other.

What this does not show

Two tasks for the repeated-draw test, one DiffusionGemma-based server, and 8 draws. Thresholds were chosen and scored on the same samples, so the auto-accept shares are descriptive, not a promise for new data. CLM was run from the reference head in its first week; its results may change with a different head or setup.

Setup: one A100 80GB, driver 580.178.04. openjev 0.4.0 engine called in-process against vLLM 1b3b88e serving google/diffusiongemma-26B-A4B-it bf16 (--async-scheduling, canvas 64); per-message draws use seeds base + k × 7919 for k = 0..7, as openjev's own samples option does. laya 0.3.7. CLM repository cca045f, head Contrastive-LM/CLM-v0.1-8B revision 87655cb, Qwen3-8B served by the same vLLM with --runner pooling. Same samples as the earlier posts: BANKING77 154 (seed 20260918), TREC test set 500, AG News 200. AUROC and calibration use each model's top-option probability. McNemar tests are exact and two-sided. Measured 2026-09-24.