Kev vs MoJev: Two More Open Jev Alternatives, Accuracy and Confidence Measured

Kev vs MoJev: Two More Open Jev Alternatives, Accuracy and Confidence Measured

Two more open models that answer Jev's typed questions have appeared since my last comparison. Kev is a family of LoRA adapters on Qwen3.5 and Qwen3.8 models, in 0.8B, 4B, 9B and 27B sizes, and each checkpoint ships with a fitted temperature so its probabilities are calibrated by default. MoJev is a single 0.85B model that scores every option in one forward pass and reports 93.23% accuracy with 0.79% calibration error. Both accept requests in the same format as Jev's System One API.

I ran both on the same tasks as the earlier comparison and the confidence test: BANKING77 (154 messages, 77 options), TREC (500 questions, 6 options) and AG News (200 articles, 4 options), scored against human labels, with the same label names, instructions and one-line descriptions.

One fact decides how to read everything below. Kev was trained on the train splits of all three datasets. Its README lists BANKING77, TREC and AG News among its "trained sources", and its data code draws training examples from each dataset's train split. My samples come from the test splits, so Kev never trained on them, but these are the tasks it knows best. The fair comparison for Kev is therefore not laya or openjev, which were not trained on these tasks, but a small classifier trained on the same train splits. I had one from the first post: MiniLM embeddings with logistic regression.

Accuracy

Latency is the median for one request at a time on one A100 over local HTTP. The slower end of each range is BANKING77, which sends 77 options. The trained classifier does not use names or descriptions, so it has one number per dataset.

Against the trained classifier, every Kev size lost on BANKING77 and tied on AG News. On TREC, Kev was ahead except for the 4B with label names only. On BANKING77 the classifier got 138 of 154 right and Kev-9B 127 (on the messages only one of them got right, 18 against 7, p = 0.04). On TREC with descriptions, Kev-9B got 483 of 500 against the classifier's 452 (37 against 6, p < 0.001). On AG News the two were within four articles of each other (p ≥ 0.45).

There is one more overlap to check. Kev picks checkpoints on a development partition and fits its temperature on a calibration partition, and both are drawn from the datasets' test splits. By matching the text, 80 of my 500 TREC questions, 6 of my 154 BANKING77 messages and 1 of my 200 AG News articles appear in those partitions. They were never trained on, but they did help choose the model. Leaving them out changes little: on the remaining 420 TREC questions with descriptions, Kev-9B scored 96.9% against the classifier's 91.0% (p < 0.001). Kev-9B's names-only lead over the classifier is no longer significant there (93.8% against 91.0%, p = 0.10), and Kev was not more accurate on the overlapping questions than on the others.

Size helped on BANKING77 and hardly anywhere else. From 0.8B to 9B, BANKING77 went from 113 to 127 correct (p = 0.004), while AG News moved from 174 to 178 (p = 0.34). 4B and 9B were close on BANKING77 (125 against 127, p = 0.79). The 4B model was the odd one on TREC with label names only, at 450 against 0.8B's 473 (p < 0.001), and adding descriptions brought it back to 477.

MoJev was far behind on the two tasks with more than four options and was the only model here that descriptions made worse: TREC fell from 240 to 182 correct and AG News from 165 to 145 (both p < 0.001).

Probabilities: Kev's were usable, MoJev's were too low

As in the confidence post, the routing question is how much you can accept automatically while the accepted answers stay at least 95% correct. I searched thresholds on the same samples the shares are scored on, so they describe these samples and are not a promise for new data.

BANKING77 is where the difference shows. With the same 77 options, laya's probabilities could not clear 95% at any threshold and openjev's first draw reached 28.6%. Kev-0.8B could hand off 84 of 154 messages, and all 47 answers it gave at 90% or more were correct. The 100% on TREC with descriptions means less than it looks: Kev was already 96.4–96.6% accurate there, so accepting everything stays above 95%.

Kev's calibration error, the gap between stated probability and actual accuracy, was 2–12% across the 15 runs. The worst was Kev-4B on TREC with names only (12%), the same run where its accuracy dipped, and there its auto-accept share fell to 58.6%. The temperature Kev ships with was fitted on data from these same datasets, so this is the best case for its calibration too.

MoJev's probabilities were consistently too low. On BANKING77 its average top probability was 0.23 while its accuracy was 0.44, and it never gave an answer at 90% or more on BANKING77 or TREC. Its calibration error was 6–22%, against the 0.79% it reports. The data may explain part of the gap: its number comes from 12,000 decisions in its own training-and-evaluation mixture, not from public benchmarks like these. Under-confidence is the safer failure, since it does not auto-accept wrong answers, but it also accepts almost nothing. On BANKING77 its probabilities still ranked right answers above wrong ones fairly often (AUROC 0.74, where 0.5 is chance), so a temperature fitted on your own labels might fix the scale. On TREC with names only they carried almost no signal (0.53). I did not test refitting.

To run MoJev I had to patch one line. On torch 2.14 its attention mask was float32 while the encoder ran in bf16, and PyTorch's attention kernel refused the mismatch. The patch casts the mask to bf16 and keeps blocked positions at bf16's smallest finite value instead of letting them round to minus infinity. It should not change which positions are masked, but it is a local change, and the diff is in the results folder.

SOTAAZ course

This course alone $49 · 23 lectures · first 3 freeWatch free →

What to do with this

• If your task looks like one Kev trained on, try Kev-0.8B first. At 14–25 ms it was within three points of the 9B on TREC and AG News (nine behind on BANKING77), and its probabilities were good enough to auto-accept more than half of a 77-option task. Kev's README reports lower accuracy on sources it did not train on, which I did not test.
• Train a small classifier on your labels before choosing either. On BANKING77 a MiniLM plus logistic regression running in about 5 ms on a GPU (7 ms on a CPU) beat every decision model here. Decision models earn their place when labels are scarce or the options change, not when you already have thousands of labelled examples.
• Check a reported calibration error on your own data. MoJev's 0.79% became 6–22% on public benchmarks.
• Test descriptions both ways. They helped Kev-4B on TREC (450 to 477 correct) and hurt MoJev on both tasks.

What this does not show

All three datasets are among Kev's trained sources, so this is Kev's best case, not a measure of how it handles new tasks. I did not run Kev-27B. Each task is one sample, auto-accept thresholds were searched on the same samples, and latency was measured one request at a time on one GPU.

Setup: one A100 80GB PCIe, driver 580.178.04. Kev repository 09ff745 with adapters jaredpalmer/kev-0.8b 9a45d25, kev-4b 139fdd9 and kev-9b 2629c06 on Qwen3.5 base revisions dc7cdfe, 1001bb4 and 68c46c4, served with python -m kev.serve in bf16 (torch 2.8.0, transformers 5.17.0, peft 0.21.0). MoJev repository a74d58c and checkpoint MoLeMo-Lab/mojev 0c8695b, served with mojev serve on torch 2.14.0 with the one-line mask patch described above. Both were called through their System One endpoints by the same client. Samples as in the earlier posts: BANKING77 154 (seed 20260918), TREC test set 500, AG News 200. The trained classifier is all-MiniLM-L6-v2 plus logistic regression, trained on 10,003 BANKING77, 5,452 TREC and 4,000 AG News training examples. The BANKING77 run was repeated on GPU to save per-message output for the paired tests; it scored 138 of 154, one fewer than the 139 in the first post. Overlap with Kev's development and calibration partitions was found by exact text match after lowercasing and removing punctuation. Calibration error is ECE over 10 bins of the top-option probability. For Kev this is the largest value in its probabilities, not its confidence field, which rescales that value by the number of options ((p − 1/K) / (1 − 1/K)) and is not a probability. The classifier's latency is 4.6 ms on GPU for BANKING77 and 5.7 and 10.8 ms on CPU for TREC and AG News. McNemar tests are exact and two-sided. Measured 2026-09-25.