How Many Labels Is a Decision Model Worth? Kev on Three Tasks It Never Saw

How Many Labels Is a Decision Model Worth? Kev on Three Tasks It Never Saw

Last time Kev looked good: calibrated probabilities, and more than half of a 77-option task accepted automatically at 95% accuracy. But all three datasets were ones Kev had trained on, and the post said so. The question that decides whether a decision model is worth using is what happens on your own task, which it has never seen.

So I picked three tasks that appear nowhere in Kev's training data and scored them against human labels.

• CLINC150: 150 intents for a virtual assistant, plus questions that fit none of them. Kev's authors keep it as an evaluation-only set; it was never trained on.
• MASSIVE (English): 60 intents for a voice assistant. Not in Kev's repository at all.
• Financial news tweets: 20 topics such as Earnings, Fed | Central Banks and M&A | Investments. Not in Kev's repository either. I avoided CFPB complaints, which Kev did train on.

This time the comparison is not a classifier trained on thousands of examples, but the question the last post left open: how many labelled examples per class does a small classifier need to match a decision model? I trained MiniLM embeddings with logistic regression on 1, 2, 5, 10 and 20 examples per class and on the full training set, repeating each small draw five times.

Accuracy: Kev-9B was worth a handful of labels

CLINC's 100 out-of-scope questions have no correct option, so every model gets them wrong and 75% is the ceiling. On the 300 in-scope questions Kev-9B was 82.7% accurate. The small-label rows are means over five draws, with a standard deviation of one to five points between draws.

Where exactly Kev-9B sits depends on the task. Against the first draw of each classifier, counting the messages only one of the two got right (Kev first, classifier second):

• CLINC150: statistically tied with 2 labels per class (45 against 32, p = 0.17) and 5 (28 against 39, p = 0.22). From 10 labels the classifier was ahead (14 against 42, p < 0.001).
• MASSIVE: ahead of 2 labels (41 against 15, p < 0.001), tied with 5, 10 and 20 (p ≥ 0.52). Only the classifier trained on everything was ahead (13 against 28, p = 0.03).
• Financial tweets: ahead of 5 labels (58 against 24, p < 0.001), tied with 10, 20 and even all 16,990 training tweets (27 against 38, p = 0.22).

So on a task it has never seen, Kev-9B was worth somewhere between 2 and 20 labelled examples per class. Kev's README reports lower accuracy on new sources than on trained ones, and this is consistent with that: on familiar data last time, it took a classifier trained on thousands of examples to beat Kev on BANKING77, and that classifier still lost to Kev on TREC.

Size mattered more here than on familiar data. Kev-4B and Kev-9B were statistically tied on all three tasks (p ≥ 0.32), but Kev-0.8B fell apart on the financial tweets: it answered "Financials" to 152 of 200, the most generic topic name, and scored 22.5%. On data it knew, the 0.8B model had been within three points of the 9B on two tasks.

Probabilities: too low on tasks it had not seen

As before, thresholds were searched on the same samples they are scored on, so these shares describe the samples.

Kev-9B's calibration error, the gap between stated probability and actual accuracy, was 11–17% on these tasks, against 2–9% for the same model on the tasks it trained on. The direction matters: on all three tasks its average top probability was 10–17 points below its accuracy. Kev was not over-confident on unfamiliar data; it was too unsure. That does not change the auto-accept shares above, which come from searching the threshold on labelled data and depend only on how well the probabilities rank right answers above wrong ones. It matters if you take the stated probability at face value: a rule such as "accept when Kev says 95%" would accept far fewer answers than it safely could. The 4B model was further off in the same direction (17–28 points low), and its probabilities ranked right answers above wrong ones a little less well (AUROC 0.74–0.90 against 0.76–0.91). It accepted less than the 9B on two tasks even though their accuracy was tied. The one exception was Kev-0.8B on the financial tweets, 31 points over-confident: the run where it answered "Financials" to almost everything.

The classifier trained on all labels was the best calibrated of everything here (4–7%). The 5-label classifier was far too unsure, 31–37 points under its accuracy, so probabilities from a model trained on a handful of examples should not be taken at face value either.

SOTAAZ course

This course alone $49 · 23 lectures · first 3 freeWatch free →

Questions no option answers

CLINC's 100 out-of-scope questions are the realistic case for routing: someone asks something your options do not cover, and a decision model has no "none of these" to pick. Kev-9B still had to pick an intent, but it did so with low confidence. Its average top probability was 0.20 on out-of-scope questions against 0.62 on in-scope ones, and the probability alone separated the two groups well (AUROC 0.88, where 0.5 is chance). At the threshold that kept accepted answers 95% correct, 5 of the 100 out-of-scope questions got through. Kev-4B let 3 through and Kev-0.8B 1, but they also accepted fewer answers overall.

That makes the probability usable as the "hand this to a person" signal, even without a "none of these" option. It is not a replacement for one: five in a hundred still slipped past.

MoJev and laya

Both were well behind on all three tasks. MoJev was again under-confident: on CLINC150's in-scope questions its average top probability was 0.07 while it was 39% accurate. laya was the only over-confident model, with probabilities 34–65 points above its accuracy. laya could not run CLINC150 at its default settings at all (150 options exceeded its option budget), so its rows use the 512-token budget its documentation recommends for many options. laya's own benchmark reports 78.3% on English MASSIVE with 20 options per question; with all 60 intents offered here it scored 27–34% depending on the budget. The setups differ in more than the option count, but three times as many options is the obvious suspect.

What to do with this

• Count your labels before choosing. With about 20 labelled examples per class, a small trained classifier matched or beat Kev-9B on accuracy and let more answers through at 95% accuracy on all three tasks; trained on everything, it was also far better calibrated. With one or two per class, Kev-9B was ahead or tied.
• Use Kev-4B or 9B on new tasks, not 0.8B. The smallest model collapsed onto one label on a task it had not seen.
• Set thresholds on your own labels, not on the stated probability. The shipped temperature held on familiar tasks and ran 10–17 points low on new ones, so a fixed "95% means 95%" rule would send too much to people. Picking the threshold from a labelled sample, as in the table, avoids that; refitting the temperature would make the stated numbers honest again, which I did not test.
• Use a low probability as the out-of-scope signal, and still check a sample. It caught 95 of 100 questions no option fitted, at a 95%-accuracy threshold. The price is that the same threshold auto-accepted only 171 of CLINC's 400 messages; the rest went to a person.

What this does not show

Three tasks, all short English texts and all intent or topic labels. One sample per task, and the paired tests compare Kev with the first of the five classifier draws. The classifier uses one embedding model and no tuning beyond a fixed regularisation. Latency was not measured, because the GPUs were shared with another job during these runs. Kev-27B was not run.

Setup: one A100 80GB PCIe, driver 580.178.04. Kev repository 09ff745, adapters jaredpalmer/kev-0.8b 9a45d25, kev-4b 139fdd9, kev-9b 2629c06, served with python -m kev.serve in bf16. MoJev a74d58c with the one-line mask patch from the previous post; laya 0.3.7 with head_max_len 512. Samples: CLINC150 (clinc/clinc_oos, "plus", CC-BY-3.0) test split, 2 per intent plus 100 out-of-scope questions; MASSIVE English (mteb/amazon_massive_intent, CC-BY-4.0) test split, 3 per intent: 175 messages over 59 intents (general_greet has 1, cooking_query none in the test split), with all 60 intents offered as options; financial news tweets (zeroshot/twitter-financial-news-topic, MIT) validation split, 10 per topic; seed 20260926. Label names are the datasets' own. None of the three datasets is among Kev's training sources. CLINC150 is in Kev's evaluation-only suite, whose development partition comes from CLINC's validation split; this sample comes from the test split. An exact text match against the Kev data files present locally found only three short phrases, such as "what is the weather", in unrelated suites. Classifier: all-MiniLM-L6-v2 plus logistic regression (C = 10) on each dataset's train split, k examples per class drawn five times. Calibration error is ECE over 10 bins of the top-option probability; McNemar tests are exact and two-sided. Measured 2026-09-26.