Models & Algorithms•SOTAAZ Lab••KR

Where Decision Models Read Their Answer: laya, DiffusionGemma, Kev, CLM and a Plain Classifier in Code

Five ways to turn a customer message and 77 intent labels into a probability per label, followed through the code of each. On the same 154 BANKING77 messages they scored from 3.2% to 90.3%. laya cuts every label to three tokens, and giving it more room lifts it from 37.0% to 46.1%; the others keep labels whole, place the options differently, and read the score elsewhere.

Where Decision Models Read Their Answer: laya, DiffusionGemma, Kev, CLM and a Plain Classifier in Code

A bank customer types "What is the top-ups limit?" and a decision model has to pick one of 77 intents. The right one is top_up_limits. laya, a 421M decision model, picked top_up_by_cash_or_cheque. When we fed laya's input builder the same 77 labels and counted tokens, both labels came out as the same three tokens, top_up, as did four other top-up intents. Inside laya's input, six different answers were spelled identically.

That one message is a small version of a bigger spread. On the same 154 BANKING77 messages, with label names only, the five systems this post opens scored from 3.2% to 90.3%. Jev is listed for reference:

SystemAccuracy on the same 154 messages
laya, default settings37.0%
DiffusionGemma through openjev, one read66.9%
Kev-9B (BANKING77 is in its training data)82.5%
CLM, label names only3.2%
MiniLM embeddings + logistic regression, trained on 10,003 labelled messages90.3%
*Reference: Jev 1.13 (TypeSafe API, internals not public)**76.0%*

These numbers come from our earlier posts (laya, openjev and the classifier, Kev and Jev, CLM). The Kev row needs its caveat in the same breath: Kev was trained on BANKING77, so this task is not new to it the way it is to laya or openjev.

This post opens the code of each system and follows that one message until it becomes a probability. The architectures are public for all of them except Jev, so the differences can be read straight from the source.

What a decision model does

A decision model does not write an answer. You send it a state (here, the customer's message), a question, and a fixed list of options. It sends back one probability per option. Our ten-minute explainer covers why that is useful: you act on the confident answers and pass the rest to a slower model or a person.

Every system below has to solve the same two problems. It has to show the model the options somehow, and it has to read one score per option from somewhere inside the network. The five systems answer those two questions differently, and with 77 options the first one becomes especially important.

One customer message branches into five lanes: laya puts all 77 options in one input and scores each MASK token; DiffusionGemma lists the names in the prompt and reads one blank answer slot; Kev, trained on BANKING77, scores each option's </opt> against <decide>; CLM embeds the message and each option apart; the classifier never sees the options. Each lane ends in its accuracy on the same 154 messages: 37.0%, 66.9%, 82.5%, 3.2%, 90.3%.

laya: every option inside one input, one score per [MASK]

laya is a ModernBERT-large encoder with a small head on top. It builds one sequence per question in laya/common.py:

[CLS] choice question: <instructions> [SEP] [MASK] option0 [MASK] option1 ... [SEP] <state> [SEP]

Each option starts with its own [MASK] token. After the encoder runs, the head picks up the hidden vector at every [MASK] position, turns each into a single number, and takes a softmax over the options:

python
idx = marker_pos.clamp(min=0)[:, :, None].expand(-1, -1, h.size(-1))
m = torch.gather(h, 1, idx)
logits = self.scorer(m).squeeze(-1).float()
laya's architecture: one input row holds the CLS token, the instructions, then a MASK token before each option, then the message. The encoder lets every token attend to every other. The hidden vector at each MASK goes to a scorer MLP that gives one number, and a softmax runs over the 77 numbers. Under the row: the 77 options share a 192-token budget, so each is cut to a MASK token and three label tokens.

Because this is an encoder, every [MASK] can attend to the message and to every other option, which is the strength of the design. The cost is that all the options have to fit into the instruction-and-options part of the input, and that part has a fixed size, head_max_len, which is 192 tokens in the English checkpoint. When the options don't fit, this block cuts every option to the same length:

python
opt_budget = head_max_len - sum(len(o) for o in opt_ids)
if opt_budget < 16:
    per = max(4, (head_max_len - 16) // max(1, len(opt_ids)))
    opt_ids = [o[:per] for o in opt_ids]
    opt_budget = head_max_len - sum(len(o) for o in opt_ids)
head_ids = head_ids[: max(8, opt_budget)]

With 77 options, per hits its floor of 4: one [MASK] and three label tokens. We ran this function with laya's own tokenizer on the 77 BANKING77 labels and the message above (tokenizer only, no model):

  • The labels are 3 to 17 tokens long, 5 at the median, 476 in total. The budget is 192.
  • 64 of the 77 labels get cut. After cutting, only 67 distinct labels remain. Six labels become top_up, three become card_payment.
  • The instruction keeps 8 tokens and now reads "choice question: Which intent does this banking".
  • The options take 308 positions, 77 times four, well past the 192 budget, because the floor of 4 overrides it. The message itself is 8 tokens and still fits.
All 77 BANKING77 labels in a grid, first as full names, then as laya keeps them at a 192-token budget, three tokens each. Labels that became identical are filled and outlined: six top_up, three card_payment, and pairs of lost_or, balance_not and cash_with.

Raising the budget to 512 (with max_len 1024) leaves the instruction whole and cuts 37 labels instead of 64, with 72 distinct labels left. In our run that setting answered top_up_limits correctly for the message above, and accuracy rose from 37.0% to 46.1%. Shortlisting to 20 labels with an embedding model first, the other fix laya's README suggests, reached 59.1%, though MiniLM alone already picked the right label for 87 of the 154 messages (laya on the shortlist got 91). laya's own benchmark notes say it plainly: "Keep choice questions under ~20 options" (BENCHMARKS.md).

DiffusionGemma: leave one blank on the answer sheet and read it

openjev wraps a different idea, taken from the vLLM pull request for DiffusionGemma-26B-A4B that we tested in our DiffusionGemma post. A diffusion language model fills a whole block of text at once instead of one token at a time. openjev writes the answer sheet onto that block in advance and leaves exactly one position blank (openjev/engine.py):

python
canvas = list(template) + [TURN_CLOSE]
canvas += [PAD] * (self.canvas_width(template) - len(canvas))
for s in slots:
    canvas[s["pos"]] = rng.randrange(VOCAB)

The template is q1: X. The options themselves go into the system prompt as a legend, A: card_arrival, B: card_linking and so on. One denoising step later, openjev reads the logits at the blank, keeps only the token ids of the legend symbols, and takes a softmax over those.

DiffusionGemma through openjev: the prompt holds a legend of 77 symbols with their intent names and the message; after it comes the answer canvas, with a thought block, q1:, one cell holding a random token, end of turn and padding. The diffusion model fills the whole canvas in one step, the logits at that one cell are kept only for the 77 legend symbol ids, and a softmax runs over them.

The catch is that each option has to be one token at that one position, so the label is the symbol, never the intent name. openjev checks every label and refuses the request otherwise:

python
if len(e) != len(base) or len(diffs) != 1 or (pos is not None and diffs[0] != pos):
    raise SchemaError(f"question {q['key']!r}: labels do not share one template slot")

For 77 options this avoids laya's problem. The intent names are never cut, because they sit in the prompt, which has room for all of them, and the answer is still one position. The model's job becomes reading a 77-line legend and writing one letter, and openjev got 66.9%. The legend is still a prompt, though, and wording matters. In that post, vLLM's own example server, which uses the same one-slot read but stops at 26 options, fell from 54.5% to 31.8% when we added one sentence describing the input.

openjev's answer canvas: an empty thought block, q1:, one cell holding a random token, then end-of-turn and padding. The random cell flickers, then is read, and bars show the top five probabilities for this message: top_up_limits 0.9998, every other label below 0.001.

Kev: options as spans in a decoder, scored by pointing

Kev is a Qwen3.5 base model (0.8B, 4B or 9B) with a LoRA adapter and a small head; our first Kev post measured it on the tasks it trained on. It packs the request into one sequence, reusing five rarely used Qwen special tokens as markers (kev/model.py):

<state> message <q> instructions <opt> card_arrival </opt> <opt> card_linking </opt> ... <decide>

Kev does not read the probability of a label token. Instead, it takes the hidden vector at the end of each option, the </opt> token, and compares it with the hidden vector at <decide>, which comes after every option and so has seen all of them:

python
def forward(self, h_decide, h_opts):  # [d], [K,d] -> logits [K]
    z = (self.k(h_opts) @ self.q(h_decide)) * self.scale
    return z if self.training or self.temperature == 1.0 else z / self.temperature
Kev's architecture: one row with state, message, q, instructions, then each option wrapped in opt and /opt tokens, then decide. The decoder lets each token see only what comes before it, so decide has seen every option. The hidden vectors at each /opt token and at decide go into the pointer head, which scores each option by a dot product times a scale divided by the temperature, followed by a softmax over 77.

That is one score per option, like laya, but the options are not squeezed into a fixed share of the input. Every label goes in whole, and each question's row, message included, can run to 8,192 tokens when served. For BANKING77 that means all 77 names arrive intact. The temperature that softens the scores is stored in the checkpoint; for the 9B revision we tested (2629c06) it is 2.297.

Kev-9B scored 82.5% on our 154 messages, ten more right than Jev, though that gap is not statistically clear. Part of it is training data: BANKING77 is one of the tasks Kev was trained on. On three task sets Kev had not seen, Jev came out ahead in two (Kev vs Jev), so the layout explains why 77 options are not a problem for Kev, not why it beats the others.

Kev's sequence: the message, the instructions, then option spans for top_up_failed, top_up_limits, top_up_reverted and verify_top_up, each ending in a highlighted </opt> cell, then <decide>. Arrows from <decide> reach each </opt> one by one; the thick one lands on top_up_limits at 0.94.

CLM: two embeddings and a cosine

CLM takes the opposite route to laya. The message and each option never share an input. A Qwen3-8B encoder embeds the message with the question's instructions, embeds every option separately, a small trained head projects both, and the score is the scaled cosine between them (src/clm/engine.py):

python
cos = za[k:k + len(texts)] @ zq[i]
k += len(texts)
answers[qid] = answer_from_logits(questions[qid], keys, (scale * cos / temperature).tolist())
CLM's architecture: on the left, the instructions and message go through Qwen3-8B and the head into one vector; on the right, each of the 77 options goes through the same path on its own, and those vectors can be cached. The score is the scaled cosine between the two vectors divided by the temperature, followed by a softmax over 77.

Option embeddings can be cached, so adding options is cheap, and nothing is ever cut. But the option is now judged by its text alone, and top_up_limits as a bare label is a poor sentence to compare a question against. With label names only, CLM scored 3.2% on our 154 messages, where random choice gives 1.3% (decision models and confidence). One-line descriptions helped on the tasks where we had them, which fits how it was trained, on answer sentences rather than one-word labels.

The plain classifier: the options are never in the input

The classifier we used as a baseline does not show the model the options at all. These lines are from our benchmark script, with the timing calls taken out:

python
m = SentenceTransformer(a.model, device=a.device)  # all-MiniLM-L6-v2
Xtr = m.encode([r["text"] for r in train], batch_size=256, normalize_embeddings=True, show_progress_bar=False)
clf = LogisticRegression(max_iter=2000, C=10.0)
clf.fit(Xtr, [r["category"] for r in train])
The classifier: only the message goes in, MiniLM turns it into a 384-dimensional vector, and 77 weight rows, one per intent and learned from 10,003 labelled messages, each give a score. A softmax runs over the 77 scores.

Each of the 77 intents is a row of weights learned from labelled messages, 10,003 of them. Nothing about the options competes for input space, because the input is only the message. That is why 77 options cost it nothing, and it scored 90.3% on CPU. The price is the training set. Without labelled examples this option does not exist, which is the case decision models were built for.

The five systems side by side

A table drawn as a figure. Rows: laya, DiffusionGemma, Kev, CLM, classifier. Columns: where the score is read, are options inside the input, are long labels cut, needs labelled examples, accuracy on the same 154 messages.
Where the score is readOptions in the inputLong labels cut at 77 optionsNeeds labelled examples154 messages
laya[MASK] before each optionyes, in a shared 192-token budgetyes, to 3 tokensno37.0%
DiffusionGemma (openjev)one blank answer positionnames in the prompt, answer is one symbolnono66.9%
Kev-9B</opt> of each option, against <decide>yes, each label wholenono (but trained on BANKING77)82.5%
CLMcosine between two embeddingseach option embedded on its ownnono3.2%
MiniLM + logistic regressionlearned weight row per intentnon/ayes, 10,00390.3%

All five used the same 154 messages and label names without descriptions. They differ in model size, training data and hardware, so the table shows how each system handles the options, not a ranking of architectures.

Jev is the one system we cannot open. TypeSafe has said it uses a new model architecture, a parallel sampler and a training method called RLCD, and that it produces all outputs in one request. Its internals are not public, so we have left it off the comparison table.

Before you send 77 options

If you plan to use a decision model on a long option list, find out where your options go. If they share a fixed budget inside the input, as with laya, count the tokens yourself before trusting the result. Here is the check we used, with laya installed:

python
from transformers import AutoTokenizer
from laya.common import build_sequence

tok = AutoTokenizer.from_pretrained("convaiinnovations/laya", subfolder="tokenizer")
question = {"t": "choice", "ins": instructions, "crit": {l: None for l in labels}}
# max_len and head_max_len: use your checkpoint's config values
seq, markers = build_sequence(tok, message, question, max_len=512, head_max_len=192)
ends = markers[1:] + [seq.index(tok.sep_token_id, markers[-1])]
kept = [tok.decode(seq[m + 1:e]).strip() for m, e in zip(markers, ends)]
print(len(set(kept)), "distinct labels out of", len(kept))

If you already have a loaded agent, agent.tok, agent.cfg["max_len"] and agent.cfg["head_max_len"] give the same three values.

If that number is smaller than your label count, some of your answers are identical from the model's point of view. Shortlist first, shorten the labels so they differ in their first tokens, or pick a system that keeps labels whole.

Code read at these commits: laya 010bace, openjev 3e296f0, Kev 09ff745, CLM cca045f. Accuracy figures are from the result files of the linked posts; token counts use laya's own tokenizer on the 77 BANKING77 labels.