Open-Source Jev Alternatives, Measured: laya, openjev and NanoJev on Three Public Datasets

Open-Source Jev Alternatives, Measured: laya, openjev and NanoJev on Three Public Datasets

TypeSafe's Jev answers a structured question with a probability instead of generated text. Since its launch, open-source projects with the same kind of interface have drawn a crowd; one of them has 18,000 GitHub stars. Each publishes its own numbers, on its own samples, prompts and hardware, and some on data it was trained on. What I could not find was all of them run on the same public data with human labels.

So I ran three of them on one A100, one request at a time:

• laya: a 421M ModernBERT encoder with a decision head, trained with reinforcement learning against proper scoring rules. Version 0.3.7.
• openjev: a Jev-compatible server over DiffusionGemma 26B-A4B, built on the vLLM structured-read PR I measured in my previous post. Version 0.4.0.
• NanoJev: Qwen3-0.6B with decision heads, trained on game decisions (maze, Snake, ViZDoom). Release unified-games-v1.

And on three datasets, chosen for how many options each question has:

With six options, a 421M encoder matches or beats a 26B model

TREC asks what kind of answer a question wants: an abbreviation, an entity, a description, a person, a location or a number. Every tool got the same instruction and the six label names.

laya, whose published training data does not list TREC, landed four points under a classifier trained on the full TREC training set (38 against 57 on the questions only one of them got right, p = 0.06). It is also the only one that did not need the descriptions. Its typed-decisions checkpoint scored 81.8% here, below the base checkpoint.

The DiffusionGemma servers failed in one specific way. Of the 65 questions whose answer is a person ("Who invented the telephone?"), openjev got 0 and the example server got 1; nearly all went to entity. They read the label human literally. With one line per label taken from the TREC taxonomy ("human: a person, a group of people or an organization"), the example server went from 335 to 438 correct and got 57 of those 65 right. The descriptions moved it by 20 points; they moved laya by 2 (p = 0.25).

NanoJev scored below simply answering description every time (27.6%). It was trained on games, and it shows.

With 77 options, the ranking flips

BANKING77 is the task from my earlier posts, same 154 messages, bare intent names as labels.

laya's own documentation predicts its default result. All options share a 192-token budget in the input, so 77 labels are cut to four tokens each and stop being distinguishable. The fix it recommends, a 512-token budget, took it from 57 to 71 correct (p = 0.04). Its other recommendation, cutting the list to 20 candidates with an embedding model first, reached 91. The embedding model nearly gets there alone: the nearest label name is right for 87 messages, and the shortlist's own top candidate, which embeds the instruction along with the message, for 82. laya on the 20 candidates added 4 to 9 messages over those, and neither difference is significant (p = 0.57 and 0.16). Most of that row is MiniLM's work. The typed-decisions checkpoint, for completeness, scored 39.0% at defaults and 46.1% at 512 tokens.

openjev is DiffusionGemma under the same vLLM commit as the example server in my previous post, with the same generic system prompt. The biggest difference is that openjev puts all 77 labels in one question, naming them A–Z, a–z and single-token letter pairs, where the example server stops at 26 and makes you split. Two smaller ones: the example server shows the question id to the model and openjev hides it, and past Z the labels are written differently. On an identical vLLM instance, openjev's one read beat the example server's two stages 33 to 12 on the messages only one of them got right (p = 0.002).

That narrows a question I left open in the previous post, without closing it. My transformers version scored 105 both in one read and in the same nine-group two-stage read, so splitting by itself is not the cause. openjev, with the example server's generic prompt, reached 103 in one read, level with my version (against my two-stage run: 120 of 154 answers the same, 9 against 11, p = 0.82). So my task-specific prompt is not what put my version 14 points ahead. The loss sits in the example server's two-stage implementation: the way it labels groups, the first answer it restates before the second read, or how it lays out the options. Which of those, I have not separated.

AG News is laya's home turf

On AG News laya scored 94.5% (typed-decisions checkpoint: 95.0%), above the logistic regression trained on 4,000 articles (88.5%, 15 against 3, p = 0.008). But laya's own docs list AG News in its training mix, so this measures retention, not a new task. The DiffusionGemma servers landed at 86.5% (example server) and 85.0% (openjev), and NanoJev at 31.0%, just over chance.

Where Jev itself stands

I have no Jev API access, so these are other people's measurements on other samples. AbdelStark's pilot reports Jev 1.13.0 at 0.910 on AG News and 0.870 on a 72-label BANKING77 variant, 100 examples each, with label descriptions supplied. nibzard's benchmark reports 76.3% with all 77 intents as options, which puts Jev mid-table there: gpt-oss-120b scored 81.3% and GLM-5.3 80.4% on the same task. With 77 options, then, Jev lands about 9 points above the best open alternative here, on a different sample. The 0.870 is a 72-label variant with label descriptions supplied, the kind of input that moved DiffusionGemma by 20 points on TREC.

Two things that changed answers without changing the model

Labels are part of the prompt. A one-line description per option was worth 20 points to the example server and 14 to openjev on TREC, and almost nothing to laya. If you evaluate a DiffusionGemma-based tool, write the descriptions before you judge it.

So is the server. openjev and the example server run the same weights under the same vLLM commit. On TREC with descriptions, the example server was ahead 44 to 6 (p < 0.001); on BANKING77, openjev was ahead because it can ask all 77 at once. Even restarting vLLM with one extra flag (--async-scheduling) changed 28 of 154 example-server answers, although accuracy stayed within two messages and back-to-back runs with the same flags reproduced every answer.

Which one to use

• A handful of options, English text, low latency: laya. 23 ms on a GPU for a 421M model, no descriptions needed, within a few points of a trained classifier on TREC. Check that your task is not one it already saw in training before you trust a headline number.
• Dozens of options: none of these replaces training a small classifier. With a few thousand labelled examples, MiniLM plus logistic regression beat every tool on BANKING77 by more than 20 points, and on TREC it was level with laya once laya had descriptions (32 against 41, p = 0.35). Without labels, openjev at 67% is the best open option I measured, and it needs a GPU that can hold a 26B model.
• NanoJev: not for text classification. It is a game-decision model, and its own results are on games.

What this does not show

One GPU, one request at a time, one instruction per dataset, and no tuning beyond what each project's docs recommend. openjev normally serves an NVFP4 checkpoint; an A100 cannot run FP4 natively, so it served the bf16 weights here. Its Docker image also patches vLLM in two places, raising the label-id cap from 128 to 512 and fixing image attention; I ran it against the unpatched commit, and neither patch applies to 77 labels of text. On TREC and AG News the example server received the question id intent, which it shows to the model; openjev hides ids. I did not measure calibration, only which option each tool ranked first. And the Jev figures above are not from these samples.

Setup: one A100 80GB PCIe, driver 580.178.04. laya 0.3.7 and NanoJev unified-games-v1 on torch 2.14.0+cu130, transformers 5.17. openjev 0.4.0 and the vLLM example server on vLLM 1b3b88e with --async-scheduling, canvas 64, google/diffusiongemma-26B-A4B-it bf16, one noise draw (samples: 1). The example server's BANKING77 row (54.5%) is from my previous post, without --async-scheduling; the paired comparison with openjev uses a rerun on the same instance as openjev (82 correct). Instructions: "Which intent does this banking customer's message express?", "What kind of answer does this question ask for?", "Which topic is this news article about?". NanoJev requires a description per option; where none was given it received the label name. BANKING77 (PolyAI, CC-BY-4.0), TREC (Li & Roth), AG News (Zhang et al.) test sets; samples drawn with seed 20260918. Exact two-sided McNemar tests on paired predictions. Measured 2026-09-23.