Jev's Speed Claims: Benchmarking Label-Only Alternatives on BANKING77

Jev's Speed Claims: Benchmarking Label-Only Alternatives on BANKING77

TypeSafe AI released Jev on September 15. It does not write text: given a structured question and a fixed set of options, it returns a decision with a probability. The launch post puts Jev's end-to-end response time at 70–500 ms, and cites an external LLM benchmark for 3 to 329 seconds on frontier models.

The probabilities belong to a different comparison. In its workflow evaluation, TypeSafe runs the other models through a wrapper that constrains them to structured decisions compatible with Jev's API, and notes that this "tends to be slower and more expensive than giving decisions without probabilities."

This post covers classification where only the label is needed. I measured four ways to get that label on a public dataset with human labels. Two of them are OpenAI models: I tested GPT-5.6 Terra and GPT-6 Astra. TypeSafe's workflow evaluation lists OpenAI's "Terra" as a comparison model and uses Astra as one of the models generating reference answers. The evaluation table does not specify Terra's exact version. Jev is on a waitlist for me, so it is not in the table yet.

The clearest result is the first row. On this fixed-label task, a logistic regression on top of pretrained sentence embeddings was both the fastest and the most accurate method, ahead of the two frontier models by six to seven points and 200 to 300 times quicker at the median. It needed labelled data, which the other three did not.

The task and the conditions

The BANKING77 test set (PolyAI, CC-BY-4.0) is 3,080 customer messages, each labelled by people with one of 77 banking intents, exactly 40 per intent. Its training set, which only the classifier uses, has 10,003. Several intents are near-synonyms, such as card_arrival and card_delivery_estimate.

The three language models received the same system prompt listing all 77 labels and answered zero-shot, with output constrained to one valid label: an enum in OpenAI's structured output, a grammar for the local model. None of their 462 responses was invalid or empty.

The API calls cost money, so every method was measured on the same stratified sample of 154 messages, two per intent. The local model and the classifier were also run on all 3,080 as a check on the sample: 62.3% on the sample against 63.5% on the full set for the local model, and 90.3% against 93.0% for the classifier.

The methods did not run under the same load. The API models ran with 8 concurrent requests. The local model ran one request at a time, and the classifier embedded one message at a time on CPU. Latency here is per message under those settings.

Two frontier models, label only

Both OpenAI models used the Responses API with reasoning effort low, the lowest GPT-6 Astra accepts:

• GPT-6 Astra: 128 of 154 correct, 2,163 ms median, 3,950 ms at p95, $9.81 per thousand messages.
• GPT-5.6 Terra: 129 of 154 correct, 1,351 ms median, 2,666 ms at p95, $2.02 per thousand.

Compared message by message, both were right on 125 and both were wrong on 22. Astra alone was right on 3 and Terra alone on 4. The observed accuracy is similar, and a one-message difference on 154 says nothing about which model is better at this task. Terra's price per message was about a fifth of Astra's.

Both spent six to seven reasoning tokens per message, so nearly all of the cost was input: 864 to 938 tokens per request, 873 on average, because the 77 labels appear once in the prompt and again in the output schema. Neither run registered any cached input tokens. Prices are from OpenAI's model pages: GPT-6 Astra $10 input and $50 output per million tokens, GPT-5.6 Terra $2 and $12 (checked 2026-09-18).

Median latency was 1.4 to 2.2 seconds with only the label requested. That includes the network round trip from the benchmark machine to OpenAI. Network latency may be lower for a service deployed closer to the API.

A local model with constrained decoding

Qwen3-8B at Q4_K_M ran on one A100 through llama.cpp, with a grammar allowing only the 77 labels. The label list was cached across requests, so each request processed only the message, about 20 tokens. Median latency was 45.7 ms and p95 73.5 ms, one request at a time.

Accuracy was 62.3%, about twenty points below the frontier models on the same messages. There is no per-call API fee, but it is not free: it needs a GPU and someone to run it.

A classifier on pretrained embeddings

The classifier uses all-MiniLM-L6-v2, a pretrained 22M-parameter sentence embedding model, unchanged. Only a logistic regression on top of its embeddings was trained, on BANKING77's 10,003 training messages. Extracting those embeddings and fitting the regression took about 12 seconds on an AMD EPYC 7742 using 64 threads.

It then classified at 6.8 ms median and 11.4 ms at p95 on the same CPU, with no GPU, and reached 90.3% on the sample and 93.0% on the full test set.

This is not a like-for-like contest with the zero-shot methods: it learned from ten thousand labelled examples and they saw none. In this sample, the trained classifier was more accurate than all three zero-shot methods. For a team that has labelled data and a fixed label set, it is a cheap thing to try first.

What Jev has published, and what this does not measure

TypeSafe has published a workflow evaluation. On its aggregate view across four workflows (security incidents, agent trace observability, invoice processing, customer service), Jev is at 67.8% accuracy, 0.4 seconds and $0.0004 per case, and Terra (workflow) at 67.9%, 10.1 seconds and $0.0304 (checked 2026-09-19). The two aggregate scores against that evaluation's reference answers are similar; that does not mean equal performance against human labels, or statistical equivalence. The reference answers there are not human labels. TypeSafe describes them as the average of GPT-6 Astra's and Claude Fable 5.1's answers at high thinking effort.

Those figures and the ones in this post are not comparable. The tasks differ, and one set is scored against human labels while the other is scored against two frontier models. I have not measured Jev's accuracy or latency on BANKING77. When I get access, it runs on the same 154 messages with the same prompt, and that row gets filled in.

Jev is described as returning calibrated probabilities. This comparison did not evaluate probability calibration for any model, and a logistic regression also outputs probabilities. Whether any of these are well calibrated is a separate measurement.

What to use for label-only classification

• You have labelled examples for a fixed set of labels. Try a classifier on pretrained embeddings first. Here it was the most accurate and fastest method and ran on CPU.
• No labelled data, and accuracy matters more than latency. A frontier model asked only for the label. On this task Terra and Astra were one message apart, at a fifth of the price for Terra.
• No labelled data, latency matters, and you can run a GPU. A local model with constrained decoding, after measuring its accuracy on your own labels. Here it was about twenty points short.

What this does not show

One dataset, in English, with 77 fine-grained intents, some of them near-synonyms; a task with different labels could rank these methods differently. The API results are a single run each on 154 messages. The methods ran under different concurrency, as described above. The local model was not tuned for batched serving. And Jev is not measured.

To get the Jev result when it is measured, subscribe below.

Setup: BANKING77 test set (PolyAI, CC-BY-4.0), stratified sample of 154 messages (two per intent, seed 20260918). Language models share one system prompt and answer zero-shot. OpenAI: Responses API, structured output with the labels as an enum, reasoning effort low, 8 concurrent requests. Local: Qwen3-8B Q4_K_M, llama.cpp 69320fe, A100 80GB, grammar-constrained, cache_prompt on, 1 request at a time. Classifier: pretrained all-MiniLM-L6-v2 embeddings with scikit-learn logistic regression (C=10), AMD EPYC 7742, 64 threads. Measured on 2026-09-18.