Build a Jev-Style Decision Model from Scratch

What you'll learn

• ✓Explain how a decision model returns a probability distribution over supplied options without generating text
• ✓Implement delimiters, an option-isolating attention mask, shared position ids and a pointer head in PyTorch
• ✓Train yes/no, choice and ordered-score questions with LoRA and one cross-entropy loss
• ✓Measure calibration (reliability diagrams, ECE, Brier, confident errors) and fix overconfidence with temperature scaling
• ✓Prove option-order invariance with a permutation test
• ✓Serve a model behind a SystemOne-compatible API and route low-confidence cases to people

About this course

Jev, released by TypeSafe in September 2026, answers questions without writing a word: you send a message and typed questions, and it returns a probability for every option you gave it. Open models such as Kev and MoJev reproduce the idea. This course builds one from scratch, small enough to read end to end, and then measures whether its probabilities deserve trust.

We start by running Kev locally and making the honest case for and against decision models, using measurements from real benchmarks: when a small trained classifier is the better tool, and when a decision model earns its place. Then we build the design in PyTorch on a 0.6-billion-parameter Qwen base: borrowed special tokens as delimiters, an attention mask that keeps options from seeing each other, shared position ids that make option order irrelevant, and a pointer head that scores each option against a decide token. We train it with LoRA on public data (BANKING77, CC-BY-4.0) plus generated data, for three question types: yes/no, choice and ordered score.

Then the part most tutorials skip. We measure calibration with reliability diagrams, ECE, Brier score and confident-error rates; show that training longer made confident errors on unseen intents rise from 2.7% to 9.7% while in-distribution numbers looked better; largely correct it with a single temperature (back to 3.0%, though the one-epoch model stays better calibrated); and prove option-order invariance in float32. Finally we serve the model behind the same API shape as Jev and Kev, call it with the official TypeSafe SDK, and use its probabilities to route work between automatic decisions and people, with every case logged.

This course is not affiliated with or endorsed by TypeSafe AI; "Jev" names the model the course explains. Jev's internals are not public: the course builds the inferred design that open models follow, and says so. Every number in the lectures comes from notebooks you can run.

The on-screen presenter in the welcome and closing lectures is an AI-animated avatar.

Curriculum8 sections · 23 lectures · 1h 55m

Read alongside the course

• Jev in 10 Minutes: What a Fast Decision Model Is, and When It Is Worth Using
• Can You Trust a Decision Model's Confidence? Repeated Answers and Probabilities, Measured
• Open-Source Jev Alternatives, Measured: laya, openjev and NanoJev on Three Public Datasets