Learn AI by Building
From your first dataset to production agents — deep-dive series, hands-on notebooks, and experiments you can rerun yourself.
Paper of the Week
All Issues →Paper of the Week #7 — Told to Skip the Reasoning, It Kept Working in English and Mostly Stopped in Korean
This week's roundup. We gave Qwen3-8B the Thinking Inertia paper's "no reasoning, just the answer" instruction in English and in Korean: on 30 open-ended math questions it answered directly 0% of the time in English and 63.3% in Korean, where accuracy fell to 36.7%. Also: what a passing test guarantees, how much of an uncertainty score is answer length, agent cost tracking turns, who checks Lean's checkers, the conditions behind SSD-streaming speeds, decision models when their own server is at stake, three ways to write context down, and grading choices that reorder rankings.
Premium Series
Our Products
Courses and code built from what we measure here
Video Courses
13 hands-on courses — quantization, diffusion, RAG, on-device AI, decision models. $199 lifetime bundle, first 3 lectures of every course free
LLM Quantization and Compression Hands-On
GPTQ, AWQ, GGUF, QLoRA — fit LLMs into the memory you have. The course behind our KV-cache measurements
Free Benchmark Code
The scripts and raw results behind the measurement posts, starting with the KV cache harness. Free with an account
Premium Series
140 deep-dive posts across 21 series, bilingual KO/EN, with production-ready code and notebooks
Latest Posts
View All →
Ten Minutes to Find Out Whether the Tests a Local Model Wrote for Your Function Protect a Bug
One script, two versions of your function, a local model: it asks the model for pytest tests from your current version only, runs every test against both versions, and tells you which tests would pass today and fail after the fix. On a date function with an off-by-one, Qwen3.5-9B wrote 10 tests in 6 seconds; 3 asserted the bug and none caught it. The same scorer and prompt as our 164-problem measurement.
Paper of the Week #7 — Told to Skip the Reasoning, It Kept Working in English and Mostly Stopped in Korean
This week's roundup. We gave Qwen3-8B the Thinking Inertia paper's "no reasoning, just the answer" instruction in English and in Korean: on 30 open-ended math questions it answered directly 0% of the time in English and 63.3% in Korean, where accuracy fell to 36.7%. Also: what a passing test guarantees, how much of an uncertainty score is answer length, agent cost tracking turns, who checks Lean's checkers, the conditions behind SSD-streaming speeds, decision models when their own server is at stake, three ways to write context down, and grading choices that reorder rankings.
A Decision Model Caught Every Look-Alike You Can't See by Eye. It Missed scotlabank.com.
Jev caught all 60 addresses with a look-alike letter from another alphabet, the ones you cannot tell apart by eye, and missed ten plain typos like scotlabank.com and d0cusign.com. Kev-9B did the opposite, calling 39 of those 60 real. The checker in this post uses no model, catches all 300 of our test addresses (a number we explain away below), and nothing you paste leaves your browser.
Show a Local Model Buggy Code and Its Tests Protect the Bug, Even When It Notices
We asked Qwen3.5-9B, running locally, to write pytest tests for all 164 HumanEvalPack Python functions, once from the buggy version and once from the fixed one. From the buggy code, 16.4% of its tests passed on the bug and failed on the fix, against 2.6% from the fixed code, and the share of tests that catch the bug fell from 41.9% to 19.5%. With thinking on, on the 85 problems both modes finished, misguided tests rose from 22.4% to 37.6%. A paper's fix, giving the model a spec it wrote from the buggy code instead of the code itself, did not show an effect here, and about half of those specs described the bug as intended behavior.
One Letter Holds a Language Model’s World Map Together
We asked Qwen3.5 models whether each point on a latitude-longitude grid is land or water and drew their answers as world maps. Asked in Korean, with coordinates written as north latitude and west longitude, the 9B's map becomes nearly symmetric left to right (east-west symmetry 0.84, against −0.14 when asked in English) and partly top to bottom: it loses track of which hemisphere a point is in. Write the same coordinates as 37°N 123°W and the left-right symmetry disappears.
Can a 32 GB Mac Run a Model Three Times Its Memory? Qwen3.8-Flash-Next on an M2 Max
Qwen3.8-Flash-Next's 4-bit weights are about 105 GB. With slotstream, which reads the model's experts from the SSD, a 32 GB MacBook Pro (M2 Max) ran it at 9.5 tokens per second for short replies and read an 8K prompt in 55 seconds. It scored 280/300 on GSM8K and 156/164 on HumanEval, and gave byte-identical answers across repeats and across two memory plans. A long job of 300 questions took 3.6 hours, and one long request was dropped for memory pressure and succeeded on retry. Against Qwen3.5-9B, which fits in memory, GSM8K couldn't be told apart (93.3% vs 91.0%) and HumanEval solved 18 more (95.1% vs 84.1%), while the 9B decoded 3.5 times faster.