Models & Algorithms
105 posts in this category
LoRA From Scratch in 19 Lines: It Matches PEFT Exactly, and Saved Less Memory Than We Predicted
Build a LoRA layer in PyTorch, put it into Qwen2.5-0.5B and check it against PEFT: identical outputs and identical training for 30 steps. Then measure what it saves on the GPU: 34% of peak memory at 512 tokens in fp32, not the half we predicted.
Claude Code on a Local Qwen3.8-Flash-Next on a 12 GB GPU Budget: llama.cpp or Strata
Point Claude Code at a local Qwen3.8-Flash-Next held to 12 GB of VRAM by changing a few environment variables, with llama.cpp or Strata. Reading the roughly 28K-token first request took llama.cpp 139-161 s and Strata about 12 s. Strata refuses every request once the conversation passes about 33,500 tokens because of Claude Code's 32,000-token output cap, so set CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192 with it.
The DeepSeek-V2-Lite GGUF We Downloaded Ran Without Its Small KV Cache: Check Two Tensor Names First
DeepSeek-V2-Lite's attention should keep a 32K-token KV cache at 972 MiB in llama.cpp. The 2024 Q4_K_M GGUF we downloaded used 8,640 MiB because it lacks the attn_k_b and attn_v_b tensors. Converting the original weights ourselves brought the cache back to 972 MiB and the process from 20,162 to 11,330 MiB. Here is how to check a file in one command and convert it if needed.
Five LoRA Variants on One Table, and the Setting or Condition Each Abstract Leaves Out
LoRA, LoRA-FA, VeRA, Delta-LoRA and LoRA+ on one table: which of W, A and B each trains, where each paper says the saving is, and the setting or condition you only find past the abstract.
Pick llama-server's -c From Two Short Runs, Not a VRAM Estimate: Load Memory Within 1 MiB at Seven Checked Lengths, and the Estimate Overshot Gemma 3 at 128K by 13.7 GiB
We started llama-server at 4K and 8K on Qwen3-8B, Qwen3.5-9B and Gemma 3 4B, drew a line through the two memory readings, and picked the longest -c under a memory budget. At the seven lengths we then started, the line predicted load memory within 1 MiB. Prompt processing added 12 to 14 MiB and pushed two picks over budget, so we subtract 16 MiB more. A simple all-layers estimate overshot Gemma 3 at 128K by 13.7 GiB.
Harness² Selector Code: What It Sees and What It Never Sees
Reading google-research/harness2 (commit 1cfe254): the step that picks one of k harness candidates with no answer key sees only the request, the source material and anonymized, untruncated outputs. A failed pick records the first final by label but is left out of scoring, and when a successful pick leads the reflector to write a lesson, the pick's stated reason is stored verbatim in the library next to the adopted harness.
Harness²: A Loop That Edits the Harness Without an Answer Key
Harness² (arXiv:2610.14224) makes the harness around a model eight editable parts and recurses on it with no ground truth: 11.5 points on WorkBuddy-Bench from one improvement step (GLM-5 under OpenCode), tested on the OpenCode and Codex substrates, and a selection step, active only above k=1, that the authors admit can pick the weaker candidate. We read the paper and ran the released demo locally.
Claude Code, Codex, OpenCode, Hermes: How Many of the Eight Harness Parts Do They Open Up?
We compared four agent harnesses (Claude Code, Codex, OpenCode, Hermes Agent) against the eight parts named in the Harness² paper, checking in each tool's official documentation whether a part is exposed as a file or setting, and opening some of the installed tools ourselves. In the docs we read, none of the four opens all eight.
What Is an Agent Harness, and What Does Harness Engineering Do?
An agent is a model plus a harness: the prompts, rules, memory, tools and scripts around it. Where the term harness engineering comes from, the eight parts Google's Harness² paper edits, and what the practice looks like in a repository you already have.
DeepSeek's Config Says 16 KV Heads, Its Cache Holds a Ninth of That: Seven Ways LLMs Handle Attention, and Which Models Use Each
DeepSeek-V2-Lite lists 16 KV heads, which predicts 8,640 MiB of cache at 32K; llama.cpp allocated 972 MiB. A guide to MHA, MQA, GQA, sliding window, MLA, sparse attention and linear-attention hybrids: what each stores, which models use it (checked against their config.json), and what it gives up.
Implementing the Attention Methods Used in LLMs: From Multi-Head Attention to Gated DeltaNet
MHA, MQA, GQA, sliding window, attention sinks, MLA with weight absorption, DSA-style sparse attention and Gated DeltaNet, each as a short PyTorch function with a check that it does what it claims. The MLA check runs on DeepSeek-V2-Lite's real layer-0 weights; the Qwen3.5-9B state size from the code matches llama.cpp's 50.25 MiB.
Four Times the Context, Less Memory: How Six Models Store the KV Cache, Measured in llama.cpp and vLLM
In llama.cpp, Qwen3.5-9B with a 128K context allocated 4,146 MiB of KV cache; Qwen3-8B needed 4,608 MiB for 32K. We calculated the cache from config.json for six models with four kinds of cache (full attention, sliding window, MLA, linear-attention hybrid), measured it in llama.cpp and vLLM, and list what to plan with for each.
llama.cpp vs vLLM: Deployment Methods and Strategies, Part 1: Memory
Deploying the same models on llama.cpp and vLLM, part 1: memory. What each runtime reserves for the KV cache and why (Gemma 3 4B at 32K: 814 MiB against 2,682 MiB), the settings that move it (-ub, --swa-full, max_num_batched_tokens, prefix caching), and four other differences: lengths past training, --parallel, GGUF conversion and vLLM's startup kernel build.
llama.cpp vs vLLM: Deployment Methods and Strategies, Part 2: Speed and Concurrent Requests
Measured on one A100 with Qwen3-8B in bf16 on both runtimes: one request generates at the same speed (86 tok/s), vLLM reads prompts faster, and at 64 concurrent requests vLLM produced 1,585 tok/s against llama.cpp's 430 (768 with its prompt cache off). Plus a llama.cpp default that stalled 64 requests for 144 seconds, what vLLM's lower memory reserve costs, and seven predictions scored.
Ten Minutes to Find Out Whether the Tests a Local Model Wrote for Your Function Protect a Bug
One script, two versions of your function, a local model: it asks the model for pytest tests from your current version only, runs every test against both versions, and tells you which tests would pass today and fail after the fix. On a date function with an off-by-one, Qwen3.5-9B wrote 10 tests in 6 seconds; 3 asserted the bug and none caught it. The same scorer and prompt as our 164-problem measurement.
A Decision Model Caught Every Look-Alike You Can't See by Eye. It Missed scotlabank.com.
Jev caught all 60 addresses with a look-alike letter from another alphabet, the ones you cannot tell apart by eye, and missed ten plain typos like scotlabank.com and d0cusign.com. Kev-9B did the opposite, calling 39 of those 60 real. The checker in this post uses no model, catches all 300 of our test addresses (a number we explain away below), and nothing you paste leaves your browser.
Show a Local Model Buggy Code and Its Tests Protect the Bug, Even When It Notices
We asked Qwen3.5-9B, running locally, to write pytest tests for all 164 HumanEvalPack Python functions, once from the buggy version and once from the fixed one. From the buggy code, 16.4% of its tests passed on the bug and failed on the fix, against 2.6% from the fixed code, and the share of tests that catch the bug fell from 41.9% to 19.5%. With thinking on, on the 85 problems both modes finished, misguided tests rose from 22.4% to 37.6%. A paper's fix, giving the model a spec it wrote from the buggy code instead of the code itself, did not show an effect here, and about half of those specs described the bug as intended behavior.
One Letter Holds a Language Model’s World Map Together
We asked Qwen3.5 models whether each point on a latitude-longitude grid is land or water and drew their answers as world maps. Asked in Korean, with coordinates written as north latitude and west longitude, the 9B's map becomes nearly symmetric left to right (east-west symmetry 0.84, against −0.14 when asked in English) and partly top to bottom: it loses track of which hemisphere a point is in. Write the same coordinates as 37°N 123°W and the left-right symmetry disappears.
Can a 32 GB Mac Run a Model Three Times Its Memory? Qwen3.8-Flash-Next on an M2 Max
Qwen3.8-Flash-Next's 4-bit weights are about 105 GB. With slotstream, which reads the model's experts from the SSD, a 32 GB MacBook Pro (M2 Max) ran it at 9.5 tokens per second for short replies and read an 8K prompt in 55 seconds. It scored 280/300 on GSM8K and 156/164 on HumanEval, and gave byte-identical answers across repeats and across two memory plans. A long job of 300 questions took 3.6 hours, and one long request was dropped for memory pressure and succeeded on retry. Against Qwen3.5-9B, which fits in memory, GSM8K couldn't be told apart (93.3% vs 91.0%) and HumanEval solved 18 more (95.1% vs 84.1%), while the 9B decoded 3.5 times faster.
A 2-Bit Model Remembers How to Be Right and Forgets How to Be Sure
Cutting decider-4b to 2-3 bits kept its accuracy (86.4% → 85.6%) but took away its decisiveness, and the error pointed toward the base model it was fine-tuned from (cosine 0.29 for the Q2_K file, against about 0.02 for a random direction). Random noise of the same size, five seeds, also made it less decisive, often more so, but had no such direction (−0.04 to 0.07) and moved accuracy between 81.6% and 89.6%. Noise only leaned that far once it was big enough to drop accuracy to between 13% and 32%. One refitted temperature brings the probabilities back.
How Much of an Uncertainty Score Is Just Answer Length? U-Space on Qwen3.5-4B
A new paper finds that answer length alone flags a language model's wrong answers almost as well as many uncertainty scores. We ran the authors' code on Qwen3.5-4B with their items: length alone reached AUROC 65.3 on three benchmarks, from 75.2 on TriviaQA to 48.6 on SuperGPQA. After controlling for length, U-Lens beat MSP (+8.4) and length alone (+4.4 before control), but its edge over mean log-loss (+1.1) was too small to call. On the three benchmarks in the main comparisons up to a third of answers hit the 16,384-token limit and were excluded; on Omni-MATH, left out of the comparisons, 92.5% did.
How Far Can You Shrink a Decision Model? decider-4b from BF16 to Q2_K on llama.cpp
We ran the open decision model decider-4b at five GGUF sizes, from BF16 (10.4 GiB of GPU memory) to Q2_K (4.4 GiB), on 500 TREC questions. Accuracy differences were too small to detect at every size. The probabilities moved: log-loss got slightly better at Q4_K_M and worse at Q8_0 (barely), Q3_K_M and Q2_K. Q2_K changed 13% of the answers and Q3_K_M cut the share of answers above 0.9 from 40% to 23%. Q4_K_M halved the memory; no accuracy difference was detected and its log-loss improved.
Can You Act on Microsoft-Decision-1's Probabilities? We Swapped the Labels on 500 Questions
We sent 500 TREC questions to Microsoft-Decision-1 and Jev 1.13 through OpenRouter, and ran two local decision models, decider-4b and laya, with normal labels and with option names swapped against their definitions. Decision-1 followed the definitions (95.2% against 96.4%, not a difference by our rule) and at a 0.9 threshold handled 80% of the swapped cases on its own, 5 of those 400 wrong. laya followed the labels and was wrong on 110 of the 115 swapped cases it would have handled. The same request sent twice gave slightly different probabilities on Decision-1.
Calling Microsoft-Decision-1 on OpenRouter: Six Things We Hit on Day One
A short field guide from our first day with Microsoft-Decision-1 through OpenRouter: it uses the System One endpoint, not chat completions; choice options must be a map; confidence is not the top probability; the model ID carries a date; requests hit rate limits; and the same question cost us 3.5 to 4.8 times as much on Jev. Working Python included.
Does Codemode Work With Small Local Models? Qwen3.5-9B Got 28 of 30 Writing Code, 19 Calling Tools
Armin Ronacher writes that codemode, where the model writes code that calls tools instead of calling them one at a time, does not yet work with smaller models. We tested it on 30 tasks against a mock issue tracker with Qwen3.5-4B, Qwen3.5-9B and Qwen3.8-27B in llama.cpp. Codemode got more tasks right at every size (22 vs 18, 28 vs 19, 30 vs 26); only the 9B difference passes our test (p = 0.012), and most of it comes from tool-calling replies cut off at our 2,048-token output limit. On tasks both modes got right, codemode's median tokens per task were 28-45% of tool calling's.
Speculative Decoding Speedups Depend on the Task: Prompt Lookup 6.2× on a File Edit, DFlash 0.7× on an Essay (Qwen3.5-9B, llama.cpp)
We measured three kinds of speculative decoding in llama.cpp on one Qwen3.5-9B file, alone on an A100: prompt n-gram lookup, the model's own MTP head, and a DFlash draft model. Rewriting a 130-line file ran 6.2× faster with n-gram lookup and 3.2× with DFlash. On a 500-word essay, MTP was 4% faster, n-gram lookup slightly slower and DFlash 0.69×. At temperature 0, n-gram and MTP output matched the plain run token for token; DFlash output diverged on three of four tasks.
How Much Context Fits on an 8 GB GPU? Qwen3.5-9B From 8K to 256K, Against Five Real Tasks
We measured GPU memory for Qwen3.5-9B Q4_K_M at 8K to 256K context and counted the tokens of common tasks. On an 8 GB card one conversation gets 32K with the default KV cache and 64K with an 8-bit cache. A 20-turn coding chat reached 36,736 tokens, a 76-page paper 77,800, and coding agents send 648 to 15,970 tokens before you type a second message. Qwen3-8B, which keeps a cache in every layer, already needs 9.5 GiB at 32K.
We Trained Our Own Decision Model with Unsloth: 75% on BANKING77, 56% When BANKING77 Is Left Out of Training
We followed Unsloth's recipe for turning Qwen3.5-0.8B into a decision model (4.15 GB peak memory, about 20 minutes per run on a shared GPU). BANKING77 and CLINC150 matched Unsloth's table (75.4% and 75.0%), but those test rows come from datasets in the training mix. With the two datasets removed from training, the same 500 rows scored 56.5% and 55.9%. typed-decisions came out 11.8 points below the table; the recipe's decontamination step drops 850 of its 1,200 training cases, and in a post hoc run that kept them it reached 74.3%.
GPT-2 with ReLU² Instead of GELU: 0.051 Lower Loss After 1B Tokens, From the Same Starting Weights
Three GPT-2 (124M) runs with ReLU² in the MLP ended at 3.623 mean validation loss after 1B FineWeb-Edu tokens, against 3.674 for GPT-2's GELU, at GPT-2's learning rate. Each pair started from identical weights, so the activation is the only difference. The gap clears the series' pre-registered bar (0.010) five times over and was below it at no checkpoint.
Pretraining Without Backprop, Rerun: What Happens to Dust When Backprop Gets Adam or More Epochs?
We reran Q Labs' Dust at 1M tokens with its own code. Under the paper's SGD setup our numbers matched the paper's, and with 20 backprop seeds the gap between Dust at 1,024 draws and backprop was under 0.02 (5.938 vs 5.941), smaller than the paper's 0.025. Backprop with the paper's Adam settings reached 5.384 in the same single epoch, and 16 epochs over the same 1M tokens reached 5.209 with SGD while using about 5% of Dust's compute. Our first Adam run came out worse than SGD; the paper's own settings fixed it, and we note what to check in your own runs.
Where Decision Models Go Wrong: Four Situations Behind the Sentences Most of Them Miss
We re-read the per-sentence results of up to 14 Jev-style decision models on six datasets and kept the 433 sentences at least half of them got wrong. Grouping those sentences and checking a few simple features showed four recurring situations: the question's shape and the dataset's rule disagree, categories overlap, the sentence contains another option's name, and no option fits. Examples, a full table of groups, and what to check in your own options.
Is EmbeddingGemma 2 Better Than EmbeddingGemma 1 for Korean Search? We Tested It on 177 of Our Own Posts
We searched our 177 published Korean/English post pairs with each post's description as the query. With Korean queries, EmbeddingGemma 1 ranked the right post first more often than EmbeddingGemma 2: 162 vs 153 posts in Korean (p = 0.022) and 164 vs 155 when the target was the English version (p = 0.035). English-to-English showed no difference. Pre-registered, text only.
Does laya Read Your Definitions or Your Labels? A Pre-Registered Rerun of the Label-Bias Test
We reran the 'labels override definitions' tests from arXiv 2610.02586 on two laya 0.3.7 checkpoints with our own TREC and AG News sets. Changing one line of prompt rendering made predictions identical whatever the labels were called. When labels contradicted the definitions, laya's TREC accuracy fell from 88.6% to 22.2%, and its top probability ranked right and wrong answers backwards (AUROC 0.36), the same direction the paper reports.
GPT-2 with Parameter-Free RMSNorm: 0.038 Higher Loss After 1B Tokens; QK Norm Won Back 0.012, Barely Past the Bar
Swapping all of GPT-2's LayerNorms for parameter-free RMSNorm raised the mean validation loss after 1B FineWeb-Edu tokens from 3.674 to 3.712, three runs each at GPT-2's learning rate. Adding QK norm on top brought it to 3.701, a 0.012 gain that clears this series' pre-registered bar (0.011) by 0.001. Throughput differences stayed within run-to-run variation, and the logged gradient norms showed no instability to fix.
GPT-2 with RoPE Instead of Learned Position Embeddings: 0.122 Lower Loss After 1B Tokens, 5% Slower
Three GPT-2 (124M) runs with rotary position embeddings ended at 3.552 mean validation loss after 1B FineWeb-Edu tokens, against 3.674 for three runs with GPT-2's learned position embeddings. The gap clears this series' pre-registered bar (0.012) by ten times, at GPT-2's learning rate. RoPE ran 5.3% slower on this code.
Same GPT-2, Same Data, Three Seeds: The Final Loss Spread 0.018 Nats
Three GPT-2 (124M) runs on 1B FineWeb-Edu tokens that differed only in the random seed ended at 3.668, 3.669 and 3.686 validation loss (sd 0.010). Seed 0 run again with the same command landed 0.0035 away. With three seeds per side, a change has to beat 0.016 nats before this series calls it real.
Karpathy's “Let's Reproduce GPT-2” in One Read: The Four Parts, Their Numbers, and What Changed Since
A condensed walk through the 4-hour video: building GPT-2 (124M), taking a step from 1,000 ms to 90 ms on one A100, the GPT-3 training settings, and a 10B-token run that beat OpenAI's checkpoint on HellaSwag. Plus one check we ran: nanoGPT's exact GELU moves logits by up to 5 against GPT-2's.
Strata vs llama.cpp on the Same Qwen3.8-Flash-Next File: No Score Difference Found, Faster Within 12 GiB
Same IQ2_XS file in both engines: GSM8K 93.3% vs 93.0%, HumanEval 157/164 each, no vision gap showed up on 100 COCO images. Held to 12 GiB, Strata (draft decoding on) decoded 62.9 tok/s to llama.cpp's 23.4.
llama.cpp KV Cache Quantization: Which -ctk and -ctv to Use, Measured on the Current Build
The -ctk/-ctv (--cache-type-k/-v) settings to copy, the allowed types, and the GGML_CUDA_FA_QUANTS build option. Measured: q8_0 saved 4.2 GiB at 64K and decoded at 56% of f16's speed.
What Are You Deciding, and Which Mistakes Are Expensive? Chapter 1 of a Book on Building Decision Systems
Free chapter: before training anything, count what each intent can tell you, find the intents likely to be confused, and write down what each mistake costs. The code runs in under a second.
Which Mistakes Are Expensive? One Routing System Re-scored at Seven Prices for a Wrong Answer
Re-scoring 3,080 BANKING77 messages: the price of a wrong answer moved the threshold far more than the choice of model, and decision models paid only when a wrong answer cost two hand-offs or less.
How Far Does a Simple Classifier Get? Chapter 2 of a Book on Building Decision Systems
Free sample chapter: split BANKING77 before training, then two CPU classifiers reach 91.2% and 92.9% on the test set, with code that runs in about a minute.
Which Messages Should Go to a Person? A Classifier, a Decision Model and a Hand-Off, Measured
On BANKING77 a decision model after a classifier saved no hand-offs. On CLINC150 unknown questions broke the thresholds; adding 250 to validation halved the leaks.
Classifier, LLM or Decision Model? A Measured Guide to Text Classification
One path through every text-classification measurement on this blog: on the same 154 banking messages, a CPU classifier scored 90.3%, an LLM with five retrieved examples 94.8%, and Jev 76.0%. Which to use, and when.
Jev-Style Decision Models in Ollama 0.35: Nimble and Tev1, Setup and Benchmark
Jev itself is not in Ollama, but Ollama 0.35 adds a Jev-compatible API with Nimble and Tev1. How to call /v1/systemone, its 26-option limit, and how the models scored against Jev on the same questions.
Jeff vs Jev on the Same Questions: Overall Scores and a 26-Option Limit (Fixed in v1.1)
On Jeff's own 4,599 questions, Jev scored 85.7 and Jeff-2B 83.0. Jeff v1.0 never picked an option past the 26th in my tests; v1.1 fixes that, remeasured.
Kev vs Jev: Kev 0.8B to 9B Benchmarked for Accuracy and Calibration
Kev (0.8B, 4B, 9B) and Jev 1.13 on the same 1,629 labelled messages (2,329 requests). On the three datasets Kev trained on, Kev-9B was as accurate or more (TREC 93.8% vs 89.0%). On three it never saw, Jev was ahead on two (CLINC150 68.5% vs 62.0%, MASSIVE 82.9% vs 76.0%). Jev's probabilities ran high and come rounded to two decimals, so on BANKING77 no threshold reached 95% accuracy.
How to Run Qwen Locally: Which File Fits Your GPU (8, 12, 16 or 24 GB), Measured
Which Qwen GGUF fits an 8, 12, 16 or 24 GB GPU, with measured memory: 9B Q4_K_M 5.8 GiB, 27B Q3_K_XL 13.1 GiB, 35B-A3B 7.3 GiB with 30 layers' experts on the CPU.
How Many Labels Is a Decision Model Worth? Kev on Three Tasks It Never Saw
On CLINC150, MASSIVE and financial-news tweets, none of which Kev trained on, Kev-9B matched a small classifier trained on roughly 2 to 20 labelled examples per class. Its probabilities ran too low: stated confidence sat 10–17 points under its accuracy (calibration error 11–17%, against 2–9% on familiar data), so a 95%-accuracy threshold passed only 23–58% of messages. Out-of-scope questions got low probabilities: 5 of 100 passed that threshold. The 0.8B model answered 'Financials' to 152 of 200 tweets.
Kev vs MoJev: Two More Open Jev Alternatives, Accuracy and Confidence Measured
Kev (0.8B to 9B) and MoJev (0.85B) both speak Jev's API. On BANKING77, TREC and AG News with human labels, Kev's probabilities were usable: at 95% accuracy it could auto-accept 53–61% of BANKING77, where laya managed 0%. But Kev was trained on these three datasets, and a small classifier trained on the same data still beat it on BANKING77. MoJev reports 0.79% calibration error; here it was 6–22%, with probabilities too low.
Can You Trust a Decision Model's Confidence? Repeated Answers and Probabilities, Measured
Decision models sell a probability with every answer, so you can act on the sure ones and escalate the rest. I checked both ways of judging 'sure' on human-labelled data. Across 8 noise draws, 77–80% of openjev's wrong answers were unanimous. Probabilities did better, but how useful they were varied sharply by task: laya could auto-accept 88% of TREC at 95% accuracy and 0% of BANKING77. The new CLM-8B stayed near chance on all three datasets when given label names.
Jev in 10 Minutes: What a Fast Decision Model Is, and When It Is Worth Using
Jev answers a question with a probability for each option instead of writing text. What that is for, how open projects do the same thing, and what three rounds of measurement showed: with a handful of options a 421M model reached 86.6% in 23 ms, with 77 a small trained classifier on a CPU still led at 90%.
Jev Alternatives Benchmarked: laya vs openjev vs NanoJev on Three Public Datasets
I ran laya, openjev and NanoJev on BANKING77 (77 intents), TREC (6 question types) and AG News on one A100. With six options, laya's 421M encoder reached 86.6% on TREC in 23 ms. With 77, the DiffusionGemma-based openjev led at 66.9%, and a logistic regression on a CPU beat everything at 90%.
DiffusionGemma on vLLM's Example Server: One Sentence of Context Took It From 54.5% to 31.8%
I ran the example server from the merged vLLM pull request that turns DiffusionGemma into a decision model on 154 BANKING77 messages. It caps a question at 26 options, so I split the 77 intents into nine groups and read twice: 54.5% at 119 ms with one noise draw, 51.3% with the default draws. My transformers version of the same two-stage read scored 68.2%, and one added sentence describing the input dropped the server from 54.5% to 31.8%.
Qwen-Image-2.1 on One A100: What I Could Count, and What the OCR Got Wrong
8.2 seconds for a 1024x1024 image at 20 steps, 56.5 GiB peak at 2048x2048, and thirty sign images I checked myself all carried the text I asked for — while Tesseract found it in twelve. Editing is the weak half: four of eight mask-free instructions were followed, and the median image changed 84% of its pixels.
Same Model, Five Ways to Build the Input: Only Retrieval Moved It Significantly
On BANKING77 with GPT-5.6 Terra held fixed, reordering labels, grouping them and changing reasoning effort each moved one or two items in both directions against the baseline, with no significant paired difference. Attaching five retrieved training examples scored 146 of 154, fixing 19 items and breaking none (exact McNemar p about 3.8e-6).
llama.cpp -ctk and -ctv: The Default CUDA Build Supports Four Values (September 2026 Build, b10753)
llama-bench accepts eight KV cache types and llama-server accepts nine, but the default CUDA build compiles FlashAttention kernels for only f16, bf16, q8_0 and q4_0, and only when K and V match. Every other setting has no kernel and prefills at 83-284 tokens per second against about 4,600. Measured on llama.cpp 69320fe on an A100.
Jev's Speed Claims: Benchmarking Label-Only Alternatives on BANKING77
Four ways to get only a label on BANKING77's 77-intent test set, compared on the same 154 messages. Pretrained MiniLM embeddings with a logistic regression reached 90.3% accuracy at a 6.8 ms median on CPU. Follow-up posts measure open-source Jev alternatives on the same messages.
llama.cpp KV Cache Quantization: Why q8_0 Costs 9% of Throughput — or 22%
Mainline llama.cpp on one A100, Qwen3-8B Q4_K_M, llama-server with 1 to 32 concurrent slots. On a 32K prompt, q8_0 cost 9% of server throughput when each request generated 128 tokens and 22% when it generated 1,024, because prefill dominates the short case and prefill is unaffected by the KV type. Per-token decode was 34% slower, matching llama-bench. VRAM in use after startup fell from 41.0 GiB to 25.1 GiB at four 64K slots.
llama.cpp KV Cache Quantization, Measured on One A100 — q8_0 Is Free at 4K and Costs Half Your Decode Speed at 64K
Mainline llama.cpp, Qwen3-8B, one A100: -ctk q8_0 -ctv q8_0 matches f16 perplexity and cuts the 32K cache by 2.1 GiB, but decode at 64K depth drops to 55% of f16 (q4_0: 50%). Two other settings, q5_1 and a q8_0/q4_0 mix, silently ran prefill on the CPU at 43 and 63 tokens per second.
KV Cache Reduction, Measured on One A100 — Part 1: The Twelve Techniques Don't Pay in the Same Currency
Every list of KV cache techniques presents a dozen of them as peers. On one A100 the spread is enormous: prefix reuse cut a shared-prefix workload from 59.4s to 6.2s, while tuning the paged block size moved capacity by under 1%. The reason you cannot rank them on one number is that they pay out in different units.
TurboQuant in vLLM on One A100 — Capacity, Speed, and Accuracy of All Four Presets on an 8B Model
vLLM 0.28, Qwen3-8B bf16, one A100 80GB: KV capacity, batched throughput, 32K decode, needle-in-haystack, and GSM8K for bf16, fp8, and all four TurboQuant presets — the 8B size vLLM's own study skipped.
TurboQuant From Scratch on Real KV Tensors — What 3 Bits Actually Cost, and Why the Forks Beat the Paper's Layout
PolarQuant in 60 lines of PyTorch on real KV from Llama-3.2-1B and Qwen3-8B: 3-bit costs +10% perplexity, k8v4 +0.2%, QJL only pays below 4 bits, and the block-32 layout explains half the forks' edge.
TurboQuant llama.cpp CUDA Fork, Measured on an A100 — turbo4 Matches q4_0, turbo3 Breaks at Long Context
Qwen3-8B Q4_K_M on one A100, six KV types: perplexity, prefill, decode-at-depth, and VRAM measured. turbo4 matches q4_0 quality and beats q8_0 decode 2.5x at depth; turbo3 triples perplexity at 32K context.
TurboQuant Status Check, August 2026 — What Actually Shipped in vLLM, llama.cpp, and Ollama
vLLM shipped it in v0.20 and published a sobering benchmark; llama.cpp upstream rejected it in June; Ollama's implementation is dead. We also correct our own earlier "merged in llama.cpp" claim — with links.
Can AI Detectors Actually Tell If You Used ChatGPT? -- We Built One and Measured
We implemented an AI-text detector ourselves and measured when it works, when it accuses innocent people, and how it gets defeated -- detection and false-positive rates, in numbers.
Why ChatGPT Gives You a Different Answer Every Time -- One Knob, Measured
We measure, on a local model and on GPT-4o-mini, exactly how much answers change with sampling temperature. Temperature doesn't create diversity -- it amplifies what's already there.
Does llama.cpp Support TurboQuant? Not in Mainline — How to Run turbo4 in a Fork
Mainline llama.cpp has no TurboQuant (PR #21089 closed June 2026). How to run turbo4 in the TheTom or buun fork, and why we measured turbo3 tripling perplexity at 16K.
TurboQuant Explained — Google's Extreme KV Cache Compression Algorithm
Compress KV cache to 3-bit with PolarQuant + Lloyd-Max. 4.6x memory savings with zero accuracy loss, no retraining.
Qwen 3.5 Fine-Tuning Practical Guide — Build Your Own Model with LoRA
Complete guide to fine-tuning Qwen 3.5 with LoRA/QLoRA. From 8GB GPU QLoRA setup to Unsloth optimization, GGUF conversion, and Ollama deployment.
Qwen 3.5 Local Installation & Setup Guide — From Ollama to vLLM
Step-by-step guide to running Qwen 3.5 locally. From 5-minute Ollama setup to production vLLM servers, plus optimal model size selection per GPU.
Qwen 3.5 vs DeepSeek V3.2 — The 2026 Open-Source LLM Showdown
Complete comparison of Qwen 3.5 and DeepSeek V3.2: architecture, benchmarks, hardware requirements, and practical recommendations.
Agentic RAG Pipeline — Multi-step Retrieval in Production
Build a full Plan-Retrieve-Evaluate-Synthesize pipeline. Unify vector search, web search, and SQL as agent tools. Add hallucination detection and source grounding.
Self-RAG and Corrective RAG — The Agent Evaluates Its Own Retrieval
Implement Self-RAG reflection tokens and CRAG quality-based fallback. Build retry/fallback logic with LangGraph conditional edges.
Why Agentic RAG? — Query Routing and Adaptive Retrieval
Diagnose naive RAG limitations, classify query intent, and route to the optimal retrieval source with LangGraph. Implement adaptive retrieval that skips unnecessary searches.
From Evaluation to Deployment — The Complete Fine-tuning Guide
Evaluate with Perplexity and KoBEST benchmarks, merge LoRA weights, and deploy with vLLM/Ollama/HuggingFace Spaces.
QLoRA + Custom Dataset — Fine-tune 7B on a Single T4 GPU
Fine-tune a 7B model on a T4 16GB with QLoRA. Dataset construction, training execution, Wandb monitoring, and Before/After comparison.
Mastering LoRA — Fine-tune a 7B Model on a Single Notebook
From LoRA theory to Qwen 2.5 7B model setup. 99.8% parameter reduction and 86% memory savings vs full fine-tuning, explained with code.
SDFT: Learning Without Forgetting via Self-Distillation
No complex RL needed. Models teach themselves to learn new skills while preserving existing capabilities.
Qwen3-Max-Thinking Snapshot Release: A New Standard in Reasoning AI
The recent trend in the LLM market goes beyond simply learning "more data" — it's now focused on "how the model thinks." Alibaba Cloud has released an API snapshot (qwen3-max-2026-01-23) of its most powerful model, Qwen3-Max-Thinking.
YOLO26: Upgrade or Hype? The Complete Guide
Analyzing YOLO26's key features released in January 2026, comparing performance with YOLO11, and determining if it's worth upgrading through hands-on examples.
RAG Evaluation: Beyond Precision/Recall
"How do I know if my RAG is working?" — Precision/Recall aren't enough. You need to measure Faithfulness, Relevance, and Context Recall to see the real quality.
Retrieval Planning: ReAct vs Self-Ask vs Plan-and-Solve
Now that we've diagnosed Query Planning failures, it's time to fix them. Let's compare when each of these three patterns shines.
Query Planning Failures in Multi-hop RAG: Patterns and Solutions
You added Query Decomposition, but why does it still fail? Decomposition is just the beginning—the real problems emerge in Sequencing and Grounding.
Multi-hop RAG: Why It Still Fails After Temporal RAG
You added Temporal RAG, but "who is my boss's boss?" still returns wrong answers. RAG now understands time, but it still doesn't know "what to search for next."
GraphRAG: Microsoft's Global-Local Dual Search Strategy
Why can't traditional RAG answer "What are the main themes in these documents?" Microsoft Research's GraphRAG reveals the secret of community-based search.
CFG-free Distillation: Fast Generation Without Guidance
Eliminating the 2x computational cost of CFG. Achieving same quality with single forward pass.
Consistency Models: A New Paradigm for 1-Step Generation
Single-step generation without iterative sampling. OpenAI's innovative approach using self-consistency property.
SDE vs ODE in Diffusion Models: Differences, Math and When to Use Each
Stochastic vs Deterministic. A deep dive into Score-based SDEs and Probability Flow ODEs, the theoretical foundations of DDPM and DDIM.
MMDiT Architecture Explained: Inside Stable Diffusion 3 and FLUX
From U-Net to Transformer. A deep dive into MMDiT architecture treating text and image equally, plus Rectified Flow and Guidance Distillation.
Rectified Flow: Straightening Paths Toward 1-Step Generation
Flow Matching still too slow? Reflow straightens trajectories for 1-step generation. The core technique behind SD3 and FLUX.
Flow Matching vs DDPM: Why ODE Beats SDE in Diffusion Models
DDPM needs 1000 steps, Flow Matching needs 10. The mathematics of straight-line generation. Comparing SDE curved paths vs ODE straight paths.
ViBT (Vision Bridge Transformer) Explained — Noise-Free Image and Video Generation (Paper Review)
What ViBT is and how the Vision Bridge Transformer skips diffusion's noise by learning a Brownian bridge from source to target. Covers the 20B image and 1.3B video models, stabilized velocity matching, up to 4× faster inference without a condition encoder, and the current limitations.
SteadyDancer Complete Analysis: A New Paradigm for Human Image Animation with First-Frame Preservation
Make a photo dance - why existing methods fail and how SteadyDancer solves the identity problem by guaranteeing first-frame preservation through the I2V paradigm.
Still Using GPT-4o for Everything? (How to Build an AI Orchestra & Save 90%)
An 8B model as conductor routes queries to specialized experts based on difficulty. ToolOrchestra achieves GPT-4o performance at 1/10th the cost using a Compound AI System approach.
SANA: O(n²)→O(n) Linear Attention Generates 1024² Images in 0.6 Seconds
How Linear Attention solved Self-Attention quadratic complexity. The secret behind 100x faster generation compared to DiT.
PixArt-α: How to Cut Stable Diffusion Training Cost from $600K to $26K
23x training efficiency through Decomposed Training strategy. Making Text-to-Image models accessible to academic researchers.
DiT: Replacing U-Net with Transformer Finally Made Scaling Laws Work (Sora Foundation)
U-Net shows diminishing returns when scaled up. DiT improves consistently with size. Complete analysis of the architecture behind Sora.
From 512×512 to 1024×1024: How Latent Diffusion Broke the Resolution Barrier
How Latent Space solved the memory explosion problem of pixel-space diffusion. Complete analysis from VAE compression to Stable Diffusion architecture.
DDIM: 20x Faster Diffusion Sampling with Zero Quality Loss (1000→50 Steps)
Use your DDPM pretrained model as-is but sample 20x faster. Mathematical derivation of probabilistic→deterministic conversion and eta parameter tuning.
DDPM Forward and Reverse Process — Deriving the Diffusion Math Step by Step
Derives DDPM's forward process from the beta schedule, the closed form that jumps straight to any timestep, the reverse process and its posterior, the ELBO, and the reparameterization to noise prediction. Every equation worked through rather than quoted.
Why Your Translation Model Fails on Long Sentences: Context Vector Bottleneck Explained
BLEU score drops by half when sentences exceed 40 words. Deep analysis from information theory and gradient flow perspectives, proving why Attention is necessary.
Bahdanau vs Luong Attention: Which One Should You Actually Use? (Spoiler: Luong)
Experimental comparison of additive vs multiplicative attention performance and speed. Why Luong is preferred in production, proven with code.
Building Seq2Seq from Scratch: How the First Neural Architecture Solved Variable-Length I/O
How Encoder-Decoder architecture solved the fixed-size limitation of traditional neural networks. From mathematical foundations to PyTorch implementation.
AdamW vs Lion: Save 33% GPU Memory While Keeping the Same Performance
How Lion optimizer saves 33% memory compared to AdamW, and the hyperparameter tuning guide for real-world application. Use it wrong and you lose.