A 2-Bit Model Remembers How to Be Right and Forgets How to Be Sure

A 2-Bit Model Remembers How to Be Right and Forgets How to Be Sure

This post starts with a prediction we got backwards.

We were checking whether one number, a temperature, could make a decision model's probabilities honest. Decision models such as decider-4b don't write text; they return a probability for each option you give them. A temperature below 1 sharpens those probabilities and above 1 flattens them. Before computing anything we wrote down: the fewer bits we keep, the more over-confident the model gets.

On 500 TREC questions with normal labels, the best-fitting temperature went the other way, falling with every cut: 0.80 at BF16 and Q8_0, 0.77 at Q4_K_M, 0.69 at Q3_K_M, 0.68 at Q2_K. Below 1 means the model's confidence was lower than its actual hit rate. Accuracy hardly moved (86.4% at BF16, 85.6% at Q2_K). The clue had been in our own earlier post, where answers at 0.9 or above fell from 40.2% to 23.2% at Q3_K_M. We had the number and still predicted from the intuition that "rounding error makes a model brash".

So we set out to find out why, writing down a prediction with a pass/fail rule before every measurement. Here is what we found first, and then how we got there.

What we found

decider-4b is Qwen3.5-4B-Base plus fine-tuning. Fine-tuning raised accuracy on these 500 questions from 82.0% to 86.4%, and it made the model more decided: the median lead of its top option over the second (the gap between their raw scores) went from 2.21 to 2.47.

At 2 bits, decider-4b kept most of the accuracy (85.6%) and lost the decisiveness (median lead 2.21). The numbers matching the base model exactly is probably a coincidence: Q3_K_M went lower than the base model (2.11), and random noise went lower still. What is solid is the direction. We saved the model's internal state at the answer position for every question and compared it with BF16's. The error the 2-bit body introduced points toward where the base model's state would be: cosine 0.29 on average, against about 0.02 for a random direction. At Q3_K_M it was 0.47. This lean is only part of the error: about 30% of the Q2_K error's size lies along the base-model direction, and most of the rest has no particular direction.

The answers shift the same way.

The 2-bit model moved between BF16 and the base model; Q8_0, Q4_K_M and Q3_K_M stayed within a few answers of BF16. At 2 bits, 66 answers changed from BF16, and 44 of them now matched the base model's answer. This is also why accuracy barely moved. Of the 66 changes, 32 went from right to wrong and 28 from wrong to right. Take the biggest shift alone, 41 answers that moved from "entity" to "description": 21 of those questions really were descriptions and 15 were entities. The answers moved toward the base model, not toward the correct labels, so gains and losses roughly cancelled.

In one sentence: 2-bit quantization keeps how to be right and cuts how sure the model is, and what it cuts points back toward the model before fine-tuning.

Why the fine-tuning, of all things?

Something here is odd. The model doesn't know which of its weights came from pre-training and which from fine-tuning. Fine-tuning only nudged numbers that were already there, and quantization only sees the resulting numbers. Yet quantization behaves as if it forgets what was learned last. The rest of this post follows that question.

Not just any noise does this

The obvious objection: maybe any damage of that size does the same. So we took the BF16 model and, for every body tensor that Q2_K quantizes, added Gaussian noise with exactly the same total size as that tensor's Q2_K error. We left the shared embedding table at BF16, matching the 2-bit-body file below, and ran five seeds.

The noise made the model less decided too, by as much or more in every seed. So the drop in confidence is not something only quantization does. But the noise didn't lean toward the base model, and its accuracy landed anywhere from 81.6% to 89.6%. One seed scored above BF16. With 500 questions the standard error of an accuracy is about 1.5 points, so read that spread as "noise moved accuracy unpredictably", not "noise ruined it" or "noise helped". The 2-bit file is one deterministic result, not a distribution, so we don't compare its stability with the noise. What we can say is that at the same size, its accuracy stayed within a point of BF16 while three of the five noise seeds landed more than 3 points away (89.6%, 82.6%, 81.6%).

We then made the noise look more like quantization error. That error varies from block to block; where a block's weights are larger, so is the error. So for every group of 16 weights we measured the size of the actual Q2_K error and added random noise of that size, again with five seeds. Same outcome: temperature fell to 0.46-0.68, but the cosine toward the base model ranged from −0.06 to 0.09 (median 0.05) and the base-model coefficient from −0.027 to 0.024. That coefficient is how far the hidden state moved from BF16 toward the base model, as a fraction of the distance between the two (median over questions). The base model's state is far away (1.6 times the size of the BF16 state), so a cosine of 0.3 corresponds to a coefficient of about 0.07. On the same scale the 2-bit-body file scored 0.077.

What quantization has that the noise lacks is the direction. Matching the size, and even its block structure, did not produce it.

We checked two more things. llama.cpp's k-quant matrix multiply also cuts activations to 8 bits during the computation, which the noise files don't do. So we wrote the Q2_K file's values out as BF16 and ran them with BF16 math. The result matched the Q2_K file (cosine toward the base model 0.29, temperature 0.70, accuracy 85.4%), so the cause is in the values, not the arithmetic. The Q8_0 file's cosine was 0.01, the level of a random direction.

Then we made the noise bigger: 1.45 times the size of the Q2_K error (why that size is explained in "The open question" below). Now the noise did lean toward the base model, with cosines from 0.10 to 0.31 across five seeds. But by then the model was broken, at 13.4% to 32.0% accuracy. So the base model is also where the model drifts when it is badly damaged. The 2-bit quantization reached a lean of 0.29 while keeping 85% accuracy, which same-size noise never did.

How we got there

We guessed at the cause three times. For each guess we committed predictions with pass/fail rules and a timestamp before measuring. From round 2 on, the design came after seeing the earlier results.

Two facts about the model matter. decider-4b has no separate output head; the table that turns tokens into vectors at the input (token_embd) is reused at the output, and decider reads the answer by multiplying the hidden state at the answer position by that table's option-letter rows. llama.cpp's stock Q4_K_M, Q3_K_M and Q2_K files all keep that table at 6 bits. What goes to 2-3 bits is the body: in Q2_K, 176 tensors are Q2_K, 40 are Q3_K and 32 are Q4_K. The noise files above went into all 248 of those tensors. All runs used decider-ai 1.6.0 with llama-cpp-python, the same 500 questions, and decider's shipped temperature of 1.11; the scores we compare are taken before that temperature.

Round 1: head or body? We built split files. A 2-bit body with a BF16 table fitted temperature 0.70, like Q2_K (0.68); a BF16 body with a 6-bit table fitted 0.80, like BF16. The cause is in the body. The median lead narrowed from 2.468 to 2.205 at Q2_K (−10.7%). The explanation we paired with it, that quantization squashes the largest weights, was wrong: the top 0.1% of weights by size kept 99.0% of their size on average (95.7% to 100.3% per tensor). A BF16 body with only the table cut to 2 bits moved the other way (T 0.91). That table is also the input embedding, so this file changes both what the model reads and how it answers, and we can't tell which moved it.

Round 2: is the signal diluted? Small errors at every layer, followed by a final normalization to a fixed size, could shrink the share of the original "answer direction". We saved the hidden state at the answer position; multiplying it by the table's option rows reproduces the recorded scores to within 0.11, so it's the right vector. The prediction that its size stays the same was wrong: it grew (1.09x at Q2_K). So we split each Q2_K state into the part along the BF16 state and the part at right angles (exploratory). Averaged over the 500 questions, the BF16 winner's lead over its runner-up is 2.518. The part along the BF16 state contributes 2.561, 1.7% more; the right-angle part contributes −0.403; together they give the 2.158 that the same two options are apart in Q2_K. Nothing was diluted; the right-angle part cut the lead, and not at random. It lowered the winner's score relative to the average of the other five options by 0.78, and cut harder where BF16's lead was larger (correlation −0.35).

Round 3: does quantization undo the fine-tuning? The fine-tuning changed the weights by 3.8% of their size; the Q2_K error is 30%. We reasoned that on a coarse grid a small change rounds the same as no change, so the error should contain the fine-tuning in reverse. Weight by weight that was wrong: the cosine between the error and "undoing the fine-tuning" was −0.005, and the share erased was −5.8%, if anything a little added. Rounding error is spread evenly across each grid cell wherever a weight sits. For the model as a whole it was right: the hidden-state error pointed toward the base model (cosine 0.29), and the base model really is less decided (median lead 2.21 against 2.47). The base model was never trained on decider's answer format, so reading it this way is our choice of comparison; it still scored 82.0%. Fitting the instruct model fine-tuned from the same base at the same time (the two directions overlap, cosine 0.82), the Q2_K file's base-model coefficient stayed at 0.075 (0.069 with the base direction alone) and the instruct one was −0.007.

One last check: is "toward the base model" just "toward collapse"? The badly broken files answered almost everything with one class. Four of the five stochastic-rounding seeds chose "abbreviation" for 70% to 99.6% of questions, though only 9 of the 500 (1.8%) are abbreviations, and each big-noise seed collapsed onto some single class. So the base-model direction might overlap a "collapse onto one answer" direction. We took the average error of the five stochastic-rounding seeds as a collapse direction and fitted the Q2_K error to both directions at once, with the keep-or-rewrite rule set before computing. The big-noise seeds collapsed onto different classes, which partly cancel when averaged, so the stochastic-rounding average (four of five on the same class) is the stricter test. The base-model coefficient fell from 0.069 to 0.051 but stayed above the 0.04 bar. With the big-noise average it was 0.056; with the five stochastic-rounding seeds entered separately, 0.051; with all ten seeds, 0.050. At Q3_K_M it barely moved (0.080 to 0.076). The finding stands.

One thing to state plainly. The two direction vectors differ in length, so their coefficients can't be compared directly. Measured instead as a share of the error's size, and with the collapse direction taken out, the Q2_K error holds a base-model component (0.22) and a collapse component (0.23) of about the same size. The "about 30%" earlier is the base-model share with that direction alone; 0.22 is what is left of it after removing the overlap with the collapse direction. One fifth to one quarter of the base-model component overlapped the collapse direction; the rest remained. The answer counts above fit this picture: the 2-bit model did not drift toward "abbreviation", where the broken files piled up (14 answers went to 11), but toward the base model's mix, more "description" and less "entity".

The scorecard:

Eleven of twenty-two.

SOTAAZ course

This course alone $10 · 23 lectures · every lecture freeWatch free →

The open question

Here is where the question stands. Weight by weight, the Q2_K error has nothing to do with the fine-tuning (cosine −0.005). Random noise of the same size doesn't lean toward the base model, even when shaped to the same block sizes. Noise only leans that way once it is big enough to break the model. Yet the Q2_K error leans that far while barely hurting accuracy. What about quantization error lets it reach the base model with so little damage?

There are two suspects. One is that rounding is deterministic: the same weight always rounds the same way, and similar values round alike. The other is the grid itself: block-wise scales chosen by search, and values confined to a few grid points.

To separate them we kept the Q2_K scales but sent each weight to the grid point above or below at random, with probability proportional to distance (stochastic rounding). The implementation is right: recomputing nearest rounding with the same scales reproduces the file exactly in all 248 tensors. But stochastic rounding has a larger error by nature, here 1.45 times (close to the theoretical √2), and the model broke (1.8% to 14.0% accuracy). Since noise of that size also leans toward the base model once the model breaks, our pre-set rule says this result can't count as evidence for the grid. This design couldn't separate determinism from the grid.

The next experiment is stochastic rounding at matched error size. On a finer grid such as Q3_K, random rounding gives an error about as large as Q2_K's nearest rounding, and we can see whether it still leans toward the base model.

Our own hypothesis, for the record. Pre-training carved patterns deeply over trillions of tokens; fine-tuning added a thin layer on top (low-rank here, merged from a LoRA). If quantization works like sandpaper, keeping the big contours and smoothing the fine grain, and if the fine-tuning's effect depends on many small weights working together precisely, then quantization could break that coordination without erasing any single weight's change, as round 3's weight-level check found. Noise shaped to the block sizes didn't lean, so imitating the grit isn't enough, and big noise of any kind leans too. The hypothesis is still half open.

If you run quantized decision models

• On this model and task, Q2_K and Q3_K_M lost almost no accuracy but became noticeably less confident. If you route answers by a confidence threshold, fewer answers clear it after quantizing.
• One temperature fitted on a few hundred of your own labeled questions brought it back here (expected calibration error 0.100 → 0.045 at Q2_K). Refit after quantizing; decider's shipped 1.11 was fitted on BF16.
• Keeping the shared embedding table at high precision doesn't fix it. llama.cpp's stock files already do that, and the drop comes from the body.

Limits

• One model (decider-4b v2.1), one task (TREC, six options, 500 questions), one run per file. A second model (for example, a small decision model we fine-tuned ourselves earlier) would be the cheapest way to test whether this generalizes; we haven't done it.
• The "head" is shared with the input embeddings, so we couldn't cut one without the other.
• Rounds 2 and 3 and every control were designed after seeing earlier results; each round's predictions were written before that round. The decompositions and the instruct check are exploratory.
• Q3_K_M leans toward the base model more than Q2_K (0.47 against 0.29) and loses more decisiveness, which doesn't follow the bit count. The two files keep different tensors at higher precision; comparing them could show which layers hold the decisiveness. We haven't done that.

The plan with every round's predictions and timestamps, the recipes for the split and noise files, and every script and result file are in the reproduction package below.