How Big a Model Can You Fine-Tune on an 8, 12, 16 or 24 GB GPU? We Ran Every Answer Under the Card's Limit

How Big a Model Can You Fine-Tune on an 8, 12, 16 or 24 GB GPU? We Ran Every Answer Under the Card's Limit#

Part 4 ended with a question: on a card with 8, 12, 16 or 24 GB, how large a model can you fine-tune with LoRA, LoRA-FA and VeRA? Part 4's gap of up to 1.5% suggests the variants might give slightly different answers. We predicted before running that they would not, and they did not: the same answer in all 16 cells. The largest gap between a variant and LoRA in the same setting was 0.023 GiB of allocated memory and 0.043 GiB of reserved memory. What did change the answer was the base model's precision: loading the frozen model in 4 bits moved every card up one to two rungs on the Qwen3 ladder.

The answer, card by card#

The table shows, for each GPU size (columns) and training setup (rows), the largest Qwen3 model whose training ran to the end. For example, on an 8 GB GPU with the model loaded in bf16 and 1,024-token sequences, training works up to 1.7B. We ran three training steps each and counted a run as fitting if it did not stop with an out-of-memory error.

bf16 loads the model at its original precision; 4-bit (nf4) loads it compressed to 4 bits, as QLoRA does. Either way the model itself stays frozen and only the small adapter attached to it is trained.

With a 4-bit base, the 16 GB and 24 GB cells both say 14B because 14B was the largest model we prepared. On the 24 GB card, 4-bit 14B at 2,048 tokens reserved 14.72 GiB, leaving 8.8 GiB under the limit. A larger model may fit there; we did not run one. The bf16 rows have no 14B because its weights alone are 27.5 GiB, more than even the 24 GB card holds.

Most of the memory-limited runs used LoRA only. We ran LoRA-FA and VeRA under the limit in the three closest cases: 4-bit 14B on 12 GB and on 16 GB, and bf16 1.7B at 2,048 tokens on 8 GB. For everything else, we treat the three variants as giving the same answer because their memory differed by at most 0.043 GiB, far less than the room those runs had left. That reasoning covers the runs that finished. Of the runs that hit the limit with LoRA only, the closest was bf16 4B at 2,048 tokens on 12 GB; there, without a limit, LoRA-FA and VeRA allocated at most 0.011 GiB less than LoRA, and we did not run them under the limit.

An 8 GB GPU does not give you 8 GB#

A GPU sold as 8 GB has less than that available for training, for two reasons.

First, the capacity the GPU itself reports is already a little below the spec. Out-of-memory messages posted online show 8 GB GPUs reporting 7.75 to 8.00 GiB, 12 GB GPUs 11.77 GiB and 24 GB GPUs 23.65 to 23.99 GiB. Our A100 80GB reports 79.25 GiB.

Second, when training starts, the CUDA driver and libraries such as cuBLAS take memory before PyTorch loads the model, and PyTorch's memory numbers do not include it. On our A100 (CUDA 12.6) that was 0.51 to 0.65 GiB. We did not measure it on consumer GPUs and assume it is similar.

We emulated the four GPUs on one A100 by letting PyTorch use only the spec number minus 0.5: 7.5, 11.5, 15.5 and 23.5 GiB. Subtracting both of the above shows that these limits are slightly generous. An 8 GB GPU that reports 7.75 GiB leaves 7.75 − 0.65 = 7.1 GiB. The free Colab T4, a common 16 GB GPU, reports 14.75 GiB and leaves about 14.1.

So we checked whether that gap changes any answer. Mostly it does not: apart from 8 GB at 2,048 tokens and 16 GB, every run stayed at least about 0.85 GiB under its limit. On 8, 12 and 24 GB, the real card's limit is at most 0.5 GiB lower than ours, so that room is enough. A T4's limit is 1.4 GiB lower than our 16 GB limit, and 8 GB at 2,048 tokens had only about 0.2 GiB to spare. We reran those two at the lower limits. These reruns were added the day after we saw the results, but we committed predictions before each one.

• 8 GB, 2,048 tokens (limit 7.1 GiB): the 4-bit 4B still finished. bf16 1.7B ran out of memory with all three variants, so the table says 0.6B. 1.7B did finish at 7.5 GiB, so an 8 GB GPU that leaves that much may run it, but even the roomiest 8 GB GPU in those logs leaves about 8.00 − 0.51 = 7.49 GiB, just under 7.5. We did not test limits between 7.1 and 7.5.
• 16 GB (limit 14.1 GiB): 4-bit 14B finished at both lengths with all three variants. At 2,048 tokens, though, PyTorch reserved 14.03 to 14.07 GiB, within 0.07 GiB of the limit. bf16 4B still had 1.89 GiB or more to spare at 14.1, so we did not rerun it.

What we ran#

• Models: Qwen3 0.6B, 1.7B, 4B, 8B and 14B, all with the same 151,936-token vocabulary.
• Adapters: the code from Parts 3 and 4, unchanged, on q_proj and v_proj. LoRA and LoRA-FA at rank 8, alpha 16; VeRA at rank 256.
• Training settings a small card needs: batch 1, gradient checkpointing on, AdamW, SDPA attention, three steps on random tokens with the loss on every token, at 1,024 and 2,048 tokens.
• Base precision: bf16, or 4-bit nf4 through bitsandbytes with double quantization and bf16 compute, the way QLoRA loads a model.
• Software and card: PyTorch 2.7.1, Transformers 4.57.6, bitsandbytes 0.50.2, on one A100 80GB. The GPU was idle when each run started; in 9 runs another job started on it partway through, and for those we discarded only the memory-outside-PyTorch reading (the table in the predictions section). The later rechecks at tighter limits ran while another job was already on the GPU; the limit is enforced inside PyTorch, so it does not affect whether a run fits, and we did not use their outside-PyTorch readings.

One run looks like the code below. We changed the model, variant, length and precision, one run per fresh process.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from lora_layer import add_lora                      # Part 3's LoRA layer

GiB = 2**30
MODEL, BASE, SEQ = "Qwen/Qwen3-1.7B", "bf16", 2048   # BASE is "bf16" or "nf4"

quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                           bnb_4bit_use_double_quant=True,
                           bnb_4bit_compute_dtype=torch.bfloat16) if BASE == "nf4" else None
model = AutoModelForCausalLM.from_pretrained(
    MODEL, dtype=torch.bfloat16, attn_implementation="sdpa",
    quantization_config=quant, device_map={"": 0})
# with nf4, add_lora needs a fix before and after this line (see below)
add_lora(model, 8, 16, ("q_proj", "v_proj"))         # rank 8, alpha 16

model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
model.config.use_cache = False
model.train()                                        # checkpointing only runs in training mode
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-5)
torch.cuda.reset_peak_memory_stats()                 # count training only, not loading

ids = torch.randint(0, model.config.vocab_size, (1, SEQ), device="cuda")   # batch 1, random tokens
for _ in range(3):
    model(input_ids=ids, labels=ids).loss.backward()                        # loss on every token
    opt.step(); opt.zero_grad(set_to_none=True)
print(f"allocated {torch.cuda.max_memory_allocated() / GiB:.2f} GiB, "
      f"reserved {torch.cuda.max_memory_reserved() / GiB:.2f} GiB")

A 4-bit base needed one fix. Part 3's layer creates A and B in the dtype of the weight it wraps, which is uint8 for a 4-bit layer, and a uint8 tensor cannot have a gradient, so add_lora itself fails and the adapters cannot be created. While add_lora runs, we turned any uint8 tensor request into bf16, then initialized A and B again; the full code is part5_fit.py in the reproduction package. PEFT handles this for you.

We emulated each card with torch.cuda.set_per_process_memory_fraction: if PyTorch tries to hold more than 7.5, 11.5, 15.5 or 23.5 GiB, it gets an out-of-memory error, as it would on the real card.

We measured twice. The first pass ran all 48 combinations once with no limit. One unlimited peak can be held against all four cards, so it quickly narrows down which model sits at each card's edge. But an unlimited peak is not what a run uses under a limit: with memory to spare, PyTorch keeps freed space instead of returning it, and under a limit it appears to give that space back and try again. So the second pass took, for each card and setting, the largest model the first pass said would fit and the next one up, and ran both under the limit. Whether a model fits a card comes down to whether it finishes under that limit, so where the passes disagreed, the second one decides. One cell did disagree; "Why an unlimited peak cannot decide a card" below shows it.

Why the variant does not matter here#

In Part 4, LoRA-FA saved memory because it does not keep each layer's input for the backward pass, and VeRA cost more because it keeps a wider intermediate result. Those were differences in what each layer stores while the forward pass runs.

Gradient checkpointing removes most of that storage. Each layer keeps only its input; the rest is recomputed during the backward pass, one layer at a time. So the adapter's extra storage exists for one layer at a time instead of for all of them. Across all 48 first-pass runs, LoRA-FA and VeRA stayed within 0.023 GiB of LoRA's peak allocated memory, and within 0.043 GiB of its peak reserved memory.

The trained parameters are not where the memory goes either. LoRA on Qwen3-14B trains 5.2 million numbers. With their gradients and two Adam states, that is well under 0.1 GiB.

Where the memory goes: the weights, plus about 2 GiB per 1,024 tokens#

Take each run's peak and subtract the frozen weights. What is left is almost the same on every model: 1.82 to 2.19 GiB at 1,024 tokens and 3.62 to 4.34 GiB at 2,048. It roughly doubles with length. From 0.6B to 14B it grows by 0.37 GiB at 1,024 tokens and 0.72 GiB at 2,048, far less than the weights grow, and it is the same with either base.

Splitting it after the runs, most of it is the output layer. For every token the model scores all 151,936 entries in the vocabulary, and the loss step holds several copies of those scores. At 1,024 tokens that is about 1.7 GiB on every model. The layer inputs that gradient checkpointing keeps are the part that grows with model size: by our formula, 0.06 GiB on 0.6B and 0.44 GiB on 14B at the same length.

That is why small cards are tight. On an 8 GB card with a bf16 base, Qwen3-1.7B's weights are 3.21 GiB, and at 2,048 tokens the rest is 3.74 GiB, more than the weights. Going from 1,024 to 2,048 tokens on this model adds 1.86 GiB, more than half of its weights.

The 4-bit base shrinks only the weights. Qwen3-8B goes from 15.27 GiB to 5.66, and 14B fits in 9.28. Not to a quarter, because the embedding, the output layer (lm_head) and the norms stay in bf16; on 14B the embedding and lm_head alone are 2 × 151,936 × 5,120 parameters, about 2.9 GiB. The other 2 GiB per 1,024 tokens stays the same. So 4-bit helps most where the weights take most of the card: on 16 GB it moves the answer from 4B to 14B, on 12 GB at 2,048 tokens from 1.7B to 8B, and on 8 GB from 1.7B to 4B at 1,024 tokens (0.6B to 4B at 2,048). On 24 GB it moves one rung, to 14B, the largest model we tried.

So, for a Qwen3-sized vocabulary at batch 1 with checkpointing on, a rough estimate is: weights, plus 2 GiB per 1,024 tokens, plus 0.65 GiB for the CUDA driver and libraries, plus headroom (next section), compared with what CUDA reports for your card rather than the number on the spec sheet. A model with a smaller vocabulary will need less of the 2 GiB; we did not measure one.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

Why an unlimited peak cannot decide a card#

PyTorch reports two peaks. Allocated is what the tensors used. Reserved is what the allocator held from the GPU, including freed blocks it kept for reuse. Neither one, read from an unlimited run, gave the right answer for every card.

• Reserved without a limit was too pessimistic. Qwen3-1.7B at 2,048 tokens reserved 7.86 GiB with no limit, which says it does not fit in 7.5. Under the limit it reserved 7.28 and finished. It looks like PyTorch, on hitting the limit, gave back the freed space it was holding and tried again; we did not trace it.
• Allocated was too optimistic. Qwen3-14B with a 4-bit base at 1,024 tokens allocated 11.47 GiB, under the 11.5 GiB limit, and ran out of memory under it. The allocator needs room beyond the bytes in use, because freed blocks do not always fit the next request. On the 16 GB run of the same model at 2,048 tokens, it reserved 14.76 GiB for 13.62 allocated, 1.14 more.

So every answer in the table comes from a run under the card's limit.

Check it on your own card#

On your own card, run the code above as it is. If the three steps finish, it fits. No limit needed.

Only if you emulate a smaller card on a bigger GPU, as we did, add these lines before loading the model:

CAP = 7.1                                            # an 8 GB card reporting 7.75 GiB, minus 0.65
total = torch.cuda.get_device_properties(0).total_memory
torch.cuda.set_per_process_memory_fraction(CAP * GiB / total)
torch.cuda.mem_get_info = lambda device=None: (
    int(CAP * GiB) - torch.cuda.memory_reserved(), int((CAP + 0.65) * GiB))

With set_per_process_memory_fraction alone, the model can fail while loading. Before loading, Transformers reserves memory in a size it computes from the weights, capped at the free memory torch.cuda.mem_get_info reports minus 1.2 GiB, and the limit does not change that report. On the A100 it saw about 78 GiB free and tried to reserve 13 GiB under an 11.5 GiB limit. Four of our first 30 limited runs (4-bit 14B on the 12 GB card) stopped this way. After the fix, all four ran out of memory during training instead, so no answer changed. The last line makes mem_get_info report free memory against the limit. A real 12 GB card reports its own free memory, so it does not have this problem.

Our predictions, scored#

We committed these before the first run, with the calculator that produced the numbers.

We read P1 and P3 as having one cause, though this is our reading after the results, not something we predicted. Our calculation came out 2 to 12% above every measured peak, and further above at the longer length on the same model, so we had overcounted something that grows with length. We had put the output layer at 14 bytes per vocabulary entry per token, a figure measured on Qwen2.5 in Part 3. Splitting the measured peaks afterwards gives about 12 on Qwen3. P9 is a different problem: the reserved peak measured without a limit was pessimistic, as the section above shows.

At our 7.5 GiB limit, that gap changed one answer: bf16 at 2,048 tokens on the 8 GB card. Our calculation said 1.7B needed 7.53 GiB, over the limit, so we wrote down 0.6B, and under the 7.5 GiB limit 1.7B finished anyway. At 7.1 GiB, the limit closer to a real 8 GB card, it did not, so the predicted 0.6B is the answer there. When a calculation lands that close to a card's limit, it cannot decide the cell; you have to run it.

What we did not check#

• Emulated cards, not real ones. We allowed for the reported capacity and the driver's share, but a real card may also drive your desktop and monitor, and other programs can hold memory on it. We did not measure that. If your monitor is plugged into the card, leave more room than the table does, especially in the 8 GB cells at 2,048 tokens and the 16 GB 4-bit 14B cell at 2,048 tokens.
• Three steps on random tokens. Memory depends on shapes, not text, so real data at the same length should peak the same. We did not run real data.
• Adapters only on q_proj and v_proj. Adapters on every linear layer would add more; we did not measure how much.
• Batch 1 only. Batch 2 at 1,024 tokens should cost about what batch 1 at 2,048 does. We did not check.
• One model family. Every model here shares Qwen3's 151,936-token vocabulary, which sets most of the per-token cost. A model with a 32,000-token vocabulary would have a much smaller one.
• Added after the predictions: the 12-bytes-per-entry breakdown and the explanation for the reserved gap are our reading of the results, not tested predictions. The 7.1 and 14.1 GiB rechecks were added the day after we saw the results; their own predictions were committed before they ran.

Next#

Part 6 finishes the series. The newsletter form below sends it when it is out.