Four LoRA Variants in a Few Lines Each: VeRA Trained 93% Fewer Numbers and Needed Slightly More Memory

Four LoRA Variants in a Few Lines Each: VeRA Trained 93% Fewer Numbers and Needed Slightly More Memory

VeRA trains 36,864 numbers on Qwen2.5-0.5B. LoRA, on the same two projections, trains 540,672. So VeRA should be the lighter one to train. It was not: at batch 4 and 2,048 tokens in bf16, VeRA's peak was 0.227 GiB higher than LoRA's. We had predicted that before running it, from counting what each layer keeps for the backward pass, and the count came within 1% of the measured difference in three of four cases and missed the fourth by 15%.

This post is a tutorial. Part 2 put five variants of LoRA on one table, and Part 3 built LoRA itself in 19 lines and checked it against the PEFT library. Here you change that layer into the other four, a few lines each:

• LoRA-FA freezes A.
• LoRA+ gives B a larger learning rate than A.
• VeRA shares one pair of random, frozen matrices across all layers and trains two vectors per layer.
• Delta-LoRA also moves the frozen weight W, by the change in B A at each step.

For each one we say first why it should or should not save GPU memory, then show the code, then check it. Three of the four are in PEFT, so we compare against PEFT the same way as in Part 3. Delta-LoRA is not, so we check it against its own update rule.

What you need

The same setup as Part 3: PyTorch 2.7.0, Transformers 5.20.2 and PEFT 0.21.2, Qwen/Qwen2.5-0.5B-Instruct, adapters on q_proj and v_proj in all 24 layers. The checks run in fp32 with TF32 off and eager attention, on the same four-sentence batch, for 30 AdamW steps with weight decay 0. The memory runs use bf16, SDPA attention and batch 4, with the loop from Part 3's step 5. The memory runs at 2,048 tokens peak at about 23 GiB.

You also need Part 3's LoRALinear and add_lora. Everything below builds on them.

The animation shows what the optimizer touches in each variant.

Why some variants save memory and others cannot

Training memory has two parts. The fixed part is the weights plus, for each trained number, a gradient and two Adam states. The part that grows with tokens is what each layer keeps from the forward pass so it can compute gradients in the backward pass.

The two Adam states are what the AdamW optimizer carries for every trained number: a running average of its gradient and a running average of its squared gradient. AdamW uses the pair to choose each number's step size, larger when the gradient keeps pointing the same way and smaller when it jumps around. So every trained number brings three more values with it, the gradient included.

LoRA's trained numbers are already 0.11% of the model, so their gradient and Adam states are small. A variant that trains even fewer numbers saves almost nothing in the fixed part. The only place left to save is what each layer keeps per token.

So for each variant, the question is what it keeps. LoRA keeps the layer's input x (896 numbers per token), because A's gradient needs it, and A x (8 numbers) for B's gradient. q_proj and v_proj receive the same x, so x is kept once per layer. That is 896 + 8 + 8 = 912 numbers per token per layer.

LoRA-FA: freeze A

Why it should save memory: if A gets no gradient, nothing needs x. Each layer keeps only A x for B, 16 numbers instead of 912.

The code is one loop on top of add_lora:

def add_lora_fa(model, r=8, alpha=16, targets=("q_proj", "v_proj")):
    add_lora(model, r, alpha, targets)
    for m in model.modules():
        if isinstance(m, LoRALinear):
            m.A.requires_grad_(False)   # A stays at its random start
    return model

A matrix product keeps its input only when the other side needs a gradient, so freezing A is enough; the forward pass does not change.

Trained numbers drop from 540,672 to 196,608, the B matrices alone. Our count, PEFT's count and the formula 24 × 8 × (896 + 128) agree.

Against PEFT, the training does not match, and it should not. PEFT's create_lorafa_optimizer does more than freeze A. It corrects B's gradient for the frozen A and uses its own AdamW. Ours freezes A and uses the ordinary torch.optim.AdamW. We predicted that the two would differ by more than 1e-3 at some step. The largest difference was 0.037, and the final losses were 0.3669 (ours) and 0.3686 (PEFT). Our A was bit-for-bit unchanged after 30 steps.

PEFT's optimizer does three things ours does not.

• It corrects B's gradient. With A frozen, the change the adapter adds to W can only take the form B A. PEFT adjusts B's gradient so that this change is the closest one to the gradient W would have received in full fine-tuning. In symbols, it multiplies B's gradient by (A Aᵀ)⁻¹ and divides by (α/r)². A Aᵀ is 8 by 8, so recomputing it for every layer at every step is cheap.
• It computes that correction in bf16. If the GPU supports bf16, the correction runs under bf16 autocast and the result is cast back to fp32. Our comparison is in fp32, but this one step is not.
• It has its own AdamW. Its default eps is 1e-6 rather than the 1e-8 of torch.optim.AdamW, and it adds eps at a different point. The same gradient gives a slightly different step.

The 0.037 gap is these three together; we did not measure them one at a time. If you want the method in the current version of the LoRA-FA paper, the one PEFT's optimizer implements, use PEFT's optimizer. The memory saving comes from freezing A, which both versions do. By our count the correction adds only a few 8 by 8 matrices per layer, but we did not measure PEFT's memory.

LoRA+: a larger learning rate for B

Why it cannot save memory: it trains the same numbers as LoRA and keeps the same tensors. Only the optimizer changes.

The layer is untouched. You build the optimizer with two parameter groups:

def loraplus_optimizer(model, lr, ratio):
    layers = [m for m in model.modules() if isinstance(m, LoRALinear)]
    return torch.optim.AdamW([{"params": [m.A for m in layers], "lr": lr},
                              {"params": [m.B for m in layers], "lr": lr * ratio}], weight_decay=0.0)

We used A at 2.5e-4 with ratio 4, so B trains at 1e-3, the rate we gave LoRA in Part 3, Step 3: check it against PEFT.

PEFT's create_loraplus_optimizer with the same rate and ratio builds four parameter groups:

• A: the A matrices, plus any trained parameter that fits none of the groups below, at lr.
• Embedding: LoRA on embedding layers, at a rate of its own that defaults to 1e-6.
• B: the B matrices and any trained vectors (1-D parameters), at lr × ratio.
• B without weight decay: whatever would go into B that is a bias or a LayerNorm parameter, at lr × ratio.

We put no LoRA on embeddings and trained no bias or LayerNorm, so the embedding group and the last group were empty. The two filled groups matched ours: 48 A matrices at 2.5e-4 and 48 B matrices at 1e-3. From the same starting A and B, the two trained to the same loss at every one of 30 steps, a difference of 0.0.

One trap we found after the run: PEFT's LoRA+ helper ignores a weight_decay= argument. We passed weight_decay=0.01 and every group still had 0.0; the helper always gives the bias and LayerNorm group 0 and sets the weight decay of every other group from one argument of its own, loraplus_weight_decay, which defaults to 0. With loraplus_weight_decay=0.01, the A and B groups got 0.01. Our runs used weight decay 0, so this does not change the match above, but if you set weight decay through the usual name, check the optimizer's groups.

VeRA: shared random matrices, two trained vectors

Why it should cost memory, not save it: VeRA computes h = W x + b ⊙ (B (d ⊙ (A x))). A and B are random and frozen, so x is not kept. But to give d its gradient, the layer keeps A x, and at VeRA's rank of 256 that is 256 numbers. To give b its gradient, it keeps B (d ⊙ A x), one number per output. For q_proj that is 256 + 896, for v_proj 256 + 128: 1,536 numbers per token per layer against LoRA's 912.

A and B are made once, at the largest shapes any adapted layer needs, and every layer uses a slice. They come from a seed, so a saved VeRA adapter needs only the seed and the vectors.

import torch.nn.functional as F

class VeRALinear(nn.Module):
    def __init__(self, base: nn.Linear, A, B, d_init=0.1):
        super().__init__()
        self.base, self.A, self.B = base, A, B        # plain tensors: shared, never trained
        for p in self.base.parameters():
            p.requires_grad_(False)
        w = base.weight
        self.d = nn.Parameter(torch.full((A.shape[0],), d_init, device=w.device, dtype=w.dtype))
        self.b = nn.Parameter(torch.zeros(base.out_features, device=w.device, dtype=w.dtype))

    def forward(self, x):
        A = self.A[:, : self.base.in_features]
        B = self.B[: self.base.out_features, :]
        return self.base(x) + self.b * F.linear(self.d * F.linear(x, A), B)


def add_vera(model, r=256, targets=("q_proj", "v_proj"), seed=0, d_init=0.1):
    for p in model.parameters():
        p.requires_grad_(False)
    found = [(module, t, getattr(module, t)) for module in list(model.modules()) for t in targets
             if isinstance(getattr(module, t, None), nn.Linear)]
    w = found[0][2].weight
    g = torch.Generator().manual_seed(seed)          # the seed is all you need to rebuild A and B
    A = torch.empty(r, max(c.in_features for _, _, c in found))
    B = torch.empty(max(c.out_features for _, _, c in found), r)
    nn.init.kaiming_uniform_(A, generator=g); nn.init.kaiming_uniform_(B, generator=g)
    A, B = A.to(w.device, w.dtype), B.to(w.device, w.dtype)
    for module, t, child in found:
        setattr(module, t, VeRALinear(child, A, B, d_init))
    return model

b starts at zero, like LoRA's B, so the model starts as the original. d starts at 0.1; Part 2 showed why that starting value matters in the paper.

Trained numbers: 24 layers × ((256 + 896) + (256 + 128)) = 36,864, 93% fewer than LoRA. PEFT's VeraConfig(r=256, projection_prng_key=0) counts the same.

Against PEFT, it matched exactly. We had predicted that the same seed would give the same A and B as PEFT, and it did, with a largest difference of 0.0: PEFT builds them the same way, from a CPU generator seeded with the key, A first. With b filled with random values so the check tests something, the logits differed by 0.0, against logits as large as 26.2. Over 30 training steps at learning rate 4e-3, every loss matched, from 4.670 down to 2.722.

The loss falls much more slowly than LoRA's, which reached 0.009 on the same batch in 30 steps. VeRA ran at a different learning rate and trains far fewer numbers, and this batch is four sentences learned by heart, so this says nothing about which one fine-tunes better.

Delta-LoRA: W moves too

Why it should not change memory: A and B train exactly as in LoRA, and W still gets no gradient and no Adam state. After each step, W is moved by the change in B A that step produced: W += λ · (α/r) · (B A after − B A before). That needs B A for a moment, about 44 MB across all layers in bf16, after the backward pass has already freed the activations.

Delta-LoRA is not in PEFT. We wrap the optimizer's step instead:

class DeltaLoRA:
    def __init__(self, model, optimizer, K, lam):
        self.layers = [m for m in model.modules() if isinstance(m, LoRALinear)]
        self.opt, self.K, self.lam, self.t = optimizer, K, lam, 0

    @torch.no_grad()
    def step(self):
        self.t += 1
        if self.t <= self.K:              # the first K steps are plain LoRA
            self.opt.step()
            return
        before = [m.B @ m.A for m in self.layers]
        self.opt.step()
        for m, ba in zip(self.layers, before):
            m.base.weight += self.lam * m.scale * (m.B @ m.A - ba)

Call delta.step() where you would call opt.step(). The Delta-LoRA paper names the two matrices the other way round; we keep LoRA's names, with B starting at zero.

We used K = 10 and λ = 2, the λ from the paper's data-to-text setting, and started from the same A and B as a plain LoRA run.

The check is the update rule itself. If every step after K adds the change in B A, the changes add up: W after 30 steps should be the original W plus λ · (α/r) · (B A at step 30 − B A at step 10). It was, with a largest error of 2.8e-7. W had moved by up to 0.019.

The losses were identical to plain LoRA for the first 10 steps, as they must be, then fell faster: 0.0068 against 0.0089 at step 30.

What this changes is what you save. With LoRA you save A and B and put them on the original model. With Delta-LoRA that no longer gives you the trained model. We measured the trained model once more after training, then put its A and B on the original W: the loss went from 0.0062 to 0.0541, the logits moved by up to 14.7, and the top-1 token changed at 3.66% of positions. To keep the trained model you save W for every adapted projection, 22,020,096 numbers against 540,672 for A and B, about 41 times as many.

The memory, measured

All five in bf16 at batch 4, peak memory at the third step, with the loop from Part 3.

For Delta-LoRA we set K = 0 so W moves from the first step. LoRA+ and Delta-LoRA matched LoRA's peak exactly at both lengths, so the 44 MB B A did not reach the peak.

The two that differ, differ by close to the amount the count predicts. Each difference in the peak was exactly the difference in memory held at the end of the forward pass, where the kept tensors sit.

The calculation is the difference in numbers kept, times 4 sequences, times the length, times 2 bytes, times 24 layers. For LoRA-FA it was within 1% of the measurement at both lengths. For VeRA it was within 1% at 2,048 tokens, but at 1,024 the measurement was 15% above it; we have not found where the extra 0.017 GiB comes from. (Added after the run: the remaining 1% is the Adam state of the numbers one variant trains and the other does not. For LoRA-FA, adding A's Adam state, 1,376,256 bytes in bf16, makes the calculation match the measurement to the byte at both lengths; for VeRA at 2,048 it leaves 53,248 bytes. It does not explain VeRA at 1,024.)

Freezing A saved 1.4% of LoRA's peak at 1,024 tokens and 1.5% at 2,048. That is small because, in this model, LoRA's adapters are a small part of what the GPU holds at the peak; Part 3 showed most of it is the 24 layers' other activations and the logits. LoRA-FA's saving grows with the number of distinct inputs you adapt, and with how wide each input is. Adding k_proj would save nothing more, since it shares x with q and v; o_proj, the MLP's input (shared by gate_proj and up_proj) and down_proj each have an input of their own. We only adapted two projections of a 0.5B model.

Our predictions, scored

We committed these before running anything, together with the code above.

The ranges in Q8 and Q10 were half to one and a half times our calculation, wide enough that the calculation could be off by a fair amount and still hit. The closer test is the comparison above: within 1% in three of the four cells, and 15% off in the fourth.

What we did not check

• One small model, two projections, one batch. Everything is Qwen2.5-0.5B with adapters on q_proj and v_proj. The training checks use the four memorized sentences from Part 3, so they show that two implementations compute the same thing, not which variant fine-tunes better.
• Learning rates are ours. VeRA at 4e-3, LoRA+ at 2.5e-4 with ratio 4 and Delta-LoRA's K = 10 were chosen for these checks, not tuned.
• We did not test our LoRA-FA against the paper's method, only against PEFT's, and the two are meant to differ.
• Memory is PyTorch's peak allocated memory in pure bf16 at two lengths, not what nvidia-smi shows.
• Delta-LoRA was checked in fp32 only. With bf16 weights, a small update to W can be lost to rounding, as Part 3 showed for merging.
• VeRA's A and B are plain attributes, so .to() does not move them. Put the model on its device and dtype before calling add_vera.
• Added after the predictions: the LoRA+ weight decay check, the Adam state account of the remaining 1%, and the version log, written after the runs.

Next

Part 5 takes these variants off the 0.5B model and onto the question that matters for a real run: on a card with 8, 12, 16 or 24 GB, how large a model can you fine-tune with LoRA, LoRA-FA and VeRA? The newsletter form below sends it when it is out.