Five LoRA Variants on One Table, and the Setting or Condition Each Abstract Leaves Out

The VeRA paper draws its trainable vector d starting at ones. In the same paper's ablation, starting d at 1.0 was the worst value tested: 70.3 on MRPC against 90.5 when it starts at 0.1. The abstract does not mention d's starting value at all.

We found that while filling in one table of five LoRA variants, one row each, with columns for the three pieces of a layer: the pretrained weight W and the two small matrices A and B. Filling the cells was easy. Checking each row against its paper was not, because each of the four variants after LoRA has a setting or condition that only shows up past the abstract. VeRA has the starting value of d. Delta-LoRA's update to W starts only after a step K you choose. LoRA+'s abstract asks for "a well-chosen fixed ratio" of learning rates, and the ratios the authors found best run from about 2 to about 16 depending on the model and the initialization. LoRA-FA's headline memory figure assumes adapters on every linear layer of every block.

This post is the map. For each variant it says what is trained, where the paper says the saving comes from, and which setting you would have to choose. It does not measure any of them. The later posts in this series build each one in code and then measure.

If you have not used LoRA before, read LoRA Part 1 first. This post assumes you know the idea in one line.

The layer all five start from

A linear layer computes h = W x. LoRA (Hu et al., arXiv:2106.09685) freezes W and adds a low-rank correction, following the paper's equation (3) and its scaling:

h = W x + (α/r) · B A x

A is r × d_in and starts random, B is d_out × r and starts at zero, so the correction is zero at the first step. Only A and B are trained. With r much smaller than d, that is far fewer numbers than W has.

The LoRA equation h = W x + (alpha/r) · B A x drawn by hand, with each term in a colored box. W is green and labeled pretrained weight, never changes. alpha/r is blue, the scale you pick. B (up, from rank r to the output) and A (down, from the input to rank r) are orange and trained; B starts at 0 and A starts random. The line underneath reads: only A and B learn, 4,194,304 numbers at rank 8 for q and v adapters in 32 layers.

One naming warning before the table. The Delta-LoRA paper writes the update as ΔW = AB, with A of shape c × r and B of shape r × d, and starts B at zero. That is the reverse of the LoRA paper, where B is the up-projection and starts at zero. We use the LoRA paper's names everywhere in this series, so "B" always means the matrix that maps from rank r back up to the output.

A table with five rows (LoRA, LoRA-FA, VeRA, Delta-LoRA, LoRA+) and columns W, A and B. LoRA: W frozen, A and B trained, 4,194,304 trainable numbers. LoRA-FA: W and A frozen, B trained, 2,097,152. VeRA: W, A and B frozen, with two small vectors d and b trained (drawn as orange tabs), 278,528. Delta-LoRA: W updated through the change in B times A, A and B trained, 4,194,304. LoRA+: W frozen, A and B trained, 4,194,304. Counts are calculated for d = 4096 with q and v adapters in 32 layers, rank 8, and rank 256 for VeRA.

The counts in the last column are calculated from each paper's formula for a Llama-2-7B-shaped model: d = 4096, adapters on the query and value projections of 32 layers, which makes 64 adapted matrices. LoRA at rank 8 trains 2 × 64 × 4096 × 8 = 4,194,304 numbers. LoRA-FA trains only B, so half of that, 2,097,152. VeRA's paper gives L_tuned × (d_model + r), which at rank 256 is 64 × (4096 + 256) = 278,528. These are calculations, not counts from a running model. We count the real trainable parameters in the post that builds them.

Here is the same layer, drawn once per variant. Watch which blocks change color.

An animation of one linear layer drawn five times. The input x feeds the pretrained weight W and the path A then B, and the two outputs are added to give h. The five frames mark which blocks are trained, frozen or updated for LoRA, LoRA-FA, VeRA, Delta-LoRA and LoRA+, with the update formula and the calculated number of trainable parameters for each.

LoRA-FA: freeze A, and the saving is in activations

LoRA-FA (Zhang et al., arXiv:2308.03303) freezes A at its random start and trains only B. The parameter count halves, but that is not what the authors lead with. Their abstract says LoRA-FA "significantly reduces activation memory consumption."

The paper's Section 5 gives the reason. To compute the gradient for a trained A, the layer has to keep its input x, which has d numbers per token. With A frozen, B's gradient only needs A x, which has r numbers per token. Appendix C.3 then works this out for Llama-2-7B as 18 · b · s · d bytes saved per layer, where b is batch size, s sequence length and d hidden size.

An animation that adds one token at a time to two columns. Under LoRA, where A trains, each token adds a long bar for x (4,096 numbers) plus a thin sliver for A·x (8 numbers), so six tokens keep 24,624 numbers. Under LoRA-FA, where A is frozen, each token adds only the 8-number sliver, so six tokens keep 48. The slivers are drawn wider than scale. A note says the paper's 18·b·s·d bytes per layer adds this up over seven adapted linear layers, counting each input separately.

That 18 assumes adapters on all seven linear layers in each block: query, key, value and output in attention, and the three MLP projections. Their inputs add up to 9d numbers per token, at 2 bytes each. That count is the paper's simplification: it counts each layer's input separately, although query, key and value read the same x, which PyTorch can keep once when LoRA dropout is 0, and its MLP shape (3d wide) is not Llama-2-7B's actual 11,008. A measured saving can therefore come out smaller. This is also a different setup from the q-and-v count in the table above. For batch 8, sequence 1,024, d = 4096 and 32 layers, 18 · b · s · d · L comes to 19,327,352,832 bytes: 19.3 GB in decimal units, 18.0 GiB in binary ones. The paper's "at least 18GB" matches the binary figure (our arithmetic from the paper's inputs).

The LoRA-FA saving written as 18 · b · s · d · L, with each factor in a colored box. 18 is a plain box for the paper's premise: 9d inputs at 2 bytes each, with all 7 linear layers adapted. b (batch, 8) and s (sequence length, 1,024) are blue, your settings. d (hidden size, 4,096) and L (layers, 32) are green, fixed by the model. The result line reads 19,327,352,832 bytes: 19.3 GB, or 18.0 GiB. A note says this is a simplification, because q, k and v read the same x, so a measured saving can come out smaller.

So the condition the abstract leaves out is that the saving scales with batch size, sequence length and the number of adapted layers. With short sequences or adapters only on q and v, it is much smaller. If you are fighting for long sequences on one card, this is the variant aimed at you.

There is also a setting inside the method. The latest version of the paper does more than freeze A: its abstract adds "closed-form gradient corrections" for B, which the first version's abstract did not have. The LoRA-FA optimizer in PEFT 0.21.2 applies that correction, scaling B's gradient by (r/α)² and the pseudo-inverse of A Aᵀ (with a small value added to the diagonal) before the Adam step (we read this in its source). Plain "freeze A" and PEFT's LoRA-FA are therefore not the same training run. We have not checked the paper's quality results yet.

VeRA: share random A and B, train two vectors

VeRA (Kopiczko et al., arXiv:2310.11454) goes the other way on the parameter count. A and B are random, frozen, and shared by every layer. What is trained is two vectors per layer, written as diagonal matrices: h = W x + Λb · B · Λd · A x. The vector d has length r and scales the rank dimension; b has length d_out and scales the output.

The VeRA equation h = W x + Λb · B · Λd · A x drawn by hand. W, B and A are green and frozen; B and A are shared random matrices. Λb (vector b, starts at 0) and Λd (vector d, starts at one value) are orange and trained. The line underneath reads: on MRPC, d starting at 0.1 scores 90.5, at 1.0 only 70.3.

The paper's count is L_tuned × (d_model + r) for VeRA against 2 × L_tuned × d_model × r for LoRA, which at rank 1 is about half and gets much smaller as the rank grows. There is a second saving that the count does not show: the shared random matrices can be regenerated from a random seed, so what you store per adapted model is the vectors plus one seed.

The setting is initialization, the one in the opening. The paper starts b at zero, so the layer begins unchanged, and starts every entry of d at one nonzero value that it calls a new hyperparameter. Its figure illustrates that value as ones, but its GLUE runs use 0.1, and its ablation on MRPC and RTE scores 1.0 far lower than 0.1 or 10^-7. The random A and B use Kaiming uniform initialization in those runs. Rank is also a different kind of choice: it grows only the vector d, so the paper uses ranks of 256 and 1,024 on RoBERTa, and the count above uses 256.

Delta-LoRA: let W move without a gradient for W

Delta-LoRA (Zi et al., arXiv:2309.02411) trains A and B by the optimizer as usual, and also changes W. After each optimizer step it adds the change in the product, scaled: W ← W + λ · (α/r) · (BA_new − BA_old). W never gets its own gradient or optimizer state, so the abstract can say the memory stays comparable to LoRA's.

Three settings come with it, and the abstract names none of them. First, the W update starts only after step K. Second, λ scales it. The paper uses λ = 2, K = 500 for its E2E data-to-text experiments and λ = 0.5, K = 1000 for summarization; for GLUE it uses λ = 0.5 with K tied to each task's warmup. Third, the paper removes dropout from the low-rank branch, because with dropout the gradient of W and the gradient of AB are no longer equal, and the method relies on them being equal.

An animation of optimizer steps as columns: 1, 2, 3, K, K+1, K+2, K+3. The A, B row gets a new cell at every step. The W row stays green and marked frozen through step K, then gets a pink cell at every step after it, labeled W += λ·(α/r)·(BA_new − BA_old), meaning W changes without its own gradient. A dashed line marks step K. Below: the paper uses K = 500, λ = 2 for E2E and K = 1000, λ = 0.5 for summarization, and removes dropout from the A, B branch.

Two consequences are our reading, not claims from the paper. Because W changes during training, the A and B files alone no longer reproduce the result; you also have to save W. And because W is rewritten at every step after K, the method does not drop into a QLoRA-style setup, where the base model is stored frozen in 4 bits. The paper discusses neither point. We check the first when we build the variant.

LoRA+: same parameters, different learning rates

LoRA+ (Hayou et al., arXiv:2402.12354) changes nothing about the model. It gives B a learning rate λ times larger than A's, with λ fixed and well above 1. Their argument is that with the same learning rate for both, the adapters in wide layers do not learn features efficiently. The abstract reports 1-2% better results and up to about 2x faster fine-tuning "at the same computational cost as LoRA."

The LoRA+ rule η_B = λ · η_A drawn by hand. η_B (B's learning rate) and η_A (A's learning rate) are orange boxes; λ, the ratio, is a blue box for your setting, with a note that the best values in the paper ran from 2 to 16. The line underneath reads: the abstract asks for a well-chosen ratio and gives no number.

The abstract says the ratio must be well chosen, and leaves out what it is. In the paper it varies. For RoBERTa it found about 2^4 (16) best with one initialization and 2^2 to 2^3 with another, and for Llama experiments 2^1 to 2^2. "Set B's learning rate higher" is therefore not one recipe. It is a hyperparameter you need to scan, and in the benchmark post we scan it.

Which one to try first

This table is a reading of what each paper says it is for, not a measurement.

Your constraint	Variant the paper is aimed at	What to check first
The model fits, you want faster or better LoRA with no extra memory	LoRA+	Scan λ; the paper's best values spanned 2 to 16
Long sequences or big batches run out of memory	LoRA-FA	Savings scale with batch, sequence length and adapted layers; PEFT adds a gradient correction
You store many adapters (per user or per task)	VeRA	Rank, and the starting value of `d` (0.1 in the paper's GLUE runs)
LoRA's quality gap to full fine-tuning matters	Delta-LoRA	`K`, `λ`, no dropout, saving W, no 4-bit base

A decision fork that starts from the question what is in your way and branches four ways. The model fits and you want faster or better training with no extra memory: LoRA+, check first by scanning λ, whose best values in the paper ran from 2 to 16. Long sequences or big batches run out of memory: LoRA-FA, check that the saving grows with batch, sequence length and adapted layers, and that PEFT adds a gradient correction. You store many adapters per user or per task: VeRA, check the rank and where d starts (0.1 in the GLUE runs). The quality gap to full fine-tuning matters: Delta-LoRA, check K, λ, no dropout, saving W and no 4-bit base. The things to check are blue boxes, meaning your settings.

What we did not check

Quality and speed claims are the authors'. We have not trained any of these yet. Where a number comes from a paper, the sentence says which one.
Parameter counts are calculated from the formulas for one model shape, not read from a running model.
The activation saving for LoRA-FA is the paper's calculation for Llama-2-7B with adapters on all seven linear layers, which we reproduced arithmetically. Whether PyTorch and PEFT realize it in practice is something the later measurement has to show.
The version of each paper we read is the one on arXiv as of 2026-10-21. LoRA-FA's latest version differs from its first in title and abstract, and only the latest abstract mentions the gradient correction; the first describes freezing A alone.

The next post builds LoRA from scratch in about 30 lines of PyTorch and checks that its output matches the PEFT library's. After that we implement the four variants as small changes to that code, and finally measure all five under the same conditions, with predictions written down before the runs. The newsletter form below sends each part when it is out.

Five LoRA Variants on One Table, and the Setting or Condition Each Abstract Leaves Out