5 LoRA Variants, Explained by What They Actually Freeze
Almost nobody understands what is actually being updated inside the model.
Here are 5 techniques that change how you think about model adaptation, and what each one is actually doing to the weights:
1./ LoRA - Learn the update, not the weights
The pretrained weight W is frozen. Completely untouched.
Instead of updating W directly, two small matrices are trained =>
A ∈ ℝʳˣᵈ and B ∈ ℝᵈˣʳ, where r ≪ d
The weight update is: ΔW = BA Effective weight: W' = W + BA
The entire adaptation happens in a tiny low-rank space. W never changes.
2./ LoRA-FA - What if we freeze even more?
Same structure as LoRA. One change.
A is frozen alongside W. Only B is trained. Effective weight: W' = W + BA (A is fixed)
Half the trainable matrices of LoRA. Same core idea. Fewer parameters.
3./ VeRA - What if the matrices don't need to be learned at all?
This is where it gets interesting.
A and B are both frozen, and randomly initialized. What gets trained are just two tiny scaling vectors =>
b ∈ ℝʳ and d ∈ ℝʳ
Instead of learning the low-rank matrices themselves, VeRA keeps them frozen and learns small scaling vectors that modulate their contribution.
Initialization => b = 0, d = 1
You're not learning matrices. You're learning how to scale them.
One of the most parameter-efficient techniques on this list.
4./ Delta-LoRA - What if W itself learns from the low-rank updates?
This one is fundamentally different.
Unlike standard LoRA, the base weight W is not fully frozen. It is updated through low-rank delta propagation at every step =>
W^(t+1) = W^t + c(B_(t+1)A_(t+1) − B_t A_t)
Where c is a scaling factor.
A and B are trainable. W evolves, but guided entirely by low-rank changes.
5./ LoRA+ - Same structure. Smarter learning rates.
Identical to LoRA, freeze W, train A and B.
One change => B is assigned a larger learning rate than A. η_B > η_A
A ← A − η_A · ∂J/∂A B ← B − η_B · ∂J/∂B
A small optimization change that can make LoRA training more effective.
The core idea running through all five:
You do not always need full fine-tuning to adapt a model.
LoRA updates two matrices.
LoRA-FA updates one.
LoRA+ updates two at different speeds.
Delta-LoRA lets W evolve - guided by low-rank deltas. VeRA updates two vectors.
Same goal. Five different answers to the same question:
=> What is the minimum we actually need to learn?
That is the core idea behind parameter-efficient fine-tuning.
And now you know what is actually happening inside the model.
This is a dense technical post, and it holds up well — all five techniques are real, correctly named, and their core mechanisms are described accurately. One specific numeric detail (VeRA's initialization value) I couldn't pin down precisely.
LoRA — accurate. The standard formulation (Hu et al., 2021) freezes the pretrained weight W and learns a low-rank decomposition ΔW = BA with rank r ≪ d, exactly as described.
LoRA-FA — accurate, and the name itself confirms it. "FA" literally stands for "Freeze-A" — the method freezes the down-projection matrix A (randomly initialized) and trains only the up-projection matrix B, halving the trainable parameters relative to standard LoRA. Matches the post's description precisely.
VeRA — accurate on the core mechanism, one specific number I couldn't confirm. The original paper and multiple independent sources confirm: VeRA uses a single pair of low-rank matrices, randomly initialized and frozen, shared across all layers, with only small trainable scaling vectors (b and d) learned per layer — this significantly reduces trainable parameters (the paper reports roughly 10x fewer than LoRA on RoBERTa-large) while preserving comparable performance. The "b initialized to 0" detail is consistent with the design goal that a VeRA layer starts as a no-op (identity) before training begins. The post's specific claim that d is initialized to 1 is plausible but not something I could independently confirm — the paper and HF's PEFT implementation both expose a configurable "d_initial" value, and I didn't find a source pinning it to exactly 1 as a universal default. Worth double-checking against the paper before repeating that exact number.
Delta-LoRA — accurate, and this is the most commonly misunderstood one, so getting it right matters. Unlike standard LoRA, Delta-LoRA propagates the change in the low-rank product (BA) between consecutive training steps back into the frozen base weight itself, exactly matching the formula in the post: W is updated using the delta of successive B·A products, scaled by a factor c. This is the correct distinguishing feature of Delta-LoRA versus LoRA.
LoRA+ — accurate. The core proposal (Hayou et al., 2024) is that A and B should use different learning rates, with B's learning rate set substantially higher than A's, which they show improves both convergence speed and final performance over vanilla LoRA's shared learning rate.
Nothing here misrepresents how these five methods actually work — this is a genuinely well-researched technical summary, unusual for how dense and specific it is without slipping into hand-waving.