LoRA: freeze the weights, train a low-rank correction

Low-Rank Adaptation, introduced 2021 by Microsoft researchers, is the dominant parameter-efficient fine-tuning technique: instead of updating every weight of a pretrained model, freeze the pretrained weights and train small injected matrices that stand in for the weight update. (Summarised from the source at the bottom; edited, not verified.)

The mechanism and the hypothesis underneath

The premise is that a fine-tuning update ΔW doesn't need the full rank of W — the article calls this the hypothesis that weight updates have low "intrinsic rank", i.e. the change can be represented with far fewer parameters than the matrix it modifies. LoRA exploits that by decomposing the update into low-rank factors, trains those, and leaves the base untouched. Note the epistemic status: the low-intrinsic-rank claim is the article's hypothesis, the thing LoRA is "grounded in" — not a measured property the article demonstrates.

The numbers, and where they don't line up

Applied to GPT-3, the article reports trainable parameters cut ~10,000× — from 175 billion to roughly 18 million — while GPU memory during training drops only 3× (1.2 TB to 350 GB). Those two figures are both the article's, and the gap between them is the honest footnote: the headline number counts trainable parameters, not memory, and nowhere near it. After training, adapter weights can be merged into the base model, after which there is no inference latency overhead at all.

What the adapters buy operationally

• One base model, many small adapters (the article's example: customer-service, legal, and medical adapters of ~18M params each on a 175B base) — each under 1% the storage of a full fine-tuned copy.
• Adapter swapping: load a different adapter onto a resident base without reloading the model, to switch specialisations cheaply.
• Combination: merge several adapters' updates with each other or the base to blend capabilities.
• QLoRA extends this to fine-tune a 30B-parameter model on a single 24 GB consumer GPU.
• LoRA is widely used in post-training, including with DPO.
• Against the intuition "smaller update means worse result": the article reports RepairLLaMA, a LoRA-tuned code-repair model, outperforming full-parameter fine-tuning and GPT-4 on its task.

Where the article stops (my inference, labelled)

The article gives no failure modes — no tasks where LoRA measurably loses to full fine-tuning, no account of which layers or ranks need how much capacity. Read the 10,000× as the original paper's framing, not a measured equivalence of outcomes. And "no additional inference latency" after merging implies what it says and also what it doesn't: the full base model still loads and runs at full width — LoRA shrinks the training of a specialisation, not the cost of serving it. On a memory-bound box, see Quantizing an MoE on one unified-memory GPU for what that resident-model bill actually looks like.

Source: Wikipedia, "LoRA (machine learning)", read 2026-09-08. Cost figures for GPT-series training that the article relays are unverified third-party estimates. Edited, not verified.