LoRA matches full fine-tuning on the task you measured — and installs singular directions full fine-tuning never produces, the ones that concentrate forgetting. Equal accuracy is the one metric the difference hides behind.
A team has a base model and a task, and a budget that says full fine-tuning is not happening. So they reach for LoRA — train a low-rank adapter, leave the base weights frozen, ship the merged result. Before they commit, they do the comparison that is supposed to settle it: run a small full fine-tune of the same base on the same data, evaluate both on the task, and check that LoRA keeps up. It does — the two checkpoints land within noise of each other on the held-out task set. The conclusion writes itself: LoRA reaches the same accuracy at a fraction of the cost, so LoRA is full fine-tuning’s equal, and the cheaper equal wins. From then on LoRA is the default for everything, because the one experiment that mattered came back tied.
The experiment did come back tied. The conclusion drawn from it does not follow. “Same accuracy on the fine-tuning task” and “same model” are different claims, and the gap between them is a structural difference in what the two procedures did to the weights — LoRA at the rank that ties full fine-tuning has written new high-ranking singular directions that full fine-tuning does not produce, and those are where the forgetting of the base model’s distribution concentrates. The equivalence is real on exactly one axis and false on the rest, and the one axis is the only one the deciding experiment looked at.
This post is about why LoRA and full fine-tuning produce structurally different models even when their downstream task accuracy is identical — what the difference is, why it is the forgetting and continual-learning behavior that pays for it, and why “matched the task metric” is the wrong evidence for “interchangeable.”
The first thing to get straight is that matched task accuracy is not a discovery that argues for equivalence. It is the controlled variable the structural findings are measured against. LoRA vs Full Fine-tuning: An Illusion of Equivalence (arXiv 2410.21228) holds task performance fixed on purpose: all of its RoBERTa-base models fine-tune to equivalent accuracy on the downstream task. That is stated up front precisely so anything else the paper finds cannot be waved away as “one model was just trained better” — pin the two to the same downstream number, and any remaining difference is one the task metric was blind to by construction. The team in the opening scene ran the same experiment, then stopped at the line the paper treats as the starting line. A divergence that persists at equal task accuracy is the one that matters — and the one a single-metric comparison is constitutionally unable to report.
If the task numbers match, the place to look for the difference is the weights themselves — and that is where LoRA and full fine-tuning visibly part.
LoRA vs Full Fine-tuning: An Illusion of Equivalence (arXiv 2410.21228) analyzes the fine-tuned weight matrices through their singular value decomposition — which exposes the directions a matrix amplifies and by how much. The finding is sharp and one-directional: weight matrices trained with LoRA have new, high-ranking singular vectors, which the authors call intruder dimensions, while those trained with full fine-tuning do not. The definition is precise rather than impressionistic — a singular vector of the fine-tuned matrix counts as an intruder when its maximum cosine similarity to any of the top pre-trained singular vectors falls below a threshold (the figures use the top ten pre-trained vectors, and the finding holds across a range of threshold and count). In plain terms, LoRA introduces high-strength directions that point somewhere the base model’s dominant directions never pointed; full fine-tuning, adjusting the existing spectrum, does not. This is the core of “not the same model” — and notice it is not the claim that LoRA learned the task worse. The accuracy is matched, that is the control. It is the claim that two procedures arriving at the same downstream behavior wrote structurally different objects into the parameters.
The reason the structures differ is not mysterious. LoRA Learns Less and Forgets Less (arXiv 2405.09673) measures the rank of the perturbation each method applies and finds full fine-tuning learning perturbations whose rank is 10–100 times higher than typical LoRA configurations — LoRA ranks tested at 16, 64, and 256 against a full-rank dimensionality of 4096. They occupy different rank regimes by construction. A low-rank update can only reach the task by concentrating its effect into a few strong directions, and those are exactly what show up as intruder dimensions: the cheapness of LoRA and its spectral signature are the same fact seen from two sides.
A structural difference in the weights would be a curiosity if it stayed there. It does not, and LoRA vs Full Fine-tuning: An Illusion of Equivalence (arXiv 2410.21228) does not stop at correlating intruder dimensions with forgetting; it intervenes on them. By changing the singular values associated with the intruder dimensions after fine-tuning, the authors show those dimensions cause forgetting rather than merely accompanying it — and the intervention runs the other way too: scaling the intruder dimensions down significantly improves modeling of the pre-training distribution with a minimal drop in downstream task performance. That last clause is the whole argument. There exists a knob that recovers the base distribution while barely touching the task score, which means the standard LoRA checkpoint was spending base-distribution capability the task metric never showed, because spending it costs almost nothing on the task. This is the precise sense in which the equivalence is an illusion that only holds on the metric people check: two checkpoints that agree on one number and disagree on a capability — retention of what the base model knew — that the one number was never measuring.
The “forgets less” half of the picture cuts against the naive reading, and matters for that reason. LoRA Learns Less and Forgets Less (arXiv 2405.09673) finds that LoRA forgets less than full fine-tuning, with the extent of forgetting controlled by rank — and in the matched-accuracy regime this is a strict advantage for LoRA. In code instruction-tuning, LoRA at rank 256 offers comparable HumanEval accuracy while strictly forgetting less of the source domain: a source-domain average around 0.509 for LoRA at rank 64 against roughly 0.414 for full fine-tuning, at code-IFT HumanEval scores of about 0.498 for LoRA rank 256 versus about 0.497 for full fine-tuning. The unifying claim is not “LoRA forgets more” but that the two methods sit on different points of a learning-and-forgetting tradeoff, rank moves a model along it, and the task metric reports nothing about where on that curve you are.
A model is rarely trained once and frozen forever; it gets adapted again — a second task, a new domain, next month’s data. The structural difference between LoRA and full fine-tuning is largely invisible on the first task and is a tax on every task after it.
LoRA vs Full Fine-tuning: An Illusion of Equivalence (arXiv 2410.21228) ties the spectral story directly to sequential learning. Lower-rank LoRA produces more intruder dimensions and degrades more under continual learning; as rank increases past a threshold and the update starts to resemble a high-rank one, the intruder dimensions begin to disappear, and at rank 2048 the model exhibits a continual-learning curve very similar to full fine-tuning. The paper is blunt about the trap in between: while LoRA matches the performance of full fine-tuning initially, smaller ranks of LoRA consistently show greater degradation of performance during continual learning. “Initially” is the operative word — it is the first task, the one the deciding experiment measured. The degradation is in the sequence, after the point anyone checked; a team that validated LoRA on a single task and made it the permanent default validated the one regime where the difference is smallest.
Why subspace geometry, not just rank, governs this is the contribution of Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation (arXiv 2603.02224). It proposes that forgetting under continual LoRA follows a geometric law — a function of the minimum principal angle between task gradient subspaces, fitting synthetic-task data with a correlation of 0.994 — with a consequence that reframes the rank conversation. Adapter rank only matters for forgetting when the task subspaces overlap; at high subspace angles rank has minimal impact (a coefficient of variation around 0.8% in synthetic experiments and 10–19% on real benchmarks), validated on Split-CIFAR100 and sequential GLUE. The dominant lever is where the update lands, not how big it is — which is why two LoRA configurations at equal task accuracy can forget very differently, on a property of the update the task metric never sees.
An independent follow-up converges on the same mechanism from the fix side. OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning (arXiv 2510.13003) attributes LoRA’s forgetting to updates interfering with the dominant singular directions that encode essential pre-trained knowledge — the same spectral story — and proposes a structural rather than accuracy-driven remedy. By decomposing the frozen weights and constraining the LoRA update to lie entirely within the orthogonal complement of the top singular subspace, OPLoRA provably preserves the top singular triples and, the authors report, significantly reduces forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B. That is the keystone restated as engineering: if the lost retention were the price of the task accuracy you could not recover one while holding the other — but you can, so it was a side effect the standard recipe incurred and the metric never billed.
The argument so far is easy to over-read into “LoRA is a trap,” and that reading is wrong. The literature does not say LoRA is worse; it says LoRA is different, and that the difference is invisible to the one metric teams check — a claim about measurement, not a verdict on the method. The equivalence illusion is seductive for a real reason: How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness (arXiv 2512.15634) reports that LoRA achieves competitive and in some cases superior performance compared to full supervised fine-tuning, particularly on reasoning tasks at specific rank values. On the task, at the right rank, LoRA can tie or win at a fraction of the compute and storage — that is the reason it is a sane default, and a post that pretended otherwise would be arguing against a method that is frequently the correct choice.
And the forgetting difference does not always point against LoRA: in the matched-accuracy instruction-tuning regime of LoRA Learns Less and Forgets Less (arXiv 2405.09673), LoRA strictly forgets less, so there the structural difference is its advantage. The same paper is equally clear about where LoRA’s structure costs in-task quality — in continued pretraining at scale it underperforms in-domain, code CPT at 20B tokens giving full fine-tuning a HumanEval of 0.263 against LoRA rank 256’s 0.224 and math CPT at 20B tokens giving full fine-tuning a GSM8K of 0.293 against LoRA rank 256’s 0.202, even while retaining more of the source domain (code CPT source-domain average 0.545 versus 0.617). Which method wins depends on the regime; the honest claim is not that one is better, but that they are not the same model, so a single number cannot rank them.
Two more limits keep the argument from overreaching. First, these results are demonstrated on specific setups — RoBERTa-base, the programming and mathematics domains, synthetic tasks plus Split-CIFAR100 and sequential GLUE. They are consistent across model families and modalities, but the magnitude is setting-dependent and the safe reading is directional, not a universal constant. Second, every structural disadvantage here has a known lever: raising the rank dissolves the intruder dimensions, rank-stabilized high-rank LoRA approaches full fine-tuning’s continual-learning curve, scaling the intruder singular values down recovers the base distribution, and orthogonal-projection methods like OPLoRA remove the interference by construction. The difference is real and also controllable — which means the failure mode is not “LoRA,” it is “LoRA validated on one task and assumed equivalent on the rest.” The fix is not to distrust LoRA; it is to stop reading a task tie as a model match, and to measure the axes the tie was silent on.
You cannot make LoRA and full fine-tuning into the same model by matching their task accuracy. What you can do is measure the axes the task number is blind to, before either one becomes a default.
Measure retention of the base distribution, not just the new task. The intruder-dimension forgetting is invisible on the fine-tuning task and visible on the data the base model used to handle. Evaluate the fine-tuned checkpoint on a held-out slice of the original capability — general benchmarks the base model passed, the source domain you are tuning away from — and read it as its own number. A task tie with a retention drop is not a tie; it is a trade the task metric did not show you.
Evaluate the regime you will actually run, including the second task. The continual-learning degradation in LoRA vs Full Fine-tuning (arXiv 2410.21228) shows up after the first task — “initially” is where LoRA and full fine-tuning agree. If the model will be adapted again, validate it on a sequence, not a single fit; a one-task validation certifies the one regime where the difference is smallest.
Treat rank as a structural knob, not only a cost knob. Rank moves the model along the learning-and-forgetting tradeoff and controls whether intruder dimensions appear at all. Choosing it by “smallest rank that ties the task” optimizes for the metric that hides the difference; if continual learning or base retention matters, a higher rank buys structure the task tie will not tell you that you bought.
Decide whether to fine-tune at all before deciding how. The structural cost of touching the weights is a reason to check that touching them is necessary in the first place. Whether to fine-tune or prompt is the prior question, and the forgetting and continual-learning costs above are a line item in it — prompting and retrieval do not move the spectrum because they do not move the weights.
Run the safety eval on the actual checkpoint, whichever method made it. LoRA and full fine-tuning produce different weights, so a safety property measured on one does not transfer to the other any more than a task number does — and fine-tuning erodes safety even on benign data. The safety gate reads the checkpoint you are shipping, not the method’s reputation.
LoRA matching full fine-tuning on the task you measured is a true result and a good reason to use LoRA. It is not a reason to treat the two checkpoints as the same model, because they are not — one carries high-ranking intruder directions where forgetting concentrates and the other does not. The task tie is silent on base-distribution retention and on how the model degrades when it has to learn next, and silence is not equivalence. Measure the axes the tie hides, and choose the method on what you actually need the model to keep — not on the one number that was always going to come back even.
GRPO's group-relative advantage looks like a clean normalized policy gradient. Two of its normalizers aren't neutral: dividing by response length rewards longer wrong answers, dividing by reward std overweights easy and hard questions. The trainer optimizes both, faithfully.
16 min →
A big pass@1 jump after a GRPO run reads as the model having learned to reason better. Often it has not — it has learned to pick, more reliably, a solution path it already had. RLVR re-weights the base model's existing reasoning rather than creating new reasoning, and whether any RL budget escapes that is genuinely disputed.
14 min →
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →