Fine-tune or prompt? A 2026 decision tree.

“Fine-tune the model on our domain” is the most confident sentence in applied LLM work, and it is confident about the wrong thing. It treats fine-tuning as a single lever that closes a single gap — the gap between a generic model and one that handles your business — when the phrase “our domain” is hiding two unrelated gaps that happen to produce similar-looking complaints. The sentence is not wrong because fine-tuning fails. It is wrong because it does not say which gap it is paying to close, and half the time it is pointed at the gap fine-tuning cannot touch.

Watch how the mistake plays out. A team’s assistant gives answers that are factually fine and wrong in every other way — too long, wrong format, ignoring the house style, burying the number the user wanted three sentences deep. They decide to fine-tune, “so the model learns our domain.” Weeks go into assembling a training set. The fine-tuned model writes in the right register now. It also still gets every fact from the retrieval system, because that is where the facts live and always did. The team paid a training bill to fix a formatting problem — and, in the same run, tried to teach the model facts, which is the half that quietly did not take. They will not find out which half worked until something downstream depends on a fact the model was supposed to have absorbed and did not.

“The model doesn’t know our domain” is two complaints wearing one sentence. One is missing knowledge: the model lacks facts about your product, your customers, your data. The other is missing behavior: the model has access to the facts but does not produce them in the form, tone, or structure you need. They are different problems with different fixes, and the expensive mistake — the one the team above made — is treating them as one purchase. This essay is the decision tree that keeps them apart, and the order of the branches matters: diagnose the gap first, because the diagnosis decides everything after it.

Knowledge and behavior are not the same gap

Before any technique, separate the two, because the rest of the tree is just consequences of this split. Missing knowledge is a fact the model cannot state because the fact was not in its training data and is not in its prompt — your current pricing, this customer’s order history, last quarter’s revised refund policy. The model is not producing the fact wrong; it is not producing it at all, or it is producing a plausible invention in its place. Missing behavior is the opposite shape: a fact the model has, or could have via retrieval, but renders wrong — wrong format, wrong length, wrong register, wrong refusal boundary, the right answer delivered in a way the product cannot use.

The reason this separation is the whole essay is that the two problems have almost opposite solutions, and the solutions do not substitute for each other. Knowledge is fixed by putting the fact in front of the model at inference time — retrieval. Behavior is fixed by changing what the model does with whatever it already has — fine-tuning. Reach for the wrong one and you do not get a weak result; you get a confident result that fails on the axis you were not measuring. Fine-tune to fix missing facts and you get a model that has learned to sound authoritative about facts it still does not reliably know. Prompt your way around a behavior gap and you get a prompt that grows a new clause every time the model drifts, until the prompt itself is the maintenance burden. A worked diagnosis takes one question: is the failing output wrong because a fact is absent, or wrong because a present fact is badly shaped? Answer that, and the technique is no longer a choice.

Missing facts is a retrieval problem, not a training run

The instinct that fine-tuning “teaches the model your facts” is the most expensive wrong belief in this area, and it has been measured rather than merely argued. The study “Fine-Tuning or Retrieval?” (arXiv 2312.05934) compared the two approaches for knowledge injection and found retrieval-augmented generation consistently outperforming unsupervised fine-tuning — and, more pointedly, that LLMs “struggle to learn new factual information through unsupervised fine-tuning,” needing many paraphrases of a single fact before it partly sticks. That detail is the mechanism worth internalizing. Fine-tuning adjusts weights toward a distribution; it does not file a fact in a lookup table. A fact seen a handful of times in a training set is a faint gradient signal competing with everything else the run is teaching, and the model emerges able to state the fact sometimes, in some phrasings — which is indistinguishable, from the outside, from a model that will confidently state the opposite.

Then there is the update problem, which is where the cost compounds. Facts change. Prices change, policies change, a customer’s status changes the day after the training set was frozen. When a fact in a retrieval index changes, the fix is an edit to one document. When a fact baked into weights changes, the fix is another training run — and a training run that, per the study above, may not even take. A retrieval index is a system you update; a fine-tune is a system you re-pour. So the first branch of the tree is unconditional: if the problem is that the model lacks or misstates facts, the answer is retrieval — the chunking-and-eval discipline the chunking essay and the faithfulness essay describe — and fine-tuning is not on the table for that problem at all. Not as a weaker option. Not on the table.

Missing behavior is where fine-tuning earns its cost

The other branch is where fine-tuning stops being wasted spend and starts being the correct instrument. The LIMA work (arXiv 2305.11206) aligned a 65B model to strong instruction-following with only 1,000 carefully curated examples and no reinforcement learning, and framed the result as a “superficial alignment hypothesis”: fine-tuning largely surfaces format, style, and behavior the pretrained model already latently has, rather than installing new knowledge. Read that hypothesis next to the retrieval result and the two halves of the tree snap into focus. Pretraining already gave the model a vast latent repertoire of behaviors; fine-tuning is not adding to that repertoire so much as selecting from it and making one region of it the default. That is exactly the shape of a behavior fix — and it is good news, because it means the behavior branch does not need a giant dataset. A small, clean set of examples in the target form is often enough, since you are not teaching a skill from nothing, you are pointing at a skill the model has and saying do this one.

That is the small end. The large end of the same branch is worth stating so the distinction does not get overdrawn into “fine-tuning never adds capability.” It does, at scale. The Flan work (arXiv 2210.11416) scaled instruction fine-tuning to over 1,800 tasks and lifted 5-shot MMLU by 9.4 points over the base model — a genuine capability gain, not a stylistic one. The reconciliation is that instruction tuning across many tasks teaches a general behavior — how to follow an instruction at all — and that generalizes. What it still does not do is install the specific, volatile facts of your particular business. The line holds in one sentence: fine-tuning reshapes behavior and broad capability; it does not reliably install specific facts. Both ends of the branch obey it.

The two techniques stack — it was never an exclusive choice

Framing the decision as “fine-tune or prompt” is useful for diagnosis and actively misleading as a final architecture, because the two are not rivals competing for one slot — they compose, and the strongest systems use both. The RAG-versus-fine-tuning study on an agriculture domain (arXiv 2401.08406) measured each approach independently and then together: fine-tuning added more than 6 percentage points of accuracy, RAG added about 5 more on top of that, and crucially the gains were cumulative rather than overlapping. They were cumulative because they were fixing different gaps. Fine-tuning moved the behavior; retrieval supplied the facts; neither was doing the other’s job, so neither’s contribution was redundant.

This is why the decision tree is a diagnostic instrument and not a fork in the road where you commit to one path and burn the other. A genuinely hard domain problem — a support assistant that must answer in a precise regulated format using facts that change weekly — is a both problem, and the tree’s actual output for it is “fine-tune the behavior, retrieve the facts,” two techniques aimed at two gaps. The tree’s job is never to hand you a winner. It is to tell you which gap each technique is being paid to close, so that when you spend on fine-tuning you know it is buying behavior and you have not quietly also expected it to buy knowledge it cannot deliver. The moment you can name the gap, you can name the technique — and sometimes the honest answer names both.

Prompting first: when a training run is premature

Even once you have correctly diagnosed a behavior gap, fine-tuning is not the immediate next step — prompting is, and skipping past it is its own form of wasted spend. Prompting here includes in-context learning: putting a few worked examples of the target behavior directly in the prompt. It requires no training run, no dataset assembly, no evaluation harness for a new model artifact, and for a behavior the base model produces readily once it has seen two or three examples, it is simply the correct and complete answer. A surprising fraction of “we need to fine-tune for tone” turns out to be “we need three good examples in the system prompt,” and the team that fine-tuned never checked.

The case for crossing over to fine-tuning, when it comes, is partly about quality and substantially about cost — recurring cost. The work on parameter-efficient fine-tuning versus in-context learning (arXiv 2205.05638) showed PEFT beating in-context learning on both accuracy and computational cost, and the cost mechanism is the part to hold onto: in-context learning re-processes its examples on every single inference call. Every request pays, in tokens and latency, for the demonstration block to be read again. A fine-tune pays the example cost once, at training time, and then serves a shorter prompt forever. So the crossover is a volume question. For a behavior needed occasionally, the long prompt is fine and the training run is premature. For a behavior on the hot path of a high-traffic product, that per-call tax — paid millions of times — is the entire argument for training the behavior into the weights and reclaiming the prompt budget. Prompt to validate that the behavior is even achievable; fine-tune when the meter on the prompt gets expensive.

The LoRA default and the one place it leaks

When you do reach a fine-tune, low-rank adaptation is the default, and for solid reasons: LoRA (arXiv 2106.09685) freezes the base weights and trains a small pair of low-rank matrices, cutting trainable parameters by orders of magnitude, and — because those matrices fold back into the weights — it adds no inference latency at all. For the common case, a behavior fix, LoRA is not a compromise; it is the right tool, cheap to train and free to serve.

But “default” is not “always,” and the one place LoRA leaks is worth knowing before an eval surprises you. “LoRA Learns Less and Forgets Less” (arXiv 2405.09673) found that on genuinely hard skills — code and math — LoRA can underperform full fine-tuning, and the reason is structural rather than incidental: the weight changes those skills require are higher-rank than a typical LoRA configuration can represent, so the adapter simply does not have the degrees of freedom to capture them. The same paper found the flip side, that LoRA forgets less of the base model’s other capabilities, precisely because it perturbs the weights less. That trade has a clean reading. LoRA for a behavior fix, which is what most fine-tuning honestly is; consider full fine-tuning only when the behavior you need is a hard reasoning-shaped skill and a held-out eval actually shows LoRA leaving a gap. Do not reach for full fine-tuning on suspicion — reach for it on evidence, and let the eval, not the folklore, decide.

  "the model doesn't know our domain"
        │
        ├─ missing FACTS? ─────────────► retrieval (RAG). not fine-tuning.
        │
        └─ missing BEHAVIOR / form? ──► does prompting / ICL get it?
                                          ├─ yes, and low volume ──► prompt
                                          └─ no, or high volume ──► fine-tune
                                                                     (LoRA default;
                                                                      full FT if an
                                                                      eval shows a
                                                                      hard-skill gap)

• Diagnosed the gap as missing facts or missing behavior before choosing a technique.
• Sent every facts problem to retrieval, not to a training run.
• Tried prompting / in-context learning before committing to a fine-tune.
• Used the cumulative result — fine-tune behavior, retrieve facts — rather than picking one.
• Defaulted to LoRA; reserved full fine-tuning for an eval-proven hard-skill gap.
• Gated the whole decision on a held-out eval, so “it worked” is a number.

Reading list

• Fine-Tuning or Retrieval? — RAG beats unsupervised fine-tuning for knowledge injection: arXiv 2312.05934
• RAG vs Fine-tuning — the cumulative result: fine-tuning +6 points, RAG +5 more on top: arXiv 2401.08406
• LIMA — 1,000 examples and the superficial alignment hypothesis; fine-tuning surfaces behavior: arXiv 2305.11206
• Scaling Instruction-Finetuned Language Models — instruction tuning at task scale, +9.4 points MMLU: arXiv 2210.11416
• Few-Shot PEFT is Better and Cheaper than In-Context Learning — the recurring cost of ICL: arXiv 2205.05638
• LoRA — low-rank adaptation, orders of magnitude fewer trainable parameters, no inference latency: arXiv 2106.09685
• LoRA Learns Less and Forgets Less — where LoRA trails full fine-tuning on hard skills: arXiv 2405.09673

A model learns your facts from the prompt and your behavior from the gradient. Confuse the two and you have not bought a better model — you have bought an invoice for the half of the problem your training run was never able to touch.