Most teams asking about fine tuning an llm are asking the wrong question. The question isn’t whether you can fine-tune. You can, on a single GPU, for four figures, on a 7B model. The real question is whether the model’s weights are the right place to encode the behavior you need, or whether retrieval, routing, or a sharper prompt would do the same job for less money and less risk. This guide walks through how to tell the difference, as of June 2026.
What fine-tuning actually changes
Fine-tuning starts with a pretrained model and continues training on a smaller, targeted dataset so the model better matches a task, style, or domain. Unlike retrieval, which feeds information into the prompt at inference time, fine-tuning rewrites parameters. That distinction shows up everywhere in the IBM comparison of RAG and fine-tuning, and it’s the single most useful lens for thinking about when to reach for it.
Behavior gets compressed into weights. Knowledge does not, at least not reliably.
What that means in practice: fine-tuning works well for tone, output format, refusal patterns, structured extraction, classification, and domain-specific reasoning steps. It works badly for “the model should know our Q2 numbers” because those numbers change and the weights don’t. A fine-tuned model has memorized a way of behaving, not a current set of facts.
I’d put it this way. Fine-tuning is behavior compression. If the behavior is stable and repeated, compressing it pays off. If it depends on information that changes weekly, you’re compressing the wrong thing.
When does fine-tuning beat prompting or RAG?
Fine-tuning wins on stable, narrow, high-volume tasks where latency matters. Beyond about 100,000 daily queries on the same kind of task, PE Collective’s 2026 cost analysis puts fine-tuning at 10 to 50 times cheaper per query than RAG, with retrieval itself adding 50 to 300 milliseconds of latency per call.
That’s the economic case. The behavioral case is separate and often stronger. Prompts drift. A 12-shot prompt that holds in February breaks in May when someone adds a new instruction at the top, or when the base model gets quietly updated. Weight updates are sticky. Once the behavior is in the parameters, it stays there.
Where fine-tuning earns its keep:
- Ticket classification and routing where the label set is fixed
- Entity extraction in a specialized domain like clinical notes or legal contracts
- Structured generation that must follow a strict schema every time
- Style or compliance voice that prompting can’t lock in
- Latency-sensitive surfaces where a retrieval round trip is disqualifying
- Small models (3B to 7B) that need to punch above their weight on a narrow task
The flip side is just as useful. Don’t fine-tune for knowledge that changes, answers that need citations, multi-tenant document search, or sprawling corpora where most queries touch a small slice. Those are retrieval problems. Bake them into the weights and you’re signing up for a retraining cycle every time the source material shifts.
Fine-tuning changes behavior. RAG changes information access. Most production systems eventually need both.
The fine-tuning process, step by step
The pipeline isn’t complicated. The hard parts are data quality and evaluation, and most teams underestimate both.
- Define the target behavior. Write down the exact outputs you want, what should stay unchanged, and what counts as success, before you collect a single example.
- Collect and clean data. Representative examples, deduplicated, with the noisy and harmful stuff removed.
- Format examples as instruction-input-output pairs, conversation turns, or whatever structure your training framework expects.
- Pick a base model. Instruction-tuned or base, open-weight or hosted, sized for your latency budget.
- Pick a method. Full fine-tuning, LoRA, QLoRA, or one of the newer gradient-free approaches.
- Train with proper validation. Holdout sets, out-of-domain checks, and regression tests on tasks you don’t want to break.
- Evaluate on target and non-target tasks both.
- Deploy with monitoring, versioning, and a rollback path.
Curious what AI could do for your business?
No jargon and no hard sell. Just a friendly look at where AI fits, and where it doesn't.
Databricks’ practical guide to fine-tuning walks the parameter-efficient methods in detail. LoRA and QLoRA dominate production because they cut compute by roughly an order of magnitude relative to full fine-tuning.
The part teams get wrong is data prep. Sample-level quality matters, but token-level quality matters more than most people realize. Token cleaning work presented at ICML 2025 showed that uninformative or redundant tokens inside otherwise clean samples drag down downstream performance, and that filtering tokens by their influence on model updates improves results. A separate line of work at NeurIPS 2025, Wu and colleagues on low-perplexity token learning, found that high-perplexity tokens are a major driver of catastrophic forgetting, and that masking them preserves general capability.
If you’ve ever fine-tuned a model and watched it get sharper on the target task while quietly losing its ability to summarize anything, you’ve felt this effect. It’s not your imagination. The tokens you train on matter as much as the examples you choose.
SFT, LoRA, GRPO, ES: which method fits which problem
The set of methods widened considerably in 2025 and 2026. Here’s the practical version.
| Method | Best fit | Main tradeoff |
|---|---|---|
| Supervised fine-tuning | Predictable output imitation | Needs labeled examples |
| LoRA / QLoRA | Budget-conscious adaptation, multiple task adapters | May still forget; can underfit hard tasks |
| GRPO (RL) | Verifiable reasoning with enough data | Reward design, rollout compute |
| Evolution Strategies | Low-data, instruction-tuned base, parallel compute | Newer, less standardized |
The headline news is Evolution Strategies. Qiu and colleagues’ 2026 paper on ES at scale showed that gradient-free optimization now scales to full-parameter fine-tuning of billion-parameter models, without backprop and without storing optimizer states the way RL methods do. ES outperforms GRPO in low-data regimes, especially under 1,000 examples or below 10% of available data, and it works best on instruction-tuned bases. GRPO still wins with larger datasets and base models.
What could a custom AI agent take off your plate?
We build production-grade AI systems that quietly handle the busywork, so your team can focus on the work that actually matters.
What changed: gradient-free optimization used to be considered too inefficient for modern LLMs. That assumption is no longer safe. If you’re working with a few hundred high-quality examples and an instruction-tuned model, ES is now a real option.
For most teams the default path is still LoRA on an open-weight 7B model with a clean dataset. Start there. Reach for RL or ES only when the task has a verifiable reward and imitation isn’t enough.
Catastrophic forgetting and how to keep it in check
The single biggest risk in production fine-tuning is the model getting better at the new task while losing capability on tasks it used to handle. This is real, it’s measurable, and it can sneak past you if you only evaluate on the target task.
A few mitigations worth knowing:
- Selective token masking. Mask high-perplexity tokens during training. Wu and colleagues at NeurIPS 2025 reported this preserves general capability substantially better than training on raw ground-truth data.
- Training on model-generated data. Generated sequences contain fewer high-perplexity tokens, which means less forgetting. Counterintuitive, but supported in the NeurIPS results.
- Sparse Memory Finetuning. A method described in the 2025 OpenReview paper on sparse memory finetuning updates only a sparse subset of memory rows while leaving the pretrained path intact. In the reported comparison, old-knowledge degradation dropped from 89% under full fine-tuning to 11% under SMF. Not a marginal improvement.
- Held-out general capability tests. Keep a small benchmark suite of tasks the model used to be able to do. Run it after every training run. If it regresses, the new adapter doesn’t ship.
LoRA and other parameter-efficient methods reduce forgetting but don’t eliminate it. They change the shape of the problem rather than removing it. Build the evaluation around that fact and you’ll catch regressions before users do.
How fine-tuning fits with RAG, routing, and long context
Long context windows tempted a lot of teams in 2024 and 2025 to skip retrieval entirely. The temptation has cooled. Liu and colleagues’ TACL paper on long-context models documented the “lost in the middle” effect: models reliably attend to the start and end of a long prompt and miss content buried in the middle. Even in models explicitly trained for long context, this position bias persists. For corpora over roughly 10 megabytes of text, retrieval still beats stuffing.
Routing is the layer most teams skip and shouldn’t. Dynamic routing decides whether a given query should hit a small fine-tuned model, a RAG-backed model, a frontier model, or a long-context path. Done well, it cuts cost without sacrificing quality on the queries that need horsepower. Fine-tuning is a specialization tool. Routing is a resource allocation tool. They work together.
The mental model I’d offer: think of fine-tuning as one of several backend specializations, and routing as the control layer that decides which backend answers each query. In mature systems they’re complementary, not competitive.
How to decide for your team
Walk through these questions in order before you commit to a training run.
- Is the behavior stable, or will the target shift in three months?
- Do you have at least a few hundred clean examples, or a credible plan to generate them?
- Does latency or per-query cost make a retrieval round trip painful at your volume?
- Have you tried a serious prompting and retrieval baseline first?
- Do you have an evaluation harness that measures both target task and prior capability?
- Can you tolerate the operational overhead of versioning, monitoring, and rollback?
If you can answer yes to most of these, fine-tuning will probably earn its keep. If two or three are weak, fix those first. The cheapest fine-tuning project is the one you didn’t have to run because routing and retrieval already solved the problem.
