I had gotten good at prompting. I could coax our LLM into generating structured JSON, writing decent marketing copy, and summarizing meeting notes like a pro. But I kept running into the same wall. The model did not know our internal jargon. Its knowledge cut off before our latest product launched. And no matter how carefully I wrote the system prompt, it could not replicate the specific, slightly irreverent tone our team uses in customer communications.
I tried longer system prompts. I tried few-shot examples with our actual support tickets. I tried chaining prompts with explicit step-by-step instructions. Each fix improved things marginally, but the ceiling was real. The model was helpful and generic. We needed helpful and ours.
It was time to move from being a model user to a model customizer. The question was: which customization approach? “Fine-tuning” used to be the default answer, but it was costly and complex. The landscape has changed dramatically. Today there is a whole spectrum of techniques — from lightweight adapters to retrieval systems — and picking the right one saves you weeks of wasted effort.
This is my practical breakdown of the options, based on working through this decision myself.
Related: If you need prompting fundamentals first, see my Prompt Engineering Guide. For the RAG deep dive, see I Built a Q&A Bot Over Our Company Docs.
The Spectrum of Customization
Customizing an LLM is not a single choice. It is a spectrum from heavyweight to lightweight, and each approach solves a different kind of problem. Understanding where each one fits saved me from going straight to the most expensive option.
1. Full Fine-Tuning: When You Need Deep Domain Fluency
Full fine-tuning means taking a pre-trained model and continuing its training on your own dataset, updating all the weights. I considered this first because it felt like the “real” answer — the thorough approach.
What it is good for. Deeply embedding a specific domain language, style, or knowledge into the model’s core. If you need the model to think in the language of molecular biology, legal contracts, or 18th-century poetry, this is how you do it. The model does not just follow instructions about the domain — it internalizes the domain.
The catch. It is expensive. You need a large, high-quality dataset — thousands of examples at minimum, often tens of thousands for meaningful results. You need serious GPU power for training (think A100s, not your laptop). And you end up with a completely new full-size model for every task, which is costly to host, version, and manage. If you have three different use cases, you now have three multi-gigabyte models to deploy and maintain.
My take. Powerful but overkill for most use cases today. I ruled it out for our situation because we did not need a new base model. We needed the existing model to behave differently in a specific context. Unless you are building for a highly specialized domain with unique terminology that does not exist in the model’s training data, full fine-tuning is probably not where you should start.
2. LoRA: The Option That Changed My Mind
What if you could get the benefits of fine-tuning without updating every parameter in a multi-billion parameter model? That is exactly what Parameter-Efficient Fine-Tuning (PEFT) methods do, and the most popular one by far is LoRA (Low-Rank Adaptation).
How it works. Imagine the model is a giant machine with millions of knobs. Instead of re-calibrating all of them, LoRA adds a small new control panel with just a few extra knobs. It learns a new task by only turning these new knobs, leaving the original machine intact. The new settings are stored in a tiny adapter file — often just a few megabytes compared to the multi-gigabyte base model.
The practical implication is huge: you can have one base model and dozens of tiny LoRA adapters for different tasks. One adapter for summarizing legal documents, another for writing Python code comments, another for acting as your company’s support agent. Swap adapters at inference time. No separate model per use case.
What it is good for. Teaching the model a new skill, tone, or style without the massive cost of a full fine-tune. This is where LoRA shines — behavioral changes. Making the model sound like your brand, follow a specific format consistently, or adopt a particular writing style that few-shot examples cannot fully capture.
The catch. While great for style and behavior, LoRA is less effective for teaching the model new factual knowledge. The core knowledge from the pre-trained model is still dominant. If the model does not know about your product, a LoRA adapter will not teach it. It will just change how it talks about things it already knows (or confidently hallucinates about things it does not).
My take. For most teams, LoRA is the new default for fine-tuning. It gives you roughly 80% of the benefit for 1% of the cost and complexity. This is where I started when I needed to change how the model talked. A few hundred well-curated examples of our team’s writing style, a LoRA training run on a single GPU, and the model started sounding like us.
3. RAG: When Knowledge Is the Problem, Not Style
Instead of cramming knowledge into the model’s memory, what if you just gave it access to a library and taught it to look things up? That is Retrieval-Augmented Generation.
How it works.
- Indexing: Take your documents — wiki pages, product docs, support tickets, policy manuals — and break them into chunks. Use an embedding model to turn them into vectors and store them in a vector database.
- Retrieval: When a user asks a question, search the vector database for the most relevant chunks.
- Augmentation: Stuff those retrieved chunks into the prompt, telling the model “Using the following information, answer this question.”
The model is no longer relying on what it memorized during training. It is reading fresh, specific context at query time and generating an answer grounded in your actual documents.
What it is good for. Giving the model access to up-to-date, proprietary, or rapidly changing information. It is the best way to reduce hallucinations and ensure answers are grounded in verifiable sources. If your documentation changes weekly or your product ships new features monthly, RAG handles this naturally — you update the index, not the model.
The catch. Quality depends entirely on retrieval quality. If the search step pulls irrelevant documents, the answer will be poor regardless of how capable the model is. Garbage in, garbage out. RAG also adds architectural complexity: the vector database, the embedding pipeline, the chunking strategy, and the retrieval tuning. It is not hard, but it is not free either.
My take. RAG is the go-to for knowledge problems. If you are building a Q&A bot over documentation or need answers about recent events, RAG is almost always better than fine-tuning for that purpose. For the detailed build-out, see How to Improve RAG Quality.
The Decision Framework I Actually Used
Here is the mental model that helped me choose:
| Goal | Best Tool | Why |
|---|---|---|
| Answer questions about private documents | RAG | Most direct way to inject factual knowledge |
| Adopt a very specific personality or style | LoRA | Changes behavior without full fine-tune cost |
| Follow a complex, multi-step process | Prompting | A detailed step-by-step prompt is often better than any tuning |
| Build for a highly specialized domain (e.g., medicine) | Full Fine-Tune | When the entire domain language needs to be learned deeply |
Often the best solution is a hybrid. The pattern I ended up using was RAG for knowledge and LoRA for style. We used a LoRA adapter to make the model sound like our team’s friendly support agent, and a RAG pipeline to give it access to our latest product manuals. That combination covered the gap that prompting alone could not.
A Quick Word on Inference Tricks
Once your model is customized, you still have to run it efficiently. Two techniques worth keeping in your back pocket:
- Quantization. Reduces the precision of the model’s weights (e.g., from 16-bit to 4-bit numbers). Makes the model significantly smaller and faster with minimal accuracy loss. Near-essential for running models on consumer hardware or keeping inference costs manageable in production.
- Caching. If you get the same questions repeatedly, cache the answers. For more complex caching, you can cache the key-value pairs from the model’s attention layers, which speeds up generation of long sequences. Simple Redis-based answer caching gave us a 40% reduction in API costs for our most common queries.
What Didn’t Work / Honest Limitations
None of these approaches is a magic fix. Full fine-tuning requires resources most teams do not have. LoRA changes behavior but not knowledge — if the model does not know something, a style adapter will not help. RAG depends on retrieval quality, and bad retrieval means bad answers regardless of how good the model is.
The hybrid approach I described works well, but it also means maintaining two systems: the adapter and the retrieval pipeline. That is more moving parts, more things to monitor, and more things that can break. Each piece needs its own evaluation — you test the adapter’s style with human review, and the RAG pipeline’s accuracy with retrieval metrics. Combining them means debugging two failure modes, not one.
Start with the simplest approach that solves your actual problem. For most teams, that means great prompting first, RAG when you need knowledge, and LoRA when you need style. Only escalate to a full fine-tune when you have a truly unique domain and the resources to back it up.
Final Thoughts: Name the Gap, Then Pick the Tool
The thing that surprised me most about this process was how far prompting and RAG got us before we even touched fine-tuning. I expected to need a custom model much sooner than I did. Good prompt engineering covered 70% of what we needed. RAG covered another 20%. LoRA handled the last 10% — the tone and style that no prompt could fully capture.
If you are hitting the ceiling of what prompting can do, the decision tree is simpler than it looks: figure out whether your gap is knowledge or behavior, then pick the tool that matches. And do not underestimate the hybrid approach. RAG for what the model needs to know, LoRA for how it needs to sound, and prompting for everything else got us further than any single technique would have.
The worst mistake is jumping straight to fine-tuning before you have exhausted the cheaper options. The second worst is using RAG when your problem is actually style, or LoRA when your problem is actually knowledge. Name the gap first. The tool follows.
Previously: Prompt Injection: Attacks, Examples, and Defenses. Next: I Built a Q&A Bot Over Our Company Docs: A Practical Guide to RAG.
Leave a Reply