Concepts
RAG vs fine-tuning: which does your company need?
RAG vs fine-tuning explained for business assistants: what each changes, what it costs to keep current, where each fails, and a checklist for choosing.
Updated · 6 min read
What is the difference between RAG and fine-tuning?
Retrieval-augmented generation keeps knowledge outside the model. When a question arrives, the system searches your documents or data, puts the most relevant passages into the prompt, and the model writes its answer from them. The term comes from a 2020 research paper by Lewis and colleagues, which combined a language model’s built-in (parametric) memory with a searchable external (non-parametric) index.
Fine-tuning changes the model itself. You train an existing model further on examples of the inputs and outputs you want, and the adjustment is baked into its weights. After that, the model behaves differently on every request, with no retrieval step involved.
The practical difference follows directly: RAG changes what the model knows for this one question, while fine-tuning changes how the model behaves on all questions.
The deciding question: facts or behaviour?
Reduce the decision to one question: are you trying to teach the model facts or behaviour? Facts change: a price moved yesterday, a policy was revised this morning, an incident opened five minutes ago. Behaviour is comparatively stable: answer tersely, always return valid JSON, classify tickets into your twelve categories.
Fine-tuning is a poor way to store facts. Changing one fact means preparing data and running another training job, the old value can still surface from the weights, and nobody can point to where an answer came from. RAG is a poor way to fix behaviour: retrieving better passages will not stop a model from rambling or breaking your output format.
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | The context the model sees per question | The model’s weights, for every question |
| Best for | Changing facts, large document sets, cited answers | Tone, output format, narrow classification or extraction skills |
| Keeping it current | Update or re-index the source | Prepare new training data and retrain |
| Traceability | Each answer can cite the passages used | No direct link from answer to training example |
| Access control | Can filter results by user and tenant at query time | Anything trained in is available to every user of the model |
| Typical failure | Retrieves the wrong or stale passage | Confident answers from outdated or blended knowledge |
| Model switch | Keep the index, swap the model | Training is tied to the base model and must be redone |
When RAG is the right choice
For most business assistants the knowledge is the hard part, and it changes. RAG fits whenever the answer lives in documents you already maintain and people need to see where it came from.
- Policies, procedures, product information and knowledge articles that are edited regularly.
- Regulated or audited contexts where every answer must be traceable to a source.
- Data with different audiences, where results must be filtered by the user’s permissions.
- Large collections no prompt could hold, such as years of tickets or manuals.
- Situations where you expect to change models later and do not want to redo training.
When fine-tuning earns its place
Fine-tuning is worth the effort when you have tried clear instructions and a few examples in the prompt and the model still does not behave consistently. It needs a good set of labelled examples and a way to evaluate the result, so treat it as an engineering project, not a setting.
- A strict output format that downstream systems parse, where prompt-only compliance is not reliable enough.
- A consistent brand voice or register across thousands of replies.
- Narrow, repetitive tasks such as ticket classification, routing or field extraction, where a smaller tuned model can be faster and cheaper.
- Specialised vocabulary in medicine, law or engineering that a general model keeps misreading.
The third option: live tool calls instead of a copied index
For operational data, both options above can be the wrong answer. Incident status, monitoring metrics, change records and mailbox contents change by the minute, and copying them into a vector index means answering from a snapshot that is already stale. The alternative is to let the assistant call the system of record at question time through a tool, and answer from what it returns.
Live tool calls also keep permissions where they already live: the source system decides what the requesting identity may see, instead of a separate index that has to replicate those rules. Standards such as the Model Context Protocol (MCP) make those tools reusable across assistants and models.
Botify is built around this pattern. It queries connected systems such as Google Workspace, Microsoft 365, Slack, Odoo, ServiceNow, Dynatrace and Splunk, or any MCP server registered for the tenant, at the moment of the question; tool policies apply to every call and each call is recorded in the audit trail. Models are reached through OpenRouter, so changing the model does not mean rebuilding what the assistant knows.
Where RAG goes wrong in practice
RAG is simple to demonstrate and harder to run well. Most failures come from the retrieval step, not from the model, so measure retrieval on its own with real questions before tuning prompts.
Security deserves the same attention. OWASP’s 2025 Top 10 for LLM applications lists vector and embedding weaknesses as a separate risk, including data leaking between users or tenants that share one vector store, and it notes that RAG does not remove prompt injection: a poisoned document in the index can steer the model when it is retrieved.
- Chunks cut mid-thought, so the retrieved passage lacks the sentence that answers the question.
- A stale index that still returns last quarter’s policy after the source was updated.
- No minimum similarity score, so the model is handed irrelevant text and answers anyway.
- An index that copied documents without their permissions, exposing them to every user.
- Retrieved text treated as trusted instructions instead of untrusted data.
The hybrid, and a quick decision checklist
Combining both makes sense in one recurring case: a domain with heavy specialised vocabulary, where fine-tuning helps the model understand the terms and RAG or tool calls supply the current facts. Outside that case, the hybrid mostly adds cost and moving parts.
- Does the answer change more often than you would retrain a model? Use RAG or live tool calls.
- Must users see the source of each answer? Use RAG or live tool calls.
- Is the data already in a system with an API and permissions? Prefer live tool calls.
- Is the problem the shape or tone of the answer, not its content? Try better instructions first, then fine-tuning.
- Do you have hundreds of good labelled examples and a way to evaluate? Only then is fine-tuning realistic.
Frequently asked questions
Which is cheaper, RAG or fine-tuning?
RAG is usually cheaper to start and to keep current, because updating a source costs far less than a training run. Each RAG request is slightly more expensive, since the prompt carries retrieved text. Fine-tuning has the opposite profile: an upfront cost that recurs every time the facts or base model change.
Does RAG eliminate hallucinations?
No, but it reduces them and makes them detectable, because you can compare the answer with the passages it was given. Pair RAG with a minimum similarity score, an explicit rule not to answer without support, and source citations in every reply.
Can fine-tuning teach a model new facts?
It can, but unreliably. The model may blend new and old information, cannot tell you where a fact came from, and needs retraining whenever that fact changes. For facts that matter, keep them in a source the assistant retrieves or queries.
Do I need a vector database for RAG?
Not always. Vector search helps with paraphrased and multilingual questions, but keyword search, a database query or an API call to the system of record are all valid retrieval steps. Many strong setups combine keyword and vector search.
Do large context windows make RAG unnecessary?
Not for most business data. Even a very large context cannot hold every document and ticket you own, sending everything on each request is slow and expensive, and you still need to filter by permissions. Larger context mainly lets you retrieve more generously.