Every week someone decides the company needs RAG. Sometimes it really does. Most of the time, what the person saw was an excited post about vector databases and concluded that the path to “putting AI in the business” necessarily runs through building a retrieval pipeline. In comes a weeks-long project to solve a problem a ten-line prompt would have solved — or, worse, to solve a problem that never existed.
RAG is an excellent technique and, for a specific band of cases, it’s exactly the right answer. The goal of this text isn’t to sell it or to debunk the hype — it’s to give you an honest decision tree. When is a good prompt enough? When does dumping everything into long context solve it? When does RAG win? And when is it pure overkill, adding cost and parts that break without delivering anything? Let’s go through the four options in the order you should consider them: from the simplest to the most expensive.
What RAG is (definition)
RAG (Retrieval-Augmented Generation) is the pattern of pairing a language model with search over your sources: the system retrieves the relevant passages and only then generates the answer. Instead of training the model on the company’s knowledge, you make it consult that knowledge on the spot.
The keyword is consult. RAG doesn’t put the knowledge inside the model; it puts the knowledge within the model’s reach, at the moment of the question. That changes everything about cost, updating and traceability — and it’s why it solves some problems very well and others not at all.
The right order to consider the options
Before deciding on RAG, you have four tools on the table, and the golden rule is: step up in complexity only when the previous one fails. Start with the cheapest.
- Prompt only — the knowledge fits in what you write as an instruction
- Long context — you attach a few whole documents to the call
- RAG — you search and inject only the relevant passages from a large base
- Fine-tuning — you retrain the model for a fixed format or behavior
Each step adds build and maintenance cost. Jumping straight to step 3 or 4 because “it’s the most advanced” is the definition of over-engineering. The table below sums up the criteria:
| Approach | When it wins | Sign it’s the wrong one |
|---|---|---|
| Prompt only | Small, stable knowledge that fits in the instruction | You paste giant blocks of text into every call |
| Long context | Few fixed documents, low call volume | Per-call cost explodes with volume |
| RAG | Large base, changes often, needs to cite the source | No one maintains the base; search brings back junk |
| Fine-tuning | Rigid format, very high volume, stable behavior | The knowledge changes every week |
Step 1: when a good prompt is enough
If the knowledge the model needs fits in a well-written instruction — your product’s rules, tone of voice, a handful of examples, a few links — you don’t need anything more. A carefully built prompt with the right examples solves a surprising number of cases people think demand infrastructure.
The sign you’re forcing this step: if on every call you paste huge blocks of text that are always the same, stop. Either that becomes long context with caching, or it becomes RAG. But before stepping up, confirm the simple prompt really can’t handle it. It often can.
Step 2: when long context solves it
Modern models accept large contexts. If you have a fixed, limited set of documents — say, a few dozen pages of internal policy that hardly ever change — attaching everything straight to the call is usually simpler, faster to build and easier to maintain than setting up a search pipeline. You have no retrieval to evaluate, no index to rebuild, no vector database to operate.
The catch is economic and shows up with volume. You pay for the input tokens on every call, and billing is per million input and output tokens — Anthropic’s official pricing table and OpenAI’s show the order of magnitude. If you send the same 50,000 tokens of context on each of 100,000 calls a month, you’re paying for those 50,000 tokens 100,000 times. That’s where long context, great for a prototype and low volume, gets expensive in production — and it’s exactly the hole RAG fills, sending only the relevant piece. Context caching helps when the block repeats, but it doesn’t change the logic: if the base is large and the question uses only a slice, you shouldn’t pay for the whole base every time. The full math is in how much it costs to run AI in production.
Step 3: when RAG is the right answer
RAG wins when three conditions appear together:
- The base is large relative to what each question needs — each query uses only a fraction of the total.
- The content changes often — policies, catalogs, versioned procedures you don’t want to retrain on every change.
- You need to cite the source — show where the answer came from, for auditing or user trust.
Classic cases where this fits: an internal knowledge base (runbooks, template proposals, policies), support with a versioned catalog, onboarding with living documentation, an assistant that answers “where is this?” by pointing to the document. The green light is objective: the documents exist, someone maintains them, and the same question keeps coming up.
The anatomy of a RAG that doesn’t embarrass you
RAG isn’t “throw a PDF into a vector store.” A serious pipeline has:
- Sources curated and versioned — garbage in, garbage out
- Chunking suited to the document type, not blind pieces
- Metadata — product, language, date and, crucially, permission
- Evaluated retrieval — you measure whether it brings back the right passages
- Generation instructed to refuse when there’s no evidence
- Citations or at least source tracking
- Human feedback to correct what comes out wrong
Skip step 4 and you don’t have RAG — you have a colorful search demo no one validated.
Step 4: when it (rarely) is fine-tuning
Fine-tuning solves a different problem from the other three. RAG and context provide what the model should consult; fine-tuning teaches how it should behave — a rigid output format, a schema that must come out identical on every call, a very specific tone. The textbook case is structured extraction or classification at very high volume, where repeating a long instruction on every call gets expensive and slow.
The common mistake is thinking fine-tuning is for “teaching the company’s knowledge.” It’s bad for that: knowledge changes, and retraining on every change is unworkable and expensive. For knowledge that changes, RAG. For fixed behavior at scale, eventually fine-tuning. It’s almost never the first step — and when someone proposes it on day one, it’s usually a sign the problem wasn’t understood.
The signs you’ve fallen into over-engineering
Some symptoms that the project got more complex than it needed to be:
- They set up a vector database to index 30 documents that never change
- No one is responsible for updating the base — it’s already outdated
- The real problem was integration or process, not a lack of text
- What you really wanted was a form or a rule — not an LLM
- They spent weeks on retrieval before testing whether a prompt would do it
RAG is a means, not an end. If you can’t point to the business decision it improves, you built infrastructure to feel modern. This is the same reasoning as applying AI in the business without becoming an eternal PoC: start with the pain, not the tool.
Risks founders underestimate in RAG
Even when RAG is the right choice, it brings risks that need an owner:
- Hallucination with a fake citation — it looks professional, it’s wrong. RAG reduces hallucination, it doesn’t eliminate it: bad retrieval produces a bad answer with confidence.
- Permission — if you index everything with no access control, the intern finds the payroll document in the index. Permission metadata is not optional.
- Stale content — the 2022 policy answering a 2026 question because no one removed the old version.
- Uncapped cost — reindexing, embeddings and tokens add up. Without a cap and without observability, the bill surprises you.
- Data protection law — indexing personal or sensitive data with no legal basis, control and retention is an incident in the making. Throwing HR and finance into the index “just to test” is the start of the problem.
The last two points connect directly with LGPD and GDPR in software and with secure architecture for sensitive data. Handle them before indexing, not after the incident.
RAG, agents and the boundary between them
A frequent question: when does the case call for RAG and when for an agent? RAG answers questions by consulting a base. An agent executes multi-step tasks, calling tools and deciding the path. Often an agent uses RAG as one of its tools — but not every knowledge problem becomes an agent, and putting autonomy where a simple lookup was enough only adds cost and unpredictability. The honest line is in AI agents: when they make sense.
At Pixelize, RAG enters with scope: defined sources, permissions, a retrieval quality metric and a path to production — not “a vector database because I saw it in the feed.” If you want to assess whether your case is really RAG or a step below, check out our web and product development. Most of the time, the right answer is simpler and cheaper than the hype suggests.
Frequently asked questions
What is RAG in one sentence?
It’s pairing a language model with search over your own sources: the system retrieves the relevant passages and only then generates the answer. Instead of training the model on the company’s knowledge, you make it consult that knowledge on the spot. It’s lookup, not memorization.
When is RAG overkill?
When the knowledge fits in a well-crafted prompt, when the base is small and stable, or when no one will keep the documents up to date. In those cases, building retrieval, vectors and reindexing adds cost and parts that break without solving a problem you had. Start with the simplest thing.
Does long context replace RAG?
For a few fixed documents, often yes: dumping everything into the context is simpler and faster than maintaining a search pipeline. But long context gets expensive per call as volume grows, because you pay for all those tokens every time. RAG exists precisely to send only the right piece.
What’s the difference between RAG and fine-tuning?
RAG provides what the model should consult; fine-tuning teaches how it should respond — format, tone, fixed structure. Knowledge that changes every week calls for RAG, because retraining on every change is unworkable. A rigid format repeated at very high volume is where fine-tuning starts to pay off.
Does RAG eliminate hallucination?
It reduces it, it doesn’t eliminate it. If retrieval pulls the wrong or outdated passage, the model answers wrong with the same confidence. The quality of chunking, permissions and retrieval evaluation matters far more than the choice of vector database.
Do I need an expensive vector database on day one?
Almost never. Many cases start off fine with good search over a few curated documents. Vector infrastructure enters with volume, latency requirements and evidence that simple search can’t keep up — not because it showed up in your feed.