What is retrieval-augmented generation (RAG)?

Ask an AI model a question about your company and it will answer with the same even confidence whether it knows the answer or is inventing one. Retrieval-augmented generation — RAG — is the standard fix: before the model answers, you fetch the relevant documents and hand them to it, so it responds from your material rather than from whatever it happened to memorise in training. It works. It is also where most of the disappointment with “we built a RAG bot” comes from — and almost always for the same reason: the retrieval, not the generation.

In plain words. A large language model is the AI that writes the answer. Its context window is the finite amount of text it can read at once. Retrieval is the step that searches your documents and pulls the few passages most likely to help. To search them, each passage is first cut intochunks and turned into an embedding — a list of numbers capturing its meaning — so a vector search can find chunks whose meaning is close to the question. RAG is that whole loop: retrieve, then generate from what you retrieved. Grounding is the goal — the answer is anchored to real source text. Reranking is a second, sharper pass that reorders the retrieved chunks so the most relevant ones come first. This post is about why that retrieval loop is the part that decides your answer quality — and what to do about it.

How RAG works

In one line: retrieval-augmented generation is when youretrieve the relevant source text first, then let the model generate its answer from that text. You will also see it written without the hyphen — retrieval augmented generation — or just as RAG; they are the same thing.

The term comes from a 2020 paper by Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The idea it named is now everywhere, and it solves three real problems at once. A model’s training data is frozen at a cutoff date, so it cannot know what happened since; it never saw your private, internal documents at all; and when it lacks a fact it tends to generate a fluent guess rather than admit the gap. RAG addresses all three by putting the right source text in front of the model at question time: recent, private, and specific.

The mechanics are simple to state. You split your documents into chunks and store an embedding for each. When a question arrives, you embed the question the same way, find the chunks whose embeddings are closest, and paste those chunks into the model’s context alongside the question with an instruction to answer from them. The model reads the passages and writes an answer grounded in them — ideally with a citation back to the source. Nothing about the model changes; you have changed what it is looking at.

That is the appeal, and it is genuine: no retraining, answers that track your latest documents, and a citation trail a reader can check. The trouble is that the quality of the whole thing rests on one quiet assumption — that the chunks you retrieved are the right ones.

The R is the weak link

At WeAreDevelopers 2026 in Berlin, one of the AI-engineering talks was titledThe R in RAG: Why retrieval is often the weakest link (and how to fix it). The title is the finding. In a RAG system the generation step is largely a solved commodity — today’s models write fluently from whatever context they are given. The variable that actually moves answer quality is what you put in that context, and that is decided entirely by retrieval.

When retrieval is off, the failure is not loud. You do not get an error; you get a confident, well-written, wrong answer, because the model faithfully reasoned over the wrong passages. The common ways retrieval goes off:

• Semantic near-misses. Vector search returns chunks that areabout the topic but do not contain the specific fact asked for — close in meaning, useless for the answer.
• Too much, not too little. Stuffing twenty chunks in “to be safe” buries the two that mattered. Models read long contexts unevenly — Liu et al. (2023), Lost in the Middle, showed accuracy drops when the relevant passage sits in the middle of a long context rather than at the edges.
• Chunking that splits the answer. A fact spread across a table, a heading, and the paragraph under it gets cut into chunks that each carry half of it and retrieve neither well.
• Vocabulary mismatch. The document says “annual leave”; the user asks about “vacation days.” Pure semantic search usually bridges this, but exact identifiers — an error code, a SKU, a config key — are exactly where embeddings are weakest.

It is worth being precise about what is going wrong, because it is not a model problem and a bigger model will not fix it. The model’s confidence is flat regardless of whether its context is right — that gap between how sure an answer sounds and how grounded it actually is does not close on its own. Retrieval is what moves the answer along the axis that matters:

The fixes that actually help

There is a real landscape of techniques here, and they are complementary rather than competing. None is proprietary; all of them are described in public work and shipped in open tools. The map from failure to fix, before the detail:

Better retrieval before better models

• Reranking. Retrieve a generous first pass cheaply, then run a cross-encoder reranker over those candidates to score each one against the actual question and keep only the top few. This is usually the highest-leverage addition to a naive vector pipeline, because it directly attacks the semantic-near-miss problem.
• Hybrid retrieval. Combine keyword search (BM25) with vector search. Keyword search nails the exact identifiers embeddings miss; vector search handles the paraphrases keyword search misses. Together they cover each other’s blind spots.
• Query rewriting. Expand or rephrase the user’s question before retrieving, so a terse query becomes something a document is likely to match.

Give retrieval structure

Plain chunk-and-embed treats your knowledge as a bag of unrelated passages. When the relationships between things matter — who owns what, what depends on what — graph approaches such as Microsoft’s GraphRAG build an explicit graph over the material first and retrieve over that structure. It is more to build and maintain, and it earns its keep on connected, entity-heavy corpora rather than a flat pile of FAQs.

Often: less context, not more

The instinct when a RAG answer is wrong is to retrieve more — more chunks, a bigger context window, a larger index. Lost in the Middle is the counter-evidence: past a point, more context lowers accuracy rather than raising it. Fewer, higher-precision passages usually beat a larger, noisier set. This is the least intuitive lever and often the most effective one: spend the effort on retrieving the right passages, notmore passages.

Measure retrieval on its own

You cannot improve what you only judge by the final answer. Evaluate the retrieval step separately: for a set of real questions with known answers, is the passage that contains the answer actually in what you retrieved, and how highly is it ranked? Precision and recall on retrieval — not just a vibe-check of the generated text — is what turns “the bot feels flaky” into a number you can move.

Retrieval shows up even when you are not building RAG

Retrieval quality is not only a document-QA concern. The same problem appears the moment an AI agent has more tools than fit in its context: it has to retrieve the right tool for the task, and that retrieval can be good or bad in exactly the same ways. We wrote about that case separately — dynamic action discovery — where two implementations of the same “search for the right tool” pattern benchmarked at 94% versus 34% tool-selection accuracy on the same harness. Same lesson, different surface: the pattern is easy; getting the retrieval underneath it right is the hard part.

How we think about it

A note on where we sit, since the honest version is more useful than a pitch. mcpgate is a gateway that connects AI agents to a company’s tools; the context it has to get right is navigational — which system owns what, how things connect, which tool to reach for — not a large corpus of unstructured documents. For that shape of problem we made a deliberate bet on the less-context end of the spectrum: a small, curatedcontext map plus on-demand search over tools, rather than a heavy vector pipeline. In our own use the context that actually changed answers was small and stable, so that curation was cheaper and more reliable than indexing everything.

That is one point on the spectrum, and it has a real trade-off: it does not replace RAG for genuine document question-answering over a large, changing knowledge base — that is exactly where the retrieval techniques above earn their keep. Different problem, different tool. If you are building the document-QA kind, the order of work that pays off is the one above: get retrieval measured and reranked before you reach for a bigger model. And if you are curious why “just connect everything and let the model sort it out” is its own kind of context problem, that is the token cost of tool definitions.

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(2020) — the paper that named RAG.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023) — accuracy degrades by the position of the relevant passage in a long context.
• Microsoft, GraphRAG — open-source graph-structured retrieval.
• The R in RAG: Why retrieval is often the weakest link (and how to fix it)— AI-engineering talk, WeAreDevelopers World Congress 2026, Berlin.
• mcpgate blog: Dynamic action discovery,The token cost of MCP tool definitions.