Which context problem is actually yours? We built the fancy one, then deleted it

Every team wiring an AI agent into their own systems hits the same wall at about the same time. The model is fine. The tools connect. Then the agent confidently tells someone the wrong repository owns a service, or answers from a status that changed three sprints ago. The reflex, learned from a hundred blog posts, is immediate: we need RAG. Index the wiki, index the tickets, embed all of it, retrieve chunks at question time.

We run an MCP gateway in production — mcpgate, self-hosted, connecting AI agents to a company's tools — so we hit this wall early. We did not build a RAG pipeline. And the reason is not that RAG is bad; it's that "which technique is best" is the wrong question. The right one is:

If you want the RAG basics first, start here:what retrieval-augmented generation is and why the retrieval step decides answer quality. This post assumes you have them.

In plain words. Context engineering is deciding what an AI agent has in front of it when it acts — which facts, which tools, which state — and, just as much, what to leave out. RAG indexes documents and retrieves the relevant chunks. A context map is a small, human-curated set of navigational pages: which system owns what, how things connect, current status. A knowledge graphmodels entities and the relationships between them so you can traverse connections.Tool search retrieves the right tool on demand instead of keeping every tool definition resident.

Three tools, three problems

They are not competitors at the same level, and they compose — a mature setup might use all three. RAG finds semantically similar text; it does not understand structure. A graph makes relationships first-class so retrieval can follow branches. A context map and a knowledge graph are both "maps", but a map is curated, small, and a human stands behind it, while a graph is derived, large, and needs constant maintenance to not drift.

The honest move is to name your biggest problem first. Ours was orientation— not depth, not traversal. The context that broke our agents was a few dozen navigational facts, not a decade of Confluence. So we picked for that, and it looks nothing like a RAG stack.

The observation that drove the bet

Before any architecture, there was a plain observation from dogfooding our own gateway:the context that actually changed the agent's answers was small, and it was stable.

It wasn't sprawling prose. It was navigational: which repo a service lives in, which system is the source of truth for a number, who owns a domain (as a role, not a person), what state a project is in. A few dozen facts, most changing on the timescale of weeks, not minutes. When the agent got one wrong, it was the expensive kind of wrong — quiet and confident, the exact failure the RAG post describes, where the model reasons faithfully over the wrong premise and sounds just as sure as ever.

The "index everything" instinct assumes the hard part is volume. Ours wasn't volume — it was freshness and correctness on a small set of facts. Different problem, different tool.

We built the thorough version first. Then we deleted it.

This is the part I'd have wanted to hear from someone else before I spent the time.

Early on, we designed the proper version. Auto-updating context. A changes-API so the map would stay in sync with everything that moved — every rename, every status change, every reorg, streamed in and reconciled. It was, on paper, the responsible thing: never let the context go stale.

Building it, two things became clear. First, it was overkill for the payoff. Second — and this is the one that actually changed our minds — the things that changed constantly were mostly not the things that answered the questions. The high-churn data was noise for the agent's purposes. The context that actually solved problems barely moved from month to month. We had built an elaborate freshness machine for a freshness problem we didn't really have.

So we deleted it. What we kept is deliberately dull:

1. A small, curated context map

The context map is navigational knowledge as pages — which system owns what, how things connect, current status, owners-as-roles, links into the source repos. On a question like "which repo is X in / who owns Y / where does the detail for Z live", the agent reads the map first and answers from it, not from training data or a stale checkout.

Two decisions matter more than they look:

It's curated, not generated. We don't run an LLM inside the gateway to auto-summarize the codebase into map pages. Keeping AI out of this path is deliberate: an auto-generated map inherits every hallucination of the thing generating it and launders it into something that looks authoritative. A human (or a human-reviewed process) writes navigational pages; the gateway only reads them. The cost is that someone maintains it. The payoff is that when the map says a service lives in repo A, a person stood behind that — it isn't a statistical guess.

It syncs from source on every push — and that's the whole freshness story.No changes-API, no reconciliation engine. The map lives in a source repo and syncs to the gateway on push, so what the agent reads is fresher than any local clone or anyone's memory. That single mechanism attacks the failure we saw most: not "the agent didn't know", but "the agent knew something that used to be true." It's the cheapest thing that keeps the read path honest, and after killing the fancy version, it turned out to be enough.

2. Tool search instead of resident tool definitions

The other half isn't facts, it's capabilities: which of the thousands of actions across every connected service (Jira, GitLab, Notion, Google Workspace, Grafana, Sentry…) the agent should reach for now.

The naive answer — load every tool definition so the model can pick — doesn't scale. A full catalogue is a permanent tax on every request, and past some size it candegrade selection, not just cost — the same "less is more" effect. We covered the token side in the token cost of MCP tool definitions and the accuracy side in dynamic action discovery. So instead of keeping everything resident, the gateway retrieves the right actions on demand via search — orders of magnitude cheaper than a resident catalogue. To put a number on the tax: Anthropic's own five-server example sits at ~55,000 tokens before the conversation even starts, and a single verbose server we measured — Amplitude — is ~140,000 on its own. Behind on-demand search, an aggregated setup keeps well under a thousand tool-definition tokens resident (~900 in the setup we measured) and loads the rest only when the model asks.

Same insight as the map, applied to capability instead of fact: don't keep everything in front of the model, keep a way to fetch the right thing when the moment comes.

Why bother curating — the payoff is organizational

Curation has an obvious cost: a human maintains the map. It's worth being explicit about why that cost pays for itself, because the payoff isn't really technical — it's organizational.

A domain expert holds a lot of tacit knowledge about how things fit together — which is exactly the knowledge a newcomer, or an AI agent, most lacks. Curating it into the map writes that breadth down once, in a place everyone reads. The expert stops re-answering "where does this live / who owns that" on every follow-up. Everyone else — including people who start with very little sense of the connections — inherits the expert's orientation as their baseline. And the company depends a little less on whether that one person is available this week.

That's the same "move the starting point to the right" idea from thecontext charts: a shared, curated map raises the baseline every answer starts from. Curation isn't overhead you tolerate — it's how one person's breadth becomes everyone's starting point. (It's also where this stops being a purely technical story and becomes a knowledge-management one — the part a manager cares about — but that's a piece for another day.)

Why this isn't RAG (and why that's fine here)

Be precise, because "we don't use RAG" is easy to over-read.

Both mechanisms are, technically, retrieval. Tool search is a retrieval problem and can fail exactly like vector retrieval — semantic near-misses, the right action ranked just below the cutoff, an identifier embeddings fumble. Thedynamic action discovery post shows two implementations of the same "search for the right tool" pattern landing at 94% versus 34%tool-selection accuracy on the same harness. So we didn't escape retrieval. We escaped the heavy vector-indexing pipeline — chunking, embedding store, re-indexing loop, reranking stack — for a class of context where that machinery was overkill.

Why it's overkill here: the context is small, structured, navigational. You don't need approximate-nearest-neighbor search over a million embedded chunks to answer "which repo owns billing" when the authoritative answer is one curated page. Curation is retrieval where a human did the hard ranking once, up front, instead of a pipeline redoing it probabilistically every query.

Where it breaks (the part we're still working on)

No hedging: the setup above is not finished, and calling it "solved" would be dishonest. What's still weak:

Ranking / reranking

Tool search is today a single pass of BM25 with a relevance-threshold cut — no second ranking stage. When the right action sits just below the cutoff, the agent picks a plausible neighbor and acts — the quiet-and-confident failure, now on the capability side. A reranking pass is the obvious lever (highest-leverage in the RAG world too), and it's on the list, not in production.

Grading / evaluation

We can't yet answer "is the context layer working" with a number instead of a vibe. There's a golden set of tool-selection tasks written down, but the harness that would score it end-to-end isn't wired up, and nothing measures retrieval rank — for real questions with known answers, does the layer return the right page/action, and how highly? Until that's measured, improvements are educated guesses, not moves on a metric.

Read-repair of stale or wrong pages

Sync-on-push keeps the map fresh relative to source, but can't fix a page that's wrong at source, or a fact that drifted without anyone noticing. The honesty half of this is now in place: on the read path the agent gets a deterministic drift signal — which sources changed since a page was last verified, ranked by how material the change is — so it can hedge instead of answering confidently from a page that moved underneath it. What'snot built is the loop that acts on the signal: flag-for-repair-and-repair is still a human curation act, not something the agent closes on its own. The map can now tell you it might be stale; it can't yet heal itself.

Enforcement / harness

For the general case, reading the map first is convention, not guarantee — the gatewayinstructs the model to consult the map before answering navigational questions, but nothing gates a general answer on having done so. The enforcement machinery exists only in narrow or dormant form: one write path (creating a Jira issue) does force a round-trip that delivers map-sourced conventions before it proceeds, and there's a grading gate thatcould block an ungrounded answer — but it ships off by default with no blocking policy attached. Convention degrades under load and model drift; turning that dormant seam into a dependable "ground this class of question, or refuse" is the real open work.

What we'd tell someone making the same call

The useful version of our experience is a checklist, not "copy our stack":

1. Name your biggest context problem before building for it. Orientation, depth, or traversal? They want a map, RAG, or a graph respectively. We were surprised ours was orientation, and small.
2. Measure what actually moves answers. Ours was smaller and more stable than the instinct assumed. Yours might genuinely be large and unstructured — then build the RAG, and read the retrieval-quality post first.
3. Don't over-engineer freshness. We built auto-update + a changes-API and deleted it: the fast-changing data mostly wasn't what solved the problem. Sync the stable, curated part simply.
4. Treat tool selection as its own retrieval problem. More tools than fit in context = a retrieval problem whether you called it that or not, failing the same ways document retrieval does.
5. Keep AI out of paths where a wrong answer looks authoritative.Auto-generating your source-of-truth map quietly reintroduces the hallucination you were trying to fix.
6. Be honest about what you haven't measured. We can't yet grade our context layer to the standard we'd want, and neither, probably, can you on day one. Knowing that is the difference between "not there yet" and "broken and don't know it."

We made a specific bet for a specific shape of problem; so far it has cost less to run and been easier to keep correct than indexing everything would have. It's also visibly unfinished — no reranking, no scored evaluation yet, a drift signal but no self-repair, and grounding that's still mostly convention. A working bet with named open edges is the honest state of context engineering in production, at least in ours.

Sources & further reading

• What is retrieval-augmented generation (RAG)? — the RAG primer; retrieval quality decides answer quality.
• The token cost of MCP tool definitions — why keeping every tool resident taxes every request.
• Dynamic action discovery — tool selection as a retrieval problem; 94% vs 34% tool-selection accuracy on the same harness.
• Context map (admin docs) — how the curated, sync-on-push navigational map works.