What is context engineering?

Ask an AI model a question it does not have grounds to answer and it will not tell you that — it will answer anyway, in the same confident voice it uses when it actually knows. The gap between those two cases is not something the model manages by itself. It is something you manage, by deciding in advance what the model gets to see and use for a given task. That deliberate decision — what information, tools, and memory land in front of the model, and what does not — is context engineering.

The term is new; the problem is not. Anyone who has built a chatbot, an agent, or a RAG pipeline has been doing a version of context engineering already, usually without the name for it. What changed recently is that the practice got named, compared across projects, and started showing up as its own line item in how teams plan AI work — including as a recurring theme at technical conferences in 2026.

In plain words. A large language model answers based only on the text sitting in its context window — the finite amount it can read at once for a given request. Context is everything placed into that window besides the raw question: retrieved documents, conversation history, tool definitions, system instructions, prior facts about the user or task. Context engineering is the discipline of deciding, on purpose, what belongs in that window and what does not — as opposed toprompt engineering, which is about how you phrase the request itself.Retrieval (see RAG below) is one way to fill context; it is not the whole of context engineering, just its best-known technique. Memory is context that persists across turns or sessions rather than being fetched fresh each time. Atool (or tool definition) is a capability — search the web, query a database, send an email — the model can be given access to, along with instructions on how to call it.

Why "engineering," not just "prompting"

Prompt engineering treats the model's knowledge as fixed and asks: what wording gets the best answer out of what it already knows (or is willing to guess)? Context engineering asks a prior question: what should the model even be looking at? A perfectly worded prompt cannot make up for a model that was never given the one document, the one fact, or the one tool it actually needed.

The distinction is worth stating precisely, because the two get blurred constantly:

• Prompt engineering — how you ask. Instructions, examples, formatting, tone, chain-of-thought scaffolding. It shapes how the model uses what it has.
• Context engineering — what it has to work with. Retrieval, memory, tools, system state. It shapes what is available to use in the first place.

A well-engineered prompt over the wrong or missing context still produces a fluent, wrong answer — because nothing about phrasing fixes an absent fact. Conversely, the right context handed to a plainly-worded prompt often outperforms a beautifully-worded prompt over the wrong context. This is not a claim that prompting doesn't matter — it does, especially for reasoning tasks — but it is a different lever, and it is usually not the one that is broken when an AI answer is confidently wrong.

The parts of context

Context is not one thing; it is at least four, and most real systems combine several of them.

Retrieval

Retrieval is the best-known piece: search a document store or knowledge base and pull the passages relevant to the question into the model's context before it answers.Retrieval-augmented generation (RAG) is the standard name for this loop — retrieve, then generate from what you retrieved — and it is common enough, and deep enough on its own, that we cover it separately: seeWhat is retrieval-augmented generation (RAG)? The short version relevant here: retrieval is one method of doing context engineering, not a synonym for it. You can do context engineering with no retrieval at all (a small curated context, for instance), and you can do retrieval badly enough that it hurts more than a simpler alternative would have.

Memory

Memory is context that survives past the single request — facts about a user's preferences, a running project's state, decisions made in an earlier session. Where retrieval typically answers "what does this document say," memory typically answers "what have we already established." The engineering questions are similar in shape (what to keep, how much, how to fetch it back) but the source is different: memory comes from the system's own history rather than an external corpus.

Tools

An agent that can call tools — search the web, query a database, read a file, send a message — needs those tool definitions loaded into context so it knows what is available and how to call it. This sounds like a small detail; at scale it is not. Every tool definition costs context-window space and, past a point, choosing correctly among many tools gets harder for the model, not easier. That trade-off — and what it costs in measured tokens — is covered separately: see The token cost of MCP tool definitions.

Domain and company knowledge

The fourth piece is the structural knowledge that is not really a "document" at all: which system owns which data, how a company's tools connect, who to ask about what, what a specific internal term means. This is context too, and it is often the hardest to keep current, because it lives in people's heads and in Slack threads more than in any single searchable file. Keeping a small, deliberately curated map of this kind of knowledge — rather than trying to index everything — is one answer to that problem; we describe one implementation of it later in this post.

Why it matters more than it looks like it should

The uncomfortable fact underneath all of this: a language model's confidence is not a reliable signal of whether its context actually supports the answer. Ask it something it has good grounds for, and it answers assertively. Ask it something it has no grounds for, and — absent explicit instruction to say "I don't know" — it very often answers just as assertively, filling the gap with something plausible. The tone does not change; the correctness does. That is sometimes called the overconfidence gap, and it is the reason context engineering is not a nice-to-have layer of polish. It is the thing that actually moves an answer from plausible-sounding to grounded, when wording alone cannot.

This is also why "just use a bigger model" or "just give it a bigger context window" are weaker fixes than they sound. A bigger window can hold more, but more is not the same asrelevant — and research on long-context use (the "lost in the middle" effect, discussed in the RAG post above) shows that stuffing more into context can degrade accuracy rather than improve it. The lever that reliably helps is choosing better what goes in, not simply allowing more room for it.

The landscape: a spectrum, not one right answer

Talk to different teams about context engineering and you will hear different answers to "so what do you actually do," because the honest answer is that it is a spectrum with real trade-offs at every point, not a single correct architecture.

• Curated, hand-maintained context. A small set of facts, links, or a navigational map, kept deliberately short and updated by people (or a light automated process) rather than searched fresh each time. Cheap to reason about, easy to trust, but it does not scale to a large or fast-changing body of knowledge, and it depends on someone keeping it current.
• Large retrieval pipelines. Vector search, hybrid keyword+semantic search, reranking, sometimes graph-structured retrieval over a large or frequently changing corpus. Scales to volumes a curated list cannot, but brings its own failure modes (near-miss retrieval, chunking problems, the need to evaluate retrieval quality on its own) — detailed in the RAG post linked above.
• On-demand / just-in-time retrieval. Instead of pre-loading a fixed slice of context, the system searches for what it needs at the moment of the request. This trades upfront context cost for a dependency on the search step working well every time.
• Evaluation as its own step. Regardless of which of the above you pick, the teams that improve reliably tend to be the ones who measure whether their context step is actually returning the right material — not just whether the final answer reads well. A confident wrong answer and a confident right answer look identical from the outside unless you check the context that produced them.

None of these is "the" context engineering approach. They are points on a spectrum of how much you pre-decide versus fetch on demand, and the right point depends on how large and how volatile the underlying knowledge is — a handful of stable facts is a different problem from a document base that changes daily.

Context engineering vs prompt engineering — the short version

Since the two terms get used almost interchangeably in casual conversation, it is worth restating the distinction in one place: prompt engineering is how you ask; context engineering is what the model knows or can reach when you ask it. You can have excellent prompt engineering over poor context (a well-phrased question the model still cannot ground) or workmanlike prompting over excellent context (a plain question, answered correctly, because the right material was already in front of the model). They are complementary skills, not substitutes for each other, and improving one does not fix a problem in the other.

A note on where the term came from

Context engineering is not brand new as a practice, but it has become a much more common label over the last year or so, showing up as a named theme at AI-engineering conferences and in vendor and practitioner writing — for instance in the session "The Retrieval Layer in Context Engineering: From Intuition to Implementation" (Evgeniya Sukhodolskaya) at WeAreDevelopers World Congress 2026 in Berlin. That rise in visibility does not mean the underlying practice is new; retrieval systems, memory layers, and tool-definition management all predate the term. What the label adds is a way to talk about these as one coherent discipline instead of separate, unrelated engineering problems.

How we think about it (a disclosed note)

A brief note on where mcpgate sits in this, since a disclosed position is more useful than pretending we have no angle. mcpgate is a gateway that connects AI agents to a company's tools, and the context problem it deals with day to day is mostly the "domain and company knowledge" and "tools" pieces above — which system owns what, how things connect, which tool to reach for — rather than large-scale document retrieval. For that shape of problem we made a deliberate choice toward the curated end of the spectrum: a small, maintainedcontext map plus on-demand search over available tools, instead of a heavy retrieval pipeline. That is one point on the spectrum described above, with the trade-off any curated approach carries: it depends on the map staying current, and it is not the right answer for a large, fast-changing document corpus — that is squarely RAG territory, covered in the post linked earlier. We walk through that choice in depth — including the thorough auto-updating version we built and then deleted, and where the approach still breaks — inWhich context problem is actually yours?

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(2020) — background on RAG as one context-engineering technique.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023) — evidence that more context is not automatically better context.
• Evgeniya Sukhodolskaya, The Retrieval Layer in Context Engineering: From Intuition to Implementation — WeAreDevelopers World Congress 2026, Berlin (AI Engineering track).
• mcpgate blog: What is retrieval-augmented generation (RAG)?, The token cost of MCP tool definitions, Which context problem is actually yours?