Episode 16 — Context Engineering vs. Prompt Engineering: What Actually Drives a Good AI Answer

All opinions expressed are those of the individuals themselves, not necessarily of any company they work for.

Another Party Comment: “The Future Isn’t Prompt Engineering, It’s Context Engineering”

Sunil Baliga: Token drop time — and another one of these party comments came up. If you go to enough parties, apparently there’s no shortage of content for this podcast, so please, everyone listening, keep inviting me to your parties.

Anyway, this guy at the party told me the future of AI isn’t going to be prompt engineering — it’s going to be context engineering. He explained that prompt engineering is about giving instructions to an AI, while context engineering is about giving the AI the entire relevant context — so if you’re asking a question about a contract, you’d give it the contract, all the amendments, your company’s legal policies, whatever’s pertinent, so the AI can actually give you a better answer. He felt the future is context engineering, not prompt engineering. Interesting comment — I wanted to see what you two think.

Defining the Difference: Stateless UX vs. Stateful Infrastructure

Sajjad Khazipura: I’ll let Sam jump in — he’s been excited about this topic.

Sam Pooni: Prompt engineering optimizes the question — what exactly are we asking? Context engineering optimizes everything the model actually sees. Prompt engineering emerged around 2022–2023 as the craft of wording instructions to coax better outputs from LLMs. Context engineering is really a successor discipline: the systematic design of the entire input window — instructions, retrieved knowledge, memory, outputs, and state — treated as an architecture problem rather than a writing problem.

My thesis: prompting is UX, context engineering is infrastructure. The prompt is what’s in the message; context engineering is the pipeline that feeds it.

Sajjad: Do you see the prompt as effectively a subset of context — where the context is really the superset? Because while you can discuss the difference between the two, in practice the prompt and the context both just get fed to the model, led by the prompt.

Sam: Exactly right. Prompt engineering is a portion of the broader thing called context engineering. Prompt engineering breaks down into four things: scope (wording, structure, examples), nature (mostly stateless, per-request), technique, and unit of work — which is really a single screen’s worth of content. Context is different — it encompasses prompt engineering plus system prompts, retrieved documents, conversation history, tool call results, memory stores, environment, and state. It’s bigger. Context engineering is stateful; prompt engineering is stateless and per-request. And prompt engineering is mostly human-authored, while context engineering is machine-assembled.

Sajjad: I think that distinction matters for us as users of these systems — but as far as the LLM itself is concerned, it doesn’t see a separate prompt and context. It just sees a stream of text it has to work off of. It’s just a body of text fed to the model — the LLM doesn’t know what’s “the prompt” versus what’s “the context,” and it has no way to recognize that distinction from the content itself.

Sunil: Let me make sure I understood — you said prompts are stateless and context is stateful. Why is context stateful, specifically? State implies a relationship between a current query and something that happened previously.

Sam: Right — context engineering isn’t just prompting; it includes RAG, memory hierarchies, summarization, compaction, relevance ranking, context routing, token budgeting — all of it, and memory hierarchies themselves carry statefulness. Context engineering, taken together, is really infrastructure engineering. A prompt, by contrast, is completely stateless — you give it to the LLM, the LLM answers it, and that’s the end of that interaction.

Why Prompt Engineering Plateaued

Sam: One of the more interesting things is why prompting plateaued. Frontier models simply became less sensitive to phrasing. There used to be a real belief that clever phrasing got you better outputs, but the tricks stopped mattering over time — instruction-following improved to the point where clarity beats cleverness. Once instructions are clear, further wordsmithing only buys you single-digit-percentage gains. Missing information, on the other hand, still causes order-of-magnitude failures.

Sajjad: Taken to an extreme, people started saying things like “please, Mr. LLM, be nice to me, you’ve been outstanding so far” —

Sunil: “Good job, keep it up.”

Sajjad: Right — but what Sam’s really getting at is that what actually matters, far more than platitudes, is being precise about what you want executed, and providing the right workload structured the right way so the LLM can act on it. As I said, the LLM just sees a large blob of tokens — but it does learn to recognize the earlier parts of that stream, mostly, as instructions about the task it needs to execute. So there’s real merit in making instructions clear, unambiguous, and precisely specified — because while humans can assume context and often understand what’s meant even when it’s vague, a machine needs far more precision to comprehend the same thing.

The Cost of Over-Stuffing Context

Sunil: So is the “engineering” part of context engineering really about finding the minimum set of context you should provide? Because if you throw everything into the prompt, your token costs go up —

Sajjad: Not just token costs — even though we’ve talked on this podcast about million-token context windows, and LLMs believe they can process that much, there’s something called context rot that sets in. The model becomes unable to focus on the right parts of that context and can miss things — we’ve talked before about “finding the needle in the haystack.” Making sure an LLM picks up all the right instructions gets genuinely harder with very long prompts followed by very long context. So that’s a real challenge.

Calling all of this “prompt engineering” actually does a bit of a disservice to the scientific rigor real engineering work deserves. You can say “be precise, specify as completely as you can” — but there’s no scientific measure of precision, no scientific way to measure how complete your context actually is.

DSPy: Turning Prompt-Writing Into a Programming Discipline

Sajjad: That said, there’s a framework that came out a couple of years ago called DSPy — the idea is to turn prompt-building into an actual programming exercise. You write Python code, and the DSPy framework trains models — DSPy modules — that are then used to generate the actual prompts. I think that’s roughly right, though I could be off on some details. The core idea is you can do real Python programming to be as concise, precise, and complete as you want in specifying what you want done, and then use these models to generate your prompt.

Sunil: What’s the benefit of that? Is it something like writing conditional logic — if today is Thursday, do this; if it’s Friday, do that?

Sajjad: Exactly — you’re basically writing self-improving Python. That code gets processed by the DSPy framework, and effectively you’re turning the art of writing prompts into the science of engineering them.

Sunil: So the actual prompt that gets sent to the model is generated by the DSPy framework itself?

Sajjad: Yes — we actually use DSPy as part of our own platform. It’s a genuinely powerful tool.

Why Prompting Plateaued (Continued): The Agentic Shift

Sam: The other big reason prompting plateaued was the agentic shift. Multi-turn, tool-using workflows can’t be driven by a single static prompt — the context has to evolve turn by turn, and someone has to actively manage that evolving context. That’s really where the shift from prompt engineering to context engineering came from.

Longer context windows changed the problem too — we’ve talked about million-token windows on this podcast before, but bigger windows didn’t eliminate the need for curation; they made it harder. More room means more opportunity to pull in the wrong things. The failure mode shifted from “the model misunderstood” to “the model never saw the information it needed” — or saw too much of the wrong information.

Five Layers of Modern Context Engineering

Sam: Context engineering today is built on several layers:

1. Instruction layer (the old prompt-engineering layer): system prompts, personas, constraints, output contracts, tool definitions. This still matters — it’s a stable, version-controlled layer.
2. Knowledge retrieval layer: classic RAG (chunk, embed, vector search), hybrid retrieval, graph RAG for structured retrieval, and agentic retrieval.
3. Memory layer: short-term and long-term memory, and — as we’ve discussed before — episodic versus semantic memory.
4. State and tools layer: tool call results, environment, observations, intermediate artifacts, tool outputs.
5. Compaction and lifecycle layer: rolling summarization of history, structured note-taking, session isolation, and — at the very end — checkpointing and context handoff between sessions.

Context engineering, taken as a whole, is far more evolved than prompt engineering precisely because of this agentic, multi-turn nature — and the industry is actively moving toward better ways to manage that full lifecycle, from a cold corpus and index store through to the live in-context window.

Making Context Engineering “Scientific”

Sunil: How do you actually make context engineering scientific, then? Sajjad, you said prompting wasn’t really scientific — so how do you know how much context is too much versus too little? It’s a Goldilocks problem.

Sajjad: That’s really where the measure of success lies — how effectively you can semantically construct the right context. What that means is you’re offloading some of the intelligence out of the LLM and into the prep work that happens before context is ever handed to it. That’s where the symbolic side of a neuro-symbolic architecture starts to come in — you’re building your own understanding of what the question is actually about, and what content needs to be delivered to the LLM for it to succeed.

The deeper and richer that part of your pipeline is — at aggregating knowledge, understanding the question, decomposing it, figuring out precisely what context is needed, and supplying the right supporting evidence alongside it — the more precisely the LLM can answer. That’s genuinely a fair bit of work, which is why it’s called engineering. And to some degree, it is a science: the better you comprehend the question, the better positioned you are to fetch, accumulate, organize, and rank content by relevance and importance, so only the most important content actually reaches the LLM.

In the past, this was just naive RAG — you knew the answer was somewhere in a given textbook, so you uploaded the entire textbook and hoped the LLM would find the right answer inside it. Context engineering asks: why not give just the relevant chapter instead of the whole book? Why not the correct paragraph instead of the whole chapter? Today, people have gotten to the point of cherry-picking the exact entities and relationships needed, so the LLM effectively can’t get it wrong. The deeper and more evolved that engineering of the input window is, the more precise your answers will likely be.

That said, no matter how well you construct the context, the LLM can still occasionally go off on a tangent and make the wrong prediction — which is why, after generation, you still need a separate verification pipeline: claim-by-claim checking, regeneration, and so on. That’s a topic for another day; today we’re focused on the prep work needed to give the LLM the best possible chance to succeed.

A Concrete Example: The Math Textbook

Sajjad: Take a simple question: what is 2 plus 2? You don’t need to throw an entire high-school algebra textbook at the LLM every time. You can be discriminating — find the chapter on addition, then the specific rules of addition within it, and combine the relevant single-digit and multi-digit arithmetic rules for that specific question. The deeper you go into the actual subject matter, and the more precisely you carve out exactly the context the LLM needs, the more precise you can expect the answer to be.

What that means, practically, is that you’re now taking on some of the work. The easy, lazy path is to throw the whole textbook at the model and let it figure things out eventually. The better path is precisely picking out exactly the information the LLM needs to succeed — that’s context engineering.

Context Engineering Applies at Every Stage of a Pipeline

Sunil: If you have multi-stage LLM pipelines, would context engineering apply at every single stage — not just the first one?

Sam: Absolutely — that’s the context assembly pipeline, the full agentic loop. You get the query (the task step), then retrieve (dense, sparse, or knowledge-graph retrieval, however you do it). Then, as Sajjad described, you re-rank and filter. Then you assemble — budget plus position. Then model inference happens. Then tool calling — calls out to various environments and APIs, often through MCP servers. Then you distill, which summarizes and compacts the results with memory. Then it goes into the memory store — durable memory across its different types — and the loop goes back to retrieval again. Context is present at every point in that loop.

Sunil: It sounds like there could be genuinely different disciplines of context engineering — similar to how a system with a database, a network layer, and a UI has different specialists for each, even though they’re all part of the same application. It sounds like understanding the query, verification, and everything else you’ve described could each become their own discipline eventually, similar to what we discussed before.

Sam: Exactly right — and the design principles underneath are actually fairly simple. As Sajjad said earlier, the core idea is minimum sufficient context: finding the smallest set of high-signal tokens that maximizes the likelihood of the desired outcome. You don’t want everything — you want curation over accumulation.

Core Design Principles for Context Engineering

Sam: A few principles that come up again and again:

• Just-in-time over just-in-case. Retrieve what’s needed via tools rather than front-loading everything — because if you front-load everything, you get exactly the “lost in the middle” problem: the beginning and end get attention, and the middle drops off.
• Deliberate placement. Put critical constraints at the top, immediate task data near the bottom — nothing important gets buried in the middle of the window, because that’s exactly what gets dropped.
• Separate the control plane from the data plane. Never let retrieved data impersonate instructions.
• Structure, then measure with evals, then version and observe. Every one of these principles applies at every level of the pipeline — that’s the whole idea underneath context engineering.

Sajjad: Right — rather than feeding the entire math textbook for a simple “what is 2 plus 2” question, you can be discriminating: find the relevant chapter on addition, then the specific rules within it, and combine only what’s actually needed for that specific question — single-digit rules, multi-digit rules, whatever applies. The deeper you go into the subject matter, and the more precisely you carve out exactly the context needed, the more precise the LLM’s answer will be. The easy, lazy path is to throw the whole textbook at the model and let it sort things out; the better path is picking out exactly what’s needed — that’s context engineering.

Loop Engineering: Iterating Toward a Better Answer

Sajjad: Related to this is a concept called loop engineering. You construct context, give it to the LLM, and you may still not get the answer you want. If there’s an evaluation step after the LLM’s answer — is this good enough? — and it’s not, you go back through the loop: did I do something wrong last time? Can I engineer better context, or improve the prompt?

Loop engineering sounds exciting, and scientifically like the right thing to do — but every pass through that loop burns tokens. The more iterations, the more tokens you burn, and the real open question is whether you’re actually converging on a correct answer or compounding errors. You don’t automatically know which. Whatever governance mechanisms, checks, and verifications you have in place — and the quality of the feedback signal coming out of them — determine how well that loop actually behaves. Without that discipline, a loop can just as easily diverge as converge, aggregating and compounding errors into a genuine runaway case.

This is also, simply, a matter of burning more and more tokens — people call it “token maxing,” and there’s growing concern in the industry about how many tokens get burned this way, and whether it can be done more efficiently. The more you think seriously about doing this efficiently, the more your architecture naturally gets pulled toward a neuro-symbolic approach.

Wrap-Up: What Do I Say vs. What Does the Model Need to See

Sunil: We’re almost out of time, Sam — go ahead and close us out.

Sam: If there’s one thing to take from this episode: prompt engineering is really about “what do I say?” Context engineering is a different question entirely — what does the model need to see, when, and at what cost? In prompting, you’re always worried about what you have to say to the LLM to get the right output. In context engineering, you’re thinking about what the model needs to see, when it needs to see it, and what it costs to get it there — through the whole loop Sajjad described: retrieval happens, you cycle through, you distill and compact into durable memory, and you reiterate until the answer is actually in a state you can use. What matters is the context assembly pipeline, the agent loop, and the per-turn data flow — all working together. There’s a lot more once you get into the weeds, but we wanted to keep this accessible for anyone listening.

Sunil: This was genuinely helpful — a really interesting episode. I’m going to have the guy who raised this question at the party listen to it and see what he thinks; if he gives me feedback, I’ll share it with you both. Let’s wrap the recording — if you two can hang on a second, that’d be great. Thank you.