Episode 2 — Beyond RAG: Why Reasoning and Verification Are the Real Fix for LLM Hallucinations

Recap: Why "Humans Make Mistakes Too" Doesn't Hold Up

Sunil Baliga: All right, here we go — week two of the podcast, and we've finally got a name: Token Drop. Last week I said we'd keep it to 15, 20 minutes, no more than half an hour, and of course we blew right past that. It was a good conversation, but we ran long, so this week I'll try to keep it under 30 minutes.

Last week we talked about hallucinations, and one argument that comes up a lot is: people hallucinate too — people make mistakes — so why shouldn't we accept the same from an LLM?

That got me thinking. Like you, Sam, I started out in the semiconductor world, and one thing we never tolerated was hallucinations or mistakes in our protocols — USB and others. We built in error correction: ECC for memory, forward error correction for SerDes, 8B/10B encoding for things like USB. We didn't accept errors in machines, and we don't really accept mistakes from employees either — we give feedback, training, ask for self-reflection. So the premise that "it's okay because humans make mistakes too" doesn't really hold.

What I want to dig into today: we've talked about how errors get corrected in semiconductors and protocols — what are the techniques for correcting, or at least mitigating, hallucinations in LLMs? Sam, what do you think?

Why the Hardware-Error Analogy Breaks Down

Sam Pooni: You raised a really good point. One of the questions that came up between us earlier was about byte encoding — 8B/10B encoding schemes and similar error-correction mechanisms. The core premise there is that errors are inevitable. If we treat hallucinations the same way — as something inevitable, something we have to live with — the real question becomes: do we have an equivalent way to correct them?

Sunil: So what's the answer?

Sam: It's complicated, and I don't think we fully have one, because the analogy holds in some places and breaks down badly in others. Here's the distinction: errors in hardware are random and stochastic — they appear unpredictably. But hallucinations are structured and systematic. The distribution of hardware errors is known and can be characterized mathematically. The distribution of hallucinations is unknown and highly context-dependent.

There's also this: a correct signal exists independently of the error in hardware — you have a known ground truth to check against. With hallucinations, the ground truth may not exist, or may be undefined. Errors "announce" themselves in some sense — a decoder can often localize where a bit error occurred. Hallucinations don't announce themselves. They're fluent. They read exactly like correct answers, so you can't simply look at an output and localize where it went wrong.

Sunil: But haven't we actually done that? Like the oil and gas case study — the Santos document. The customer gave us specific questions with a known ground truth — an oil well drilling report — and we could show our technology answering correctly versus an LLM alone answering incorrectly. So in that case, we did have a ground truth to compare against.

Sam: Even there, I think Sajjad has a good angle on the ground-truth question — Sajjad, go ahead.

Machines vs. Humans: Why We Expect More from Machines

Sajjad Khazipura: When we speak of machines, the human expectation is that they perform exactly as designed — that's where we get the expression "it works like clockwork." We hold machines to a higher standard than we hold humans, and we're far less tolerant of errors in them. We expect them to work every time, according to spec.

In the physical world, yes, there can be errors — that's a function of physics and mechanics.

Sunil: The ones and zeros at the bit level.

Sajjad: Exactly — network errors, program errors, machine errors in semiconductor technology. Those happen because of physical reality: cosmic rays, quartz crystal defects, timing variation between two clocks that are each expected to run at exactly the same frequency but come from different physical crystals. That physical element is where the errors come from.

But software is different — it's abstracted to a logical level. It's mathematics, it's logic. Two plus two should always equal four; there's no room for error to creep in there. So we're conditioned to expect more from machines in the physical world, and even more once we abstract them into software systems. That's exactly why LLM hallucinations come as such a surprise — even a shock — and why we need a real way to deal with them.

Sunil: So going back to my question — in an abstracted system like this, where you're working with knowledge instead of bits, how do you mitigate it?

Where Hallucinations Come From — and Why RAG Isn't Enough

Sajjad: Often, when building LLM-based systems, we deliberately avoid relying purely on the model's parametric knowledge — what it learned during training. Instead, we augment it with knowledge held in documents.

Here's why that matters: an LLM's internal knowledge has a training cutoff, so you should expect it to be dated. Beyond that, the model stores knowledge through a lossy, compressed representation. On the way in, information gets compressed; on the way out, the model is making probabilistic inferences — predicting tokens. So there's potential for error both in how knowledge goes into the model and in how it comes back out.

To address the "dating" problem, classical enterprise architectures use RAG (retrieval-augmented generation): you augment the LLM with documents that define specific facts, and expect it to answer questions grounded in that document knowledge. That reduces hallucination risk — but it doesn't take it to zero.

Sunil: You can't get rid of it entirely.

Sajjad: Right — even in a well-built RAG architecture, LLMs still hallucinate. RAG is one way to mitigate the risk, but it's not the whole answer. You still need verification mechanisms after the fact — post-synthesis verification — to check that the answer the LLM gave actually matches the ground truth in the source document. Those verification architectures can get quite sophisticated: claim-by-claim verification, embedding-based comparisons, and so on. It's really a function of how sophisticated your verification is and how much time and budget you're willing to put into it.

Sunil: And the more verification you do, the more it costs — more tokens, more time. If you want the most truthful answer possible, you keep layering on verification.

Sajjad: Exactly — and in RAG architectures, the answer isn't always contained in a single document; it might be spread across several. So you also have to think about how you rank those documents, and how you do pre-synthesis verification — making sure you're feeding the LLM the right set of facts before it generates an answer.

Sunil: What do you mean by "pre" — is that just prompt engineering, or something different?

Sajjad: It's about taking all the documents that might potentially contain the answer and prioritizing them — as Sam mentioned, you're working within a context window, so you want the highest-priority documents (the ones most likely to contain the answer) ranked higher, and the less relevant ones ranked lower before you feed them to the model. That's called re-ranking. You can also do pre-synthesis verification just to confirm the answer is actually present in those documents before generation even happens. There are a lot of ways to run verification — it really comes down to how critical it is that the answer be truthful, and how much you're willing to invest in that.

The Problem with "One-Shot" RAG — and Why Reasoning Matters

Sam: That's an important point. Today, when you look at LLM-plus-RAG systems — grounding layers, RAG, tool calls, structured knowledge — the real problem is that they typically only get one shot at giving everything to the LLM. The system gathers all the chunks, all the related material, pushes it all in at once, and says "here it is, figure it out."

But is that how you reason? It's not. Reasoning is a chain of thought — you understand something first, and once you understand it, you ask better questions. Human understanding doesn't work by collecting everything into semantic chunks all at once. We reason one step at a time: you pick up a book, study it, form a general understanding of the outline and the main theme, and then you ask further questions. There's a back-and-forth as you assimilate information.

A typical naive RAG system only gets one shot, so it can get overwhelmed by the volume of data it's handed. And think about the attention mechanism inside a transformer — every token is weighted relative to every other token. If you hand it a million words, or content that spans multiple context windows, some of the genuinely important words get diluted and effectively lost.

That's part of why people are increasingly interested in reinforcement learning for this problem — with reinforcement learning, the system can go back, ask a question, get an answer, and then ask another question based on that. That iterative, back-and-forth reasoning capability is part of what sets human cognition apart. So the real question is: how do you bring that kind of reasoning into how these systems actually understand things?

Sunil: So you're saying — add reasoning to mitigate hallucinations?

Sam: Exactly. Sajjad and I were actually discussing this earlier — reasoning capability is one of the things that's going to be a real differentiator. Otherwise, the model is just vectorizing everything, searching that vector space, and pulling back related chunks without truly understanding them.

Sunil: But didn't we already have something like that? With one of our products, on an older-generation LLM, we had chain-of-thought reasoning where the model would output its thought process. Isn't that the same thing?

Sam: The key question is whether it was paired with a reward model. Reasoning on its own isn't enough — you need a reward model tied to it. When the model reasons well, you reward it for that. One thing newer models do differently is reward the process, not just the final answer — because a model can otherwise take shortcuts to the right answer without actually reasoning its way there. When you ask "show me every step," and reward it for the steps rather than just the outcome, you get something closer to explainable AI: the process becomes more important than the answer itself.

Sunil: But that's a property of the LLM itself, right? That's not something we can affect as just a user of the model — that's on Anthropic, OpenAI, and the other model providers.

Sam: Increasingly, though, there are agentic systems being built on top — the agent is one component, the LLM is another. These agents can look at the data and actively drive that chain-of-thought reasoning process with the LLM.

Sunil: But that's outside the LLM itself — that's a system, a solution built around the model.

Sam: Right, but the two are becoming tightly coupled rather than treated as separate systems — they need to train these agents specifically for this, and the agent walks through that slower, deliberate reasoning process. That's the current trend. And there's real tension in the industry right now between two different investments: do you build out the agentic, reasoning layer, or do you build richer underlying models? Today's knowledge representations are often reduced to simple binary relationships, but the real world is far more interconnected than that — when you flatten everything into binary relationships, you lose information.

We're drifting into a different area here, but the point is there's real tension in the industry between building agentic reasoning systems versus moving beyond binary relationships toward richer representations — some people are exploring concepts like "hyper-edges," structures that connect multiple entities together rather than just pairs. I won't go into the details, but the takeaway is: reasoning paired with a reward model needs to be there, the LLM needs to be able to explain each step it took, and combining agentic reasoning with richer-than-binary relationships is likely to make a real difference.

Are LLMs Actually "Reasoning"? On Anthropomorphizing Machines

Sunil: We're about 20 minutes in — want to go another 5 or 10 on this, or move to something else?

Sajjad: I'll just add one thing: we tend to anthropomorphize LLMs by attributing human abilities to them — reasoning being one of the big ones. A lot of models today are branded and marketed as "reasoning models," largely because of the quality of their training data and how they're trained on reasoning datasets, plus post-training alignment techniques like RLHF — essentially a reward model built on human feedback. All of these techniques have produced real improvements.

But the fundamental fact remains: these are next-token prediction machines recalling patterns from memory. There's been a lot of published research — including a recent Apple paper — on the actual reasoning, problem-solving, and math abilities of these models, and the field is increasingly converging on the same conclusion: these models are recalling patterns from memory, and they're remarkably good at it — fluent and fast enough that we anthropomorphize them as thinking, reasoning, strategizing, planning. I don't think that's accurate. I think they're recalling from memory.

That said, I don't want to undersell their abilities — they can meaningfully increase the velocity of what you're able to do. My argument is simply that you need enough guardrails and verification in place around them. That's why I see this as fundamentally an architecture problem — we need to solve it architecturally, until, hopefully, a better underlying model comes along.

I recall Andrej Karpathy's thinking on this: it's less about the sheer volume of training data and more about the chains of thought — the trajectories of human thinking that lead to solving a problem. If those human reasoning trajectories could be captured and used to train these models across every domain of human activity — math, science, entertainment, whatever — his argument is that you'd need far less training data and could train these models much more efficiently.

The problem is, we rarely record our reasoning that way. At the end of the day, most written text just states the outcome: "your car had a flat tire, and you fixed it" — not a step-by-step account of how you diagnosed and fixed it. Those detailed trajectories of thinking are far rarer in the training corpus than the outcomes themselves. Karpathy's point was that if we had enough of those trajectories, we wouldn't need nearly as much raw training data.

Sunil: You'd still need a solution system around it, though, right?

Sajjad: Yes — but that's all assuming you're working within current architectures. Look at what Yann LeCun is doing right now — he's concluded we need an entirely new architecture, and he's working on something called JEPA, along with what are being called "world models." There's a lot of work happening there, but we don't know when it will be mature enough to see real-world use — it's still some ways off.

So in the meantime, what we can actually do for business applications is build strong verification mechanisms — both pre-synthesis and post-synthesis.

Sunil: Not just post-verification — pre-synthesis too, if I understood you correctly.

Sajjad: Right, both. And depending on what verification turns up, you might end up surgically rewriting part of an answer, or the whole thing.

Confidence Calibration and Domain Scoping

Sam: Sunil, I want to bring in one more point I think you'll appreciate. Back in the networking world, we used to do something called link discovery — figuring out who's connected to whom at the physical layer. It was a genuinely hard problem. We didn't have a defined protocol for it at first; eventually you get things like LLDP and CDP for neighbor discovery, but originally we had to work it out from raw MAC/ARP-table analysis at the lowest layer, cross-referencing intersections to figure out what was actually connected to what. I wrote one of the earliest link-discovery modules myself — you'd start from that low-level table analysis, add in spanning tree protocol for directionality, and then layer on protocols like LLDP and EDP.

The reason all of that mattered was to determine a confidence level. If multiple independent sources all agreed there was a link, the probability that the link actually existed went up. That's essentially the same idea as confidence calibration for LLMs: models that can express genuine uncertainty instead of answering every question with the same fluent confidence. Can a model reliably say "I'm 70% confident in this answer" instead of bluffing?

Sunil: Seventy percent confident, or whatever the number is.

Sam: Right — that's the open research question. In networking, we could establish confidence levels using multiple independent sources. Can we do the same thing with LLM outputs? I don't know yet.

The second piece: all of this tends to be domain-scoped. We talked about this earlier today — the task space has to be fairly confined for a lot of these techniques to work well. If you have an ontology built for one domain — oil and gas, finance, semiconductors — everything outside that domain looks foreign to the system. So ground truth, in practice, tends to exist (or not) relative to a specific domain, not universally.

Putting it together, there are really five things at play: the grounding layer, self-consistency (chain-of-thought consistency), a critic model that challenges the output on a second pass, confidence calibration, and domain scoping. Those five are all part of the overall picture I've been describing.

Wrap-Up

Sunil: Sam, I want to wrap up — I said half an hour, and we're right at it. Let's pick some of this up in a future episode.

Sam: Sounds good — did you want a quick closing word on confidence calibration before we go?

Sunil: Let's save it for next time — I really want to hold the line at half an hour this week.

Sam: Cool.

Sunil: I said I would, and I don't want to break that promise two weeks in a row. Anything else, guys?

Sajjad: No, we're good.

Sunil: Great — talk to you all soon.

Sajjad: Excellent. Thanks. Bye.