Episode 19 — The LLM Is Not the Library, It's the Reader

All opinions expressed are those of the individuals themselves, not necessarily of any company they work for.

Sunil Baliga: Welcome, everyone, to Token Drop. Before we get started — if you enjoy the podcast, please subscribe on our podcast page, YouTube, Spotify, or Apple Podcasts. You'll get notified automatically when a new episode is available.

Today I want to talk about something I saw in your newsletter, Sam — you wrote that the LLM is not the library, but the reader. It's not the book; it's the person reading the book. I tried explaining that to somebody this week, so let me tell you what I told them, and you two grade me. This is my test for the week.

What I explained is: the LLM has already been trained on a large body of public information — books, internet content, and so on — and that knowledge has been burned into the model through its weights. It has a lot of knowledge inherently. Now, if I give it new knowledge it hasn't been trained on — a new document, say a potential contract from a customer — that isn't in its DNA. So the new knowledge gets stored somewhere else, typically in a vector database. There's a separation between trained knowledge, stored within the model itself, and new knowledge, stored outside of it. And the real question becomes: what's the best way to store that new knowledge? You can't retrain an LLM for every new document — that's just not realistic. How did I do?

Sam Pooni: First, let me give credit where it's due — to Sajjad. Back in January 2026, I was deep into writing articles about LLM internals and reasoning, and when I met him, he asked me one question: do you think knowledge should be represented within LLMs? Is that the right place for it? He was exploring Nyaya and other representation schemes at the time and talked to me at length about it. So the ideas came from him.

The newsletter article also touches on something called knowledge editing — which also came up in our conversations. The idea is you take specific weights and modify them to change a fact the model holds. But there are downstream effects, and research papers have noted that once the number of edits becomes large, you can see something like a 50% loss in the model's accuracy. All I did was collect the evidence. So Sajjad, take it from here.

Three Ways to Give an LLM New Knowledge

Sajjad Khazipura: He's just being humble. Long story short: we use LLMs that were pre-trained by somebody else, and we expect them to have been trained on all the skills and knowledge we plan to deploy them for. As Sunil said, if you have a new legal document, the LLM almost certainly hasn't seen it — it's your legal document, sitting in your private storage. There's no chance the model has encountered it. So if you expect it to know that knowledge and start asking questions about it, obviously it won't.

There are a few things you can do. The first is in-context learning: you provide the document as part of your prompt and ask your questions. It's like saying, "LLM, here's a document, and here are my questions about it — would you kindly answer them?" That gets the model to absorb the knowledge and answer questions about it in real time.

The second is supervised fine-tuning: if you have many such documents, you fine-tune the LLM on them and hope you get the answers you want.

The third is additional pre-training — continual learning. You can genuinely get an LLM to learn incremental new knowledge this way, but you have to go through a full pre-training cycle, which requires data science skills and the infrastructure to train the model — and we often don't even know how big the model is. So in most practical cases, you provide the document as part of the context and expect the LLM to answer.

That usually works well. But sometimes, despite being fed exactly the right information, the LLM still gets it wrong — because it's still generating predictive answers. That's why even RAG can fail; there are many RAG failure modes, and this is one of them. GraphRAG and its variants try to at least make retrieval precise and accurate, so you can hand the LLM precise, accurate knowledge along with the prompt and hope for the right answers. Even so, you still sometimes get incorrect answers — which is where verification technologies like our ClaimGuard come in: checking the answer against ground truth, and when something's wrong, going back to surgically correct it, rewrite it, or recalibrate the degree of assertiveness in the answer.

So yes — LLMs can store some knowledge and process some knowledge. But the answers are not guaranteed to be correct.

The Map Analogy: Transient Knowledge in Context

Sunil: But that storing of new knowledge, Sajjad — it's actually not stored in the LLM, right? It's stored outside.

Sajjad: In this case, yes — it's transient. Think of it like this: you're walking down a road reading a map, and the map is hard to read. So you ask a stranger on the road to read part of the map and tell you exactly where you are. They can do that — but they don't know where you came from, and they don't know where you're going. The LLM is like that stranger. In that moment, you're handing it a document and saying, "read this and help me answer these questions." But it might not understand the full context — the history of where you came from, the trajectory of where you're headed.

Does a Frontier Model's Knowledge Help with a Brand-New Document?

Sunil: In that context — the big frontier models have tremendous knowledge and much better understanding. But for a new document, does that even matter? The new knowledge isn't what they were trained on.

Sajjad: It's a nuanced answer. If it's a very primitive LLM that hasn't been trained on much data, and you hand it a legal document expecting it to read and answer questions, it may not do well. But if it's a genuinely frontier model, well-trained on a large corpus of legal documents, then your legal document is just a derivative use case of pre-existing knowledge in that domain — the foundational ability to interpret and process that kind of legal content already exists. In that case, it will likely do better than a model that never saw legal text.

So it's a spectrum. And for a common user, here's the problem: when you're handed an LLM, you don't know what content it was trained on. Has it ever seen legal documents like yours? You'll never know — it was trained on millions of documents, and nobody's going to enumerate them. So there's no guarantee the LLM actually has the requisite knowledge.

Sunil: It's like if I opened a math book to a triple integral. Back in my engineering days I knew how to do those — I probably couldn't do a single integral today, but given enough time, I could read it and understand it. That's what I take you to be saying: the LLM knows how to read that math book and understand what a triple integral is, because it's been through algebra, calculus, single and double integrals. But if a model doesn't even know algebra to begin with, it has no way to understand what the knowledge in that book means.

Sajjad: Indeed — and it's a great example. You're an engineering graduate who studied a lot of math; confronted with a triple integral, you'd recognize it and might even solve it. Someone who studied the arts, confronting a triple integral for the first time, would be confounded. And then there's the third case — you were a math person, but you haven't practiced in a long, long time, like some of us. You encounter the triple integral and have some vague, fond memories —

Sunil: Vague is right. Absolutely.

Sajjad: — and you might not know how to solve it anymore. All of those cases are true for models, too.

PEFT and LoRA: You Don't Have to Retrain Everything — But Problems Remain

Sam: One thing worth adding, because it's interesting — you mentioned needing to train models completely. That's not exactly true. There's an approach called PEFT — Parameter-Efficient Fine-Tuning — which means adapting a large model without updating all of its weights. And there's LoRA, which we can go deep on another time.

But here's the point: even these approaches have real deficiencies. There's weak knowledge injection — Sajjad can attest to it. Catastrophic forgetting still happens, even with PEFT. And there's hyperparameter sensitivity. These bottlenecks haven't gone away. PEFT and LoRA adapters make fine-tuning much faster and cheaper, but the fundamental problems persist.

[Sajjad's connection dropped briefly at this point; Sam continued solo.]

What Is a Knowledge Representation? Five Roles from a 1993 Paper

Sam: While Sajjad rejoins, let me give my take — and get a little philosophical about it. Every intelligent system faces the same problem before it can reason about anything: the world has to be encoded in some form the system can manipulate. Encoding is knowledge representation. Get it right, and reasoning becomes tractable. Get it wrong, and no amount of compute saves you.

This is an old problem. It predates neural networks, predates databases — in some sense it predates computing itself. Writing, notation, and maps are all knowledge representations. What changes with machines is that the representation must be precise enough for mechanical manipulation. A human reader fills gaps; a machine does not. If you read something and form a slightly wrong picture, you can self-correct. Machines need precision.

So what is a representation, fundamentally? It's a stand-in — a set of symbols or structures that correspond to things in the world, plus rules for manipulating those symbols such that the manipulations correspond to valid inferences about the world. This is, incidentally, what classical Indian philosophy — the Nyaya school — was grappling with: symbols corresponding to things in the world, and rules for valid inference over them.

Davis, Shrobe, and Szolovits made this precise in a 1993 paper that still holds up. A representation is simultaneously five things: a surrogate for the world; a set of ontological commitments about what exists; a theory of reasoning; a medium of computation; and a language humans use to tell machines things. And those five roles pull against each other. A representation that's easy for humans to author — natural language — is hard to reason over mechanically. A representation that supports fast, sound inference — first-order logic — is painful to author and brittle at the edges. Every practical system lives somewhere on that trade-off surface, and most of the interesting engineering decisions are about where.

Sunil: If I may interrupt — I think the reason AI has taken off is exactly that it works with natural language. Expecting humans to learn domain-specific languages is very difficult. You can just speak to it naturally, like I do with you.

Sam: Exactly — and that's also why it's so hard. Natural language carries ambiguity and all kinds of complexity. And there's another problem: some facts live in the document, but the context around them doesn't. Suppose we agree on something, and later supersede it. The reason for superseding it won't be in the document unless someone writes an addendum. So if you take the old representation at face value, temporally, it's simply wrong — it's been superseded. And even if you know it was superseded, do you know why? That's missing. That surrounding context is what makes representation so much harder than it looks.

Nobody Knows How Knowledge Is Stored in the Weights

Sajjad: (rejoining) You're right — there are these two schools of thought. In the symbolic school, everything is clearly specified, and you operate on precisely defined symbols. In neural networks, as you train, the weights get updated — and for the same concept, you might be updating many different weights. Train on a new concept, and you might update some of the same weights plus additional ones.

How that knowledge is actually stored within the weights is still a subject of research. Nobody knows. If I show a network pictures of cats zillions of times and train it to recognize cats, and then start training it on dogs — am I updating the exact same parameters, a completely disjoint set, or something overlapping? Chances are there's substantial overlap. But how the information is stored is not well understood. And this matters enormously: if we did understand how knowledge is stored, it would become easy to go in and edit it — to clean up the mistakes and gaps that cause hallucination. We can't edit that knowledge because we don't know how it's stored. It generalizes, it diffuses across the weights as training continues, some people call the results "emergent behavior" — but none of it is scientifically understood. If it were, it would be much, much easier to fix models and make them behave exactly as expected.

Knowing-How vs. Knowing-That: What LLMs Are Actually Good At

Sunil: Here's a question — what's the difference between knowledge and information? We use both terms. Are they the same?

Sam: And add a third: intelligence. What is intelligence, really? But here's an observation that gets at it — from experience and from the literature: LLMs are very bad at remembering who you are. This is Sunil, that is Sajjad, that is Sam — they're bad at that. But they're very good at doing things — how to ride a bicycle, how to drive a car. They encapsulate the how.

This is what we'd been failing at for decades. Think of image processing in the '90s — we tried if-then-else rules everywhere, and we couldn't do it. We didn't have a good representation until neural networks came along and captured the how of doing things. That's why you can say "give me a painting in the style of Michelangelo" or "a symphony like Bach" and it can generate it. The LLM can do that because it understands how to do things — as opposed to remembering facts. Which means the fact layer has to live outside the model. And that's the whole conversation: is the LLM itself a good place to hold facts?

Neural networks hold knowledge differently — not as inspectable symbols, but as weights: billions of parameters shaped by training data. Nothing in the network is labeled "Paris is the capital of France" — yet the network answers the question. The knowledge is distributed, entangled, and implicit. And this representation has properties the symbolic tradition spent decades failing to achieve: it degrades gracefully, and it handles similarity and analogy natively. "King minus man plus woman" landing near "queen" was a genuine shock in 2013 — it showed relational structure emerging from distributed statistics, without anyone encoding it. It absorbs the exception-riddled texture of real language and real perception — exactly where logic broke.

But its weaknesses mirror its strengths. You cannot point at a fact and update it. You cannot audit why an answer came out. Knowledge is frozen at training time. And confabulation is not a bug to be patched — it's a direct consequence of storing knowledge as statistical associations. The network reconstructs plausible content rather than retrieving stored content — and plausible is not the same as true. That is exactly where hallucination comes from. And that's the premise behind the neuro-symbolic direction Sajjad has been driving — not dogmatically; if better representations emerge, we'll look at them. But what we have today is property graphs, knowledge graphs, and 3D graphs, and we're still exploring the frontier of representations.

Errors of Omission vs. Errors of Commission

Sunil: Good time to hand it to Sajjad — we're almost out of time.

Sajjad: Neural networks have tremendous advantages — you can train them in largely unsupervised fashion and expect them to hold knowledge you can then retrieve. The only question is for enterprise use cases — and only those — where you need determinism and explainability. That's where they fall short. So what we're trying to do — and I think the rest of the world is converging on the same thing — is combine the two methods, because each has its own drawbacks.

In a knowledge graph, you can only store a finite amount of knowledge. There will obviously be gaps — but at least you know what the gaps are. Those are errors of omission: you've intentionally omitted some knowledge, and when someone asks about it, you simply can't answer — visibly.

Whereas with an LLM, you don't know what's stored inside. It doesn't come with a nutrition label. When you buy a jar of peanut butter, the label tells you exactly what's in it — the fat, the minerals, every ingredient, clearly listed. LLMs don't come with such labels, so you don't know what they contain — which makes it hard to decide whether to use one for a given application, or whether it will hallucinate. The frontier labs are trying to mitigate that by attempting to discover every piece of digital data in existence and train on all of it — an enormously ambitious program, billions of dollars in data centers, in the hope of training an LLM on every possible nugget of information in the universe. And wherever they inevitably fall short, you get errors of commission: the model silently fails, and you're not likely to know when.

With graphs, you have errors of omission. With LLMs, you have errors of commission. The real answer is combining the best of the two traditions to address problems you can actually wrap your mind around.

Wrap-Up

Sunil: I think we're out of time — thank you both.

Sam: Another great discussion — and we'd love for viewers to subscribe to the channel, and to join us as guests if they have something to contribute.

Sunil: Absolutely — we love the episodes with external guests. Love you guys too, but we like having external voices join us. And please hit that like button as well. Thank you, everyone.

Sajjad: Thank you.