Episode 18 — Small Language Models: Great Translators, Terrible Knowledge Stores

All opinions expressed are those of the individuals themselves, not necessarily of any company they work for.

This Week's Topic: The Small Language Model Pitch

Sunil Baliga: Token Drop, episode 18 — and since I'm traveling with limited bandwidth, I'm going audio-only this week.

What I'd like to talk about is something I've heard a few times recently: small language models. The pitch goes like this — to give AI domain knowledge, companies believe they can take a small language model and train it with domain knowledge from a customer or an industry, and that will infuse the model with what it needs to do a better job in whatever application they're using it for.

For me, that doesn't quite work, for a couple of reasons. First, language changes — knowledge changes. You spend all this time training, and when the knowledge changes, which it does pretty consistently, you have to retrain your model. It doesn't matter whether it's small or large. Second, you still have hallucinations associated with a language model — again, small or large. So let's talk about small language models this week: the pros, the cons, where they actually fit. Sajjad, what do you think?

Same Genetic Architecture, Same Problems

Sajjad Khazipura: You and I have been in several meetings where participants told us they're building domain-specific small language models, and they were pretty glib about it serving their needs. Our argument has always been that these models share the same genetic architecture as large language models — and therefore suffer the same consequences: errors and hallucinations.

They go through the exact same training process. During training, knowledge gets stored in a lossy, semantically compressed form. During retrieval, the process is probabilistic. So you have potential loss of knowledge as it's stored, and potential loss as it's retrieved — and the compounding effect of those two is errors and hallucinations. You're spot on that these models suffer from exactly the same challenges large language models do. So proceed at your own risk.

Two Cautionary Tales: BloombergGPT and Meta's Galactica

Sajjad: That brings to mind two very prominent efforts at building domain-specific language models. Bloomberg set out to build BloombergGPT, spending on the order of $10 million to train a model on their financial data. A couple of years down the line, as you've likely read in the press, Bloomberg wound that effort down — they simply couldn't control the quality, along with other challenges.

Similarly, Meta built Galactica — a scientific model trained on research and mathematics. The vision was something like an Oracle of Delphi for science: answering every scientific question, helping scientists and engineers innovate faster. They had to withdraw that model within days of its launch because it was hallucinating so badly — they pulled it to protect themselves from reputational damage.

Those are well-known industry examples where domain-specific language models failed. That's not to say they don't have benefits — we'll get to those — but Sam, weigh in.

Three Jobs, Two of Them Done Badly

Sam Pooni: Sajjad has deep experience building small models very early in the cycle, so his perspective here is invaluable. From my angle: what do large language models do, anyway? They have three jobs. They understand the question, they remember the facts, and they reason to an answer.

They're genuinely good at the first. As we covered in our earlier episode on the "lost in the middle" problem, they're unreliable at the second. And they're inconsistent at the third. That's the problem we face with large language models — and small models inherit it, because a small model still handles language the same way.

The alternative architecture people are now pitching: the small model handles language, a knowledge graph holds the facts, and a symbolic layer does the reasoning. Each part becomes testable, cheap, and auditable. The thing that made the system untrustworthy — the model inventing facts — is designed out instead of filtered out. That's the general pitch from people experimenting seriously with SLMs.

A Brief History: How Small Models Grew Up

Sam: It's worth touching on when SLMs really gained traction. Around 2024, they stopped being research-reserved and became something you could buy, ship, and put on a device — Gemma and Gemma 2, Phi-3, Llama 3.2 in 1B and 3B variants, the Qwen 2 and 2.5 families. A whole wave of them arrived.

Then in 2025, reasoning came to small models, largely through distillation — DeepSeek-R1 and, subsequently, models like Phi-4-mini and its reasoning variants. These reasoning models genuinely arrived small. And in 2026, the small models have kept growing up: Qwen 3.5, Gemma 4, the GPT-5.4 Mini and Nano variants. There's now a full spectrum.

Sajjad: You said reasoning models are built by distillation — what do you mean by that?

Sam: When we talk about small models, there are really five distinct ideas that make them possible. Distillation is the first — a small model learns from a large model's full output.

Sajjad: So that's constructing small models from bigger models. That's one path. But there's another, especially for enterprises — our classic customers. They build models not primarily because big models are expensive to deploy, though that can be a factor. The more motivating reason is that they hold proprietary knowledge — data that was never available to the large language models. So they set out to build small language models from scratch, not by distilling anything down. Many of the enterprises we've spoken with over the past few weeks are on exactly that path. And there's a third route: within a large mixture-of-experts model, you're effectively already looking at multiple smaller expert modules.

Sam: Correct — and that rounds out the five ideas: distillation; the Chinchilla insight (most models were under-trained — train longer on more data at a given size); quantization down to 4-bit weights; curated and synthetic training data, which is what the Phi results demonstrated; and sparse mixture-of-experts, which you just described.

What Actually Counts as a "Small" Language Model?

Sunil: I have a question — Sam, you called Llama 3.2 a small language model, which surprised me, because I always thought of Llama as a large language model. So what actually is a small language model? Is there a definition?

Sajjad: It depends on when you ask the question. If you'd asked in 2017, when "Attention Is All You Need" was published, the model Google open-sourced was BERT — the biggest model available to us at the time, and quite an appealing choice in those days. As parameter counts have grown, the definition of "small" has shifted with them. These days, a small language model typically ranges from about 1 billion to maybe 8 billion parameters — roughly, something manageable enough to run entirely on a single GPU. Beyond that, you're into territory requiring multiple GPUs to serve inference. But don't take "fits on one GPU" as a formal definition — it's a crude rule of thumb.

Sam: To put numbers on how much the goalposts have moved: in 2019, DistilBERT was about 66 million parameters — roughly one-fifth of BERT-large. By 2021, "small" meant 1–2 billion — about one-hundredth of GPT-3. By 2023, 7 billion — roughly one twenty-fifth of the frontier. By 2024, 1–3 billion was about one three-hundredth of a frontier model. And now, under 4 billion parameters — or roughly 4 billion active parameters in a mixture-of-experts model — is about one one-thousandth of the frontier. The absolute number keeps moving. There's no stable definition.

Sunil: So it's time-based — but it seems like no matter when you answer the question, the problems remain: you still have to train it, knowledge still changes, and it still hallucinates.

A Step Up from Fine-Tuning — But the Risk Remains

Sajjad: Indeed — the genetic makeup of the model prevents it from being accurate and precise all the time. Like any large language model, it's prone to the same problems.

That said, I do think building a small language model is a step up from supervised fine-tuning. Some companies we've spoken to — you might recall this, Sunil — say "we'll bring domain knowledge into the LLM through supervised fine-tuning." There are real risks there, like catastrophic forgetting: you're trying to teach the model new skills and new domain knowledge, and it often just doesn't work out. What the research literature increasingly suggests is that a model should have its priors — its knowledge — built in through pre-training, and fine-tuning should then adapt it to specific use cases. So compared to fine-tuning an existing model, building a small model from scratch on your data is a step up. But even here, the hallucinatory risk remains.

What We Learned Pre-Training BERT in 2019: Use the Model to Build the Graph

Sajjad: Which brings me to something we did long ago, Sunil — at our old company, in 2019, when BERT was relatively new. We had a customer engagement to build a chemical and material science knowledge discovery platform. We took BERT — the biggest model of its day — and pre-trained it on chemical and material science data. We hand-annotated documents using Stanford's Protégé tool, built and refined ontologies for chemistry and material science, and used that annotated data to fine-tune the model.

But here's the key part: we didn't put BERT in the direct query pathway. We used it to extract knowledge from documents and construct a graph — and then served information from the graph.

What we did in 2019–2021 still holds today. If you choose to build a small language model because you want to train it on your proprietary knowledge — more power to you. But use it wisely. Our suggestion: use it for extracting information and constructing a graph. Use it for domain-specific tokenization, for creating domain-specific embeddings. All of those genuinely help in constructing information retrieval pipelines. So there's real merit to small language models — you can accomplish quite a bit — provided you're judicious about how you use them.

The Model Should Stop Being the Knowledge Store

Sam: Totally — and this is my fundamental position: the model should stop being the knowledge store. I don't believe it should be one. It should become, at best, a translator — two directions: question in, structured query out; structured result in, plain English out.

Once you frame it that way, translation becomes a narrow, learnable task — and it fits comfortably in about 3 to 8 billion parameters. You don't need a huge model for it. A narrow task means you can actually test it; the blast radius is small. And the practical consequences are significant: it runs on your own hardware, there's no per-token bill scaling with usage, latency drops to tens of milliseconds — I've experimented with this directly — and your data never leaves. Fine-tuning becomes cheap and boring, which is exactly what you want.

You're not asking the model to be smart. You're asking it to be a good translator. Those are two different hires — they're not the same job. Yes, small models are worse at the job you took away from them — but on the job that's left, the gap mostly closes. That's the point.

The Graph Is the Memory

Sam: So where is the memory kept? The graph. The knowledge graph holds the facts, the shape of those facts, and where each one came from. Every fact is something you can point to — a source, a date, an answer. Update one fact, update one edge — no retraining. Absence is explicit. The schema is the contract.

Model weights are like a memory of a conversation; the graph is a signed document. That's the difference. And here's what people get wrong: many teams build the graph as a mere retrieval index — a bag of notes to pull context from. That's a filing cabinet. The graph earns its cost when queries actually execute against it — not when it merely feeds text into a prompt.

Then reasoning splits into the deterministic and the fuzzy. "Does this configuration violate a policy?" — that's mostly deterministic: rules, constraints, graph traversal, solvers. "Is 'Acme Corporation' the same as 'ACME Corp'?" — that's fuzzy, and that's where the model earns its keep. Two different jobs.

Deterministic reasoning must be replayable — same question, same answer, every time, with the steps shown. A model can't promise that — not because it's weak, but because that's simply not what sampling does; it's probabilistic by nature. Symbolic reasoning explains itself for free — the chain of rules is the explanation; you don't need a separate explainability product. If the answer has to be the same tomorrow, don't let the model decide it — let a symbolic layer decide it.

The Economics: Retraining Costs and Entitlements

Sajjad: Another thing Sunil raised at the top that we haven't fully addressed: knowledge changes, so you have to rebuild the model at some point. In Bloomberg's case — they published a paper, so this is public — they spent roughly $10 million on training. If knowledge changes enough that you need to retrain quarterly, that's another $10 million each time — something like $40 million a year. Very few organizations have that kind of budget. Those that do, more power to them; those that don't need more creative mechanisms.

And there's another enterprise consideration: entitlements. Different people have different levels of access to different documents and knowledge. You cannot burn access-control knowledge into a language model's weights — but you can implement it readily on a graph. If Sunil has access to information the rest of us shouldn't see, that level of authorization and entitlement management is straightforward on a graph, and much less so inside a small language model.

Sam: Exactly. And it's worth highlighting how the framing has shifted over time. It used to be: small means lossy — the student was always a compromised copy of the teacher. That framing held for years.

Then it turned out most existing models were under-trained — there was headroom at every size — so small models stopped being compressed big models and became models trained correctly from scratch. And as you rightly pointed out, the market for small models was never researchers economizing. It was privacy, latency, and cost — three things a cloud API cannot fix. The trade-off has moved: small models no longer give up capability across the board; they give up breadth — the long tail.

But Are They Actually Reasoning?

Sajjad: Sam, I want to challenge you on something — you've used the word "reasoning" a few times. Whether these models are really reasoning is a big open question. Subbarao Kambhampati has published extensively on this — years of PhD-level research with multiple students — and the conclusion they've arrived at is that these models are not reasoning. They appear to reason; they emit intermediate tokens, but when you assemble those tokens, you may not even be able to reconstruct a coherent thought process. It's not scientific reasoning — it's recall from memory. If you want true reasoning, you have to take it offline and use dedicated machinery.

Sam: And that's a genuinely interesting point — are they really reasoning?

Sajjad: That's exactly why I'm hesitant to use words like "emergent" and "reasoning" — they sound like magical properties. Where is the wiring in the model that supports reasoning? We haven't seen it. We haven't seen the wiring for mathematics either. The jury is still very much out.

Sam: So what does it actually mean when someone says you can "teach a small model to work through your problem"? Maybe it learns the steps; maybe it memorizes them — we genuinely don't know. And isn't real reasoning inherently a multi-step, iterative process — ask, evaluate, refine, repeat?

Sajjad: For those who subscribe to this view, there's a perspective Andrej Karpathy expressed at one point: if you had well-documented human trajectories of thought — the step-by-step process of working through a problem — you could potentially train models on far less data and achieve something closer to real reasoning. But most of the data these models are trained on — internet data — describes outcomes, not process. "I couldn't run a mile under six minutes, so I trained hard, and now I've achieved my goal." That's what people put on the internet: the outcome, rarely the process that produced it. If sufficient quantities of process-level data existed, there's a decent chance models could learn those trajectories of thought.

Sam: And that connects to a point you and Sunil both raised: with large models, when there's a discrepancy in the knowledge, we don't know where to fix it. The only remedy is full retraining — enormously costly for foundation models. SLMs dissolve that somewhat, because the smaller scale makes retraining feasible — and that's a legitimate reason for the affinity people have toward small language models. They have a place. And that place, in my opinion, is as translators — with the memory kept in the graph.

The Architecture in One Sentence: A Fluent Front-End to Machinery in the Back

Sajjad: Sunil and I have been trying to find the right language to describe how these models fit into this architecture. One framing we've landed on: use language models for the language skills — the linguistics. When a query comes in, can you split the language part of the query from the knowledge part? Then offload the knowledge and the mathematics from the LLM entirely — knowledge discovery, retrieval, mathematics, reasoning, logic — perform all of it on off-board engines, reconstitute the answer, and let the LLM generate the final response.

In that sense — to use your word, Sam — the model is literally a fluent language front-end to the machinery in the back end that does the knowledge processing, mathematics, and logic.

Wrap-Up

Sunil: I think we're out of time — I've turned my video back on to say goodbye properly.

Sam: I love it — this was one of the best conversations I've had with you both.

Sunil: And my internet didn't crash, so I'm happy. We'll talk to you all later.

Sajjad: Alright — thank you.