Episode 5 — World Models, Explained: Are Knowledge Graphs Secretly the Same Idea?

A quick note to listeners

Sunil Baliga: Sajjad, Sam — this is Token Drop, week five, so we're officially a month in. Before we dive in, I want to ask our viewers for some feedback if you're willing — leave a comment on what you think, how we could get better, what you'd like to see more of. We started this because we genuinely enjoyed talking about these topics internally and figured it'd be worth sharing more broadly, and we'd love to shape it around what people actually want.

Last week we covered open-source and open-weight LLMs, and we said we'd tackle world models this week — so let's do that. When I first heard the term "world models," I honestly didn't know what it meant. Once I read into it, it seemed like it really should be called "real-world models" — it's AI trained on real-world data, as opposed to an LLM, which is trained on the written word: books, magazine articles, web content. There's a whole other category of data AI can be trained on — real-world physical data, like data from robotics, self-driving cars, or IoT applications. So maybe we start with the basics: what is a world model, really? And then I want to get into how these things are actually trained. Sajjad, what's your take?

Do LLMs actually understand how the world works?

Sajjad Khazipura: Today's LLMs clearly have a lot of knowledge about the world — but do they genuinely understand how the world works? That's very much an open question. A lot of current effort is going into figuring out how to infuse real "world understanding" into these models, and one accepted approach is training on video, in addition to static, text-encoded knowledge. Video captures far more dynamic events than text does — if you hold a ball and let go, it falls and hits the ground. That's a great way to visually demonstrate gravity to a viewer, and the implicit hope is that training a model on enough footage like that will teach it gravity as a concept too.

Given our own backgrounds working on knowledge graphs, I've been turning over a related question: is a knowledge graph itself a kind of world model? We're already trying to model knowledge as thoroughly as we can in a graph — but no matter how hard you try, a knowledge graph captures knowledge as it was recorded at some point in time. It's inherently static, not dynamic.

That led me to an idea: world models are essentially knowledge graphs — static, discrete knowledge, modeled in as much detail as possible — plus a way to introduce real dynamism. Specifically: how does this knowledge evolve if a given event occurs? We might have a full description of a rubber ball, but until you actually push it, you won't know how it deforms. That requires more than knowledge representation — it requires some real reasoning: the ball is made of rubber, rubber is elastic, so pushing it causes deformation. You can build probabilistic graphical models on top of that reasoning layer. So I'd argue world models are really the combination of three things: knowledge graphs, reasoning, and probabilistic graphical models that predict what's likely to happen next.

That's really a roundabout way of describing what the leading frontier labs are attempting today — training a model on large volumes of video, the same way LLMs were trained on text, with the belief that it will eventually learn how the world works simply by observing enough of it. I still think that's primarily a way of infusing a lot of raw knowledge — whether genuine understanding comes out of that process is still an open question.

Sam's technical definition: predicting the next state

Sunil: Sam, what do you think?

Sam Pooni: I think Sajjad nailed it. If you ask people what a world model actually is, there's a lot of loose talk about it — but formally, a world model is a neural network that learns the dynamics of an environment from data, such that given a current state and an action, it can predict the next state. That's the core idea.

Sunil: So it's fundamentally not a language model.

Sam: Right — the real story here is that the combination of deep learning, large-scale data, and modern sequence modeling has made world models dramatically more capable than earlier attempts going back to roughly 2018. Over the past year specifically, they've become one of the hottest areas of active research in AI — Sajjad and I were actually discussing this just a few days ago.

The question is why this suddenly matters so much right now. My honest read is that LLMs have achieved impressive mastery over the digital world, but the physical world is still very much humanity's domain — building an AI that can write a novel is genuinely easier than building one that can fold laundry, navigate a city street, or pour a cup of coffee. World models are the proposed bridge between those two worlds. A classic illustration of an LLM's limits here: if you train a language model purely on simulated New York City taxi trip data, it can give you competent directions across Manhattan right up until it's forced into an unplanned detour — at which point it fails completely, because the model has learned routes, not the actual city. That gap is exactly what world models are meant to close.

From IoT and digital twins to continuous, time-aware models

Sunil: Given you've both been talking about video, it strikes me that world models could apply just as well to IoT — recording digital data from a physical network of sensors, and using that to build something like a digital twin, an AI-powered digital twin.

Sajjad: Exactly — a world model isn't discrete, static knowledge; it models a continuous space. How do you react to events? Acceleration is a great example, motion is a great example, and really any continuous signal — sound, any waveform of any modality. If you have a model that understands those continuous wave patterns, it develops the ability to predict what comes next in that signal.

Sunil: There's clearly a time element running through everything you've described.

Sajjad: Right — they're temporally connected. World models encode a temporal dimension that a purely static, discrete knowledge graph simply doesn't capture on its own. You can put in real effort to index a knowledge graph by time and try to model temporal change within it, but you end up doing a lot of unnatural, forced things to make that work.

That actually raises another question, going back to what Sam said — these are neural networks learning about a signal that's continuous in space. Take video specifically: if you train a model by repeatedly showing it a ball being dropped, and it observes that enough times, the hope is that it learns the underlying concept of gravity. But here's my pushback: we genuinely don't know what these models are actually learning internally. We don't know which features they're picking up on. Is the model learning that objects fall due to gravity — or is it learning something narrower, like "blue objects fall"? If you only ever showed it a blue ball falling, would a red ball generalize correctly, or would the model need to see a red ball, an apple, a pineapple, and everything else, trained enough times on each, before it genuinely learns gravity as a general concept rather than a pattern tied to specific objects? I honestly don't know the answer — and because these models are effectively black boxes, we don't really know what they're learning. It's the same fundamental question we already have with text-based LLMs: how much have they actually learned, and did they learn the right underlying concepts, or are they just very good next-token predictors? What happens when they encounter something out-of-distribution — like the red-ball-versus-blue-ball case? Would they still generalize correctly? We don't know.

How do you train on "unlabeled" data?

Sunil: One thing that confused me while reading about this: these world models are said to be trained on unlabeled data. I don't understand how that's possible — wouldn't you need to label it first to train on it?

Sajjad: This is the same trick used in language models. Take the line "Humpty Dumpty sat on a wall." One clever technique for training these models is to mask out the last word — "wall" — and give the model only "Humpty Dumpty sat on a ___" and ask it to predict the missing word. It might guess "ball," or "couch," or something else entirely. The supervision comes in when you reveal the actual answer: it wasn't a couch, it was "wall" — because you already had the full original sentence; all you did was hide the last word. You train the model on everything preceding that word, have it predict what comes next, and then compare the prediction against the actual answer in an iterative loop. No human is manually labeling anything — the label was implicitly already there in the original text. It's effectively a clever trick for generating supervised training data without real supervision.

I'd expect world models to use an analogous trick with video: frame-by-frame training. You drop a red ball and record many frames until it hits the ground. You mask out the final frame — the moment of impact — and ask the model to predict it. If it gets it wrong, you already have the ground truth (the actual final frame), since you recorded the whole sequence. So you can mask out parts of any input stream, have the model predict the missing piece, then reveal what was actually masked and let the model check itself — an iterative loop that looks like unsupervised training on the surface, but is really a clever form of self-supervision underneath.

Digital twins: the original "world model"

Sunil: This reminds me of digital twins from a previous company we worked at — genuinely valuable, but very difficult to build, especially for complex manufacturing systems. It seems like these world models are really an AI-era version of the old digital twin concept.

Sajjad: Exactly. The IoT team we used to work with spent much of their careers building physics models — modeling how fluids flow through pipes in a factory, applying real physics and hydraulics knowledge, deriving the governing equations, and writing the programs to model those flows. It was incredibly hard, until neural networks came along and, as Sam said, could be trained within sufficiently narrow domains — with the right safeguards to make sure predictions stayed within the distribution the model was actually trained on, since venturing outside that distribution risks incorrect predictions.

Sam: Exactly what I was getting at. Digital twins actually originate from engineering — NASA used early versions of the concept for the Apollo program. The term later took off in manufacturing and is now everywhere, from BMW factories to patient-specific models in healthcare. It's not a new idea; it's been part of the ecosystem for a long time.

A twin is tied to a specific physical instance — this exact turbine, in this exact plant — fed live by its own sensors. It typically combines a CAD or physics-based model (the structural backbone) with a live data stream (the "life" of the system). You can use it to monitor health, run what-if scenarios — "what happens if I increase the load 15% and schedule maintenance differently?" A modern twin increasingly adds a learned-dynamics component on top of the pure physics model, because pure physics simulation tends to be slow and brittle — which is basically what Sajjad's describing. A neural surrogate trained on the twin's own history can predict faster and capture phenomena the physics model never explicitly encoded. That's exactly where digital twin plus world model becomes genuinely powerful. And the reverse is also true: a world model trained inside a high-fidelity digital twin is a great way to generate cheap, safe synthetic training data for robots, without crashing real ones. People used to generate synthetic data with GANs for this kind of thing; a digital-twin-plus-world-model combination can now deliver that same kind of data for robotics.

Sunil: So if I understand correctly, you'd use a world model inside a digital twin?

Sam: Right — think of the digital twin as the live dashboard sitting on top of the physics, and the world model as the intuition layered on top of that. That's really the cleanest way to classify the relationship.

Real-world example: modeling supply chains and semiconductor fabrication

Sajjad: We've actually been able to approximate this with simpler technology too. We modeled supply chains using knowledge graphs — static knowledge of the supply chain structure, and on top of that, cause-and-effect modeling: what happens if a supplier goes down, or a specific logistics link between two suppliers gets disrupted — how does that ripple through the flow of materials across the whole chain? From there we built more sophisticated Bayesian models on top to model risk specifically. We applied this for a leading European automotive company and for a major US phone company — modeling supply chain uncertainty with Bayesian network models and tracing how risk propagated through the system. To some degree, that is a digital twin of the real-world supply chain — and you could reasonably argue it's also a model of the "world" of that company's supply chain domain specifically.

Knowledge graphs, in their most evolved form — combined with reasoning, uncertainty modeling, causality, and probabilistic reasoning capabilities — genuinely start approximating a world model. And you gain something valuable in the process: precise, explicit control over how the model should react to different events, letting you simulate failure scenarios and what-if conditions directly.

One of our customers, in my past work, actually applied this concept to model an entire semiconductor fabrication process — a genuinely complex, multi-step manufacturing pipeline. Every step was modeled as a node in a knowledge graph, forming a large, complex network capturing the flow of materials and processing across the whole fab. That gave them something close to a digital twin of their manufacturing process — they could report exactly what stage a given wafer was in, and simulate what would happen if a specific step were disrupted, including how to reroute processing if alternative paths existed. Those were exactly the kind of what-if scenarios they were trying to model and report on.

Knowledge graphs as narrow-domain world models

Sunil: So is it fair to say the knowledge graph gives you the structural model of the real world, and a world model adds the temporal dimension that's otherwise missing?

Sajjad: What I'd say is: knowledge graphs, extended with additional capabilities — reasoning, uncertainty, causality — can approximate world models within sufficiently narrow domains. We did it for supply chain, we did it for manufacturing processes. As long as you constrain the application to a specific, well-defined domain, you're effectively building a world model for that domain — a world model for supply chain, a world model for semiconductor manufacturing.

That's a genuinely different path from the classical neural-network approach some of the frontier labs are pursuing. Yann LeCun, for instance, left Meta and joined AMI Labs, where he's now working on world models. Fei-Fei Li, formerly a Stanford professor, has similarly launched a company — often referred to as World Labs — pursuing this from another angle. My understanding is they're training models directly from video, with the goal that models trained this way could eventually be embodied inside robots — letting a robot walk into an unfamiliar kitchen and, say, make you an omelet. That's the holy grail they're chasing.

There's a real counterargument, though: how much training data does that approach genuinely require, and is it sufficient? Or can you get similar value more deterministically, within narrower, well-defined domains? It's the same classical conundrum we already have in the text world — LLMs versus the kind of graph-based approach we use — and exactly the same tradeoff shows up here too. The graph-based path is more deterministic, controllable, and transparent, but limited to narrower domains you've explicitly modeled. The classical neural-network path learns much faster with far less manual modeling effort, but costs more in time, money, and compute, and remains comparatively opaque.

Combining both approaches

Sajjad: Realistically, you can probably combine the two. Graph-based, knowledge-driven models can serve as guardrails for neural-network-based world models.

Sunil: Grounding, essentially.

Sajjad: Exactly — the same approach we already use today. We use neural networks, but we also use graphs, and we combine the two as effectively as we can to build trustworthy systems. You could apply that exact same philosophy to today's definition of real-world models and graphs together.

Wrap-up: the twin state as the bridge

Sunil: I think we're at time. Sam, Sajjad, anything else to add?

Sam: Just that I fully agree with what Sajjad laid out — grounding the world model in a knowledge graph is exactly right, and the digital twin's "state" is really the cleanest bridge between the two right now. A twin's state vector can serve simultaneously as the knowledge graph's live, runtime instantiation and as the world model's input. Closing the loop means that when the world model predicts a failure, that prediction should itself become a temporary node or edge in the knowledge graph — triggering further reasoning and updating the twin's risk state. Most production systems today only implement pieces of that full loop; getting all of those pieces working together is really where this is headed. It's an active area of research right now, especially around how the knowledge graph is represented, how the digital twin functions as the live dashboard, and how the world model contributes the predictive layer on top of all of it.

Sunil: Great discussion, guys — I think we should just start calling these "real-world models." Thanks, and we'll talk soon.

Sajjad: Sounds good. Thanks — bye-bye.