Episode 26 — What Does “Deterministic AI” Actually Mean? Graphs, Jev, and the Limits of Constrained Choice

All opinions expressed are those of the individuals themselves, not necessarily of any company they work for.

Introducing Nav Mathur, Co-Founder of Cerebrix

Sunil Baliga: Okay, welcome, Token Drop episode 26 — half a year, and very happy that we've reached this stage.

Sunil: If our subscribers, or people watching this, like what they see, please subscribe. And I had another question this week — would we take a request? Yeah, if you've got an idea you'd like us to talk about, send it over and we'll talk about it.

Sunil: So today we have a very special guest, Nav Mathur. Nav is co-founder of a Silicon Valley AI company, Cerebrix. They're doing deterministic AI with graphs and agentic reasoning for supply chains. Before that, he was almost nine years at Neo4j, and before that, at Wipro, where actually Sajjad and he know each other. I had the fortune to meet Nav just recently.

Sunil: Thank you, Nav, for joining us. So let me kick it off — you talk about this deterministic enterprise AI context layer using graphs and other approaches. A couple of questions on that: what do you mean by deterministic, and how do determinism and graphs go together?

Why 95% Accuracy Per Step Becomes 59% After 10 Hops

Nav Mathur: Yeah, hi Sunil, thanks for that introduction. This is a big topic right now, because the LLMs, by their nature, are probabilistic, right? They have a 95% probability of success — that's the claim most of these LLMs make.

Nav: 95% sounds really great, but once you start chaining 10 hops or more, that drops to something like 59%. From an enterprise standpoint, if you get good results 59% of the time, I don't think any CEO — or even any director — is going to accept that. That's the inherent problem with LLMs.

The Harness (or Sidecar): Reminding the LLM What It's Doing

Nav: Now, you need some sort of a harness to remind the LLM what you're working on, what the last thing it did was, and so on. Every time it's processing, it needs a constraint built into it — that's the harness, or a sidecar, that you need for this determinism to happen.

Constraining LLMs to Approved Enterprise Tools and Data

Nav: Even despite that, there are several other things that prevent a deterministic output. In the enterprise, you want to operate on enterprise data — not take something randomly off the internet, mash it up with your enterprise data, and present the results, because you're not sure what the LLM is picking up from the internet, whether it's valid or invalid.

Nav: The other way to complement and achieve determinism is to have pre-built tools, and tell the LLM it's only to operate within those tools — and those tools operate on enterprise data, or approved external data the enterprise has already approved. Public APIs they've used and tested before, or contracted third-party data sources, and so on. Together, that ensures the LLM isn't making things up — because it loves to answer every question, even when it may not actually know the answer deterministically, or in a 100% accurate way.

The Upstream/Downstream Part ID Problem: How Confusion Becomes “Disastrous Hallucination”

Nav: The third element is when you have a multi-step process. A good example: if you look at parts, there could be multiple parts in a hierarchy — you could have an SoC, and the SoC has several parts inside it, and so on. You could have an upstream part and a downstream part, and you're trying to connect the upstream part to the downstream part, and find maybe the supplier of the downstream part.

Nav: Once you have a multi-step process, there's a part ID, and the LLM is only aware of the part ID. It can totally confuse the upstream part ID with the downstream part ID as it goes further into the operational workflow — and that creates disastrous hallucinations. You'll still get an answer, but it'll be completely inaccurate.

Nav: So today, after every step, you're refining what you're asking the LLM based on the previous data it found. Step one, maybe it found the upstream supplier — then you give it that upstream supplier information and say, “now find me all the downstream suppliers for this component.” Then you want the inventory, and so on. Once it finds the upstream supplier and tries to find the downstream supplier, when it goes to look up inventory, it might look it up based on the upstream supplier instead — because to the LLM, they're all just supplier IDs, and it gets confused.

Sunil: So where is determinism in that flow, Nav? What is determinism?

Nav: Great question. The determinism comes from reminding the LLM: we're talking about the downstream supplier ID here, for finding the inventory, not the upstream one. You need to carry some context in a sidecar, so at every step of the LLM call, you remind it what it's actually looking for.

Nav: The other way to do it is to not use an LLM at all for that part — to do it in code, which is the most deterministic way. It's complex, because you need to carry a lot of payload from each step — a downstream step may not just depend on the previous step, it might depend on three or four steps prior. So you need to carry that entire payload and understand the dependencies between all the calls.

Determinism Defined: Same Inputs, Same Output — Except the Data Keeps Changing

Sunil: So determinism is same inputs and same output — is that determinism?

Nav: That is correct.

Sajjad Khazipura: There are two facets, right?

Nav: Yeah.

Sajjad: Go ahead, go ahead, Sunil.

Nav: No, I was just going to say — it's largely correct, but the data keeps changing. Determinism could mean that the rationale of your question — what you're trying to do — always remains the same. It's always the same sequence of operations, and even if the underlying data changes, you're getting the answer to the question in the right way.

Sajjad's Two Facets of Determinism: Consistency vs. Wording

Sajjad: I think there are two facets to determinism. One is returning consistently the same answer every time. If I asked you the price of a Rolex watch, you'd expect the same answer no matter how many times the question is asked. That's one facet of determinism.

Sajjad: The other facet: you could answer the same question in multiple roundabout ways. That happens a lot with LLMs — you ask a direct question, and you might get the same answer, but worded differently almost every time. It might semantically mean the same thing, but the wording changes. And neither of those is great for enterprise applications — they expect input in a certain format, and they expect the input to be correct.

Sunil: And that's because the LLM is probabilistic? Is that why you're getting all these different wordings?

Sajjad: Correct. Ask “what is the capital of India,” and you'll get one answer the first time. Ask again, and you might get the same answer worded slightly differently. Which means you need to build parsers — to feed that information into an enterprise application, you need to parse the English sentence, pick out the correct content. And it's not easy to build parsers that can handle all kinds of language. That's what makes enterprise integration harder.

Sam's Reframe: Determinism in Graphs Comes from Deletion, Not Structure

Sam Pooni: I want to throw you both a question. The interesting part isn't that the graph is deterministic — databases have been deterministic for fifty years. The determinism here comes from deleting things. Every road you draw is a behavior you never have to test, defend, or explain. Safety by construction is mostly subtraction.

Sam: So my question: in graphs — Nav, you've worked on Neo4j — Neo4j's job is boring, in a sense. The real point is: hold the map, answer whether there's a path in a millisecond, and change only when a human draws a new road. The whole behavior list is really about what you can delete. You exclude things to say, this is what it takes to arrive at the answer, and it should happen within a millisecond.

Sunil: I don't understand what you're saying — if I'm using a simple analogy, I'm going from San Jose to San Francisco, and I can take 280 or 101. That's a path. Are you saying exclude the 101 path because there's too much traffic — go on 280?

Sam: I'm saying, why do you need so many paths, essentially? You want to go from point A to point B, so you need to know how to go from A to B. If something else comes up, you exclude it — because the time you have to respond in a graph is in milliseconds, and you can't evaluate every possible path, because that becomes a heuristic at that point. The main job a graph database has to do isn't just being deterministic — it needs to go further than that. What is determinism in a graph? It's about deleting things.

Nav: Yeah, another way to say it is to constrain — delete, constrain, same idea. Actually, that's how the LLM itself operates. They have a finite boundary, based on the model you choose, and the type of weight — medium weight, high weight, whatever. It basically builds a time limit: I'm going to parse my graph, and whatever result I get by this time, if I'm not finished, I'll just respond with what I have.

Sam: Exactly. So, Sajjad, what do you think? For me, it's about eliminating things, and creating the things that need to fall into place. That's how I look at determinism.

Directed Acyclic Graphs and Modeling a Supply Chain as a Tree

Sajjad: Usually these are directed acyclic graphs, for the applications Nav is building — modeling a supply chain. Say you have a phone device, and you're trying to model that supply chain. At the peak of the pyramid, there's the device made by one company, but it has maybe 10 subsystems, and each of those 10 subsystems might have multiple components. That graph is like a tree — a top-down tree — and this kind of tree is usually a directed acyclic graph.

Sajjad: So to get to a point from this graph, usually there's just one path. There could be other paths — say you had a phone and you're dual-sourcing every module, so the same component is supplied by two different vendors. That's one thing, but then you're consciously making a choice: if vendor A can't supply something, I go to vendor B. Or you split demand across two vendors — one supplies 60%, another supplies 40%. But all of that is quite deterministic, traversing this graph.

Nav: But you have situations where the demand picture changes — let's say you have parts that are critical to multiple products you're building: the phone, the tablet, the eyewear. They all share a component — say, a camera — and that camera has other components. The nature of the graph, top-down, isn't the issue. The issue is: can you identify critical parts dynamically, based on what the order volume is?

Dynamic Demand Shifts and Predictive “What-If” Analysis

Nav: In general, you've estimated you're going to sell 60% phones and 40% tablets, and that's how you're splitting your component allocation. But your demand has changed this month — because you launched a promotion on the phone, or whatever — and now you have to change which component is critical, to make more of this and less of that. You need an early-warning sign on that before it's too late. So you need to see: seasonally, when I run this promotion, this is what happened last year, and I need to change that right now. Those are the kinds of predictive signals we can get using the graph — figuring out, at this moment, what the critical component is. It's like what-if analysis as well.

Sam's DAG Framework: No Loops, Fixed Paths, “A Boring Agent”

Sam: Take the case of agents — how can they fail? There are two ways: they go where they shouldn't, or they loop forever. Those are really the only two things. In a graph, you draw only the roads you allow — no role, no action. But if you go to a DAG: no loops, every trip ends, no retracing, drawn roads rather than decisions. Go a little further with a DAG, and you have a fixed order, cacheable steps, and the longest path tells you what must stay resident — in the case of an LLM, essentially.

Sam: Neo4j holds those roads. Before adding one, it checks that it doesn't close a circle — so you have fewer roads, no circles, and a boring agent. That's what makes it durable. Is that what we're trying to move to, or is it something different?

Nav: The LLMs are really good at understanding language. You ask a question in natural language; a pre-built system may not understand that language. So the LLMs are really good at taking natural language and creating a structured output from it — a reasoning rationale: step 1, step 2, step 3, step 4. Ask the same question a different way, and it'll come back to the same four steps. That's the power of LLMs. That's the power that's most valid in the realm of enterprises — not so much the scientific-research or blog-writing use case, which is a different kind of thing LLMs are also good at, but not really an enterprise use case for the most part.

LLM Calling Tools vs. Tools Calling the LLM: Two Neuro-Symbolic Architectures

Sajjad: I have a question about the pursuit of determinism. The classical, conventional wisdom these days is to build a harness around an LLM, define a bunch of tools, and the LLM picks and chooses among those tools — the LLM is the one guiding the agent's work. That's one way: the LLM doing tool calling, to reduce indeterminism and make things more deterministic.

Sajjad: The alternative approach, for a very well-defined domain and well-defined business process: you have the tool calling the LLM whenever it needs to. You have a workflow modeled in the graph, and the graph says, at this step, call the LLM to do this function; at this step, call it to do a different function. So you have two ways of accomplishing the same work — the LLM calling the tools, or the tools calling the LLM when they need help — and both can be called neuro-symbolic architectures. Have you had to deal with this question — what's the better alternative for what use case?

Nav: When you traverse an edge in a graph, between two nodes, you sometimes have to apply certain rules, certain inferencing. That's always been a debate at Neo4j — how to accomplish that. If you look at RDF, for example, those rules are inherently built into the model. In a labeled property graph, you don't have that — it moves into the application layer. So you can have a schema for rules, and when you hit a node or edge associated with some rules, you look up what rules to run, at the application level, and run them.

Nav: You can also go to an LLM — most of the time you're making a decision about which of multiple paths emanating from a node is the best one to take, and an LLM can help decide that.

Introducing Jev: The New “No Hallucination” Model

Nav: Actually, just recently there's a new model that launched — I'm not sure if you've seen it — it's called Jev.

Sajjad: Yeah, yeah.

Nav: It's really complementary to using an LLM, because it gives you the option of offering choices, and it'll pick a choice for you based on a score. At Neo4j, my team and I built a recommendation engine called KeyMaker — I have a US patent for it — which is basically what we built as a way to score outcomes and help make a decision.

Sajjad: That's actually really interesting that you bring up Jev. I was fascinated by it — launched Monday, I think, or sometime last week. There are zillions of applications people are putting up on Twitter right now using Jev. The claim was: no hallucinations, high speed, significantly reduced costs. Those were the claims, and there were really good demos. But as the week progressed, people started reporting problems — non-determinism started to catch up.

Nav: Correct.

The Strawberry Problem Returns: Deterministic, But Still Wrong

Sajjad: It's deterministic in the sense that if you give it four choices — red, green, blue, orange — and say only pick one of these, if this event occurred, it will deterministically pick only one of those four. It won't go beyond that range. To that extent, they call it hallucination-free.

Sajjad: But where it's not hallucination-free — where hallucinations still creep in — is in what input should map to which choice. You might say, if I get this input, you should pick red, but it might pick blue for that input instead. Indeterminism creeps in there. Somebody described this very nicely as the “strawberry frontier” — remember when ChatGPT came out, and a few years down the line people started asking it how many R's are in the word “strawberry”? It would fail — it'd say one R, or two R's. People tried that with the Jev demo, and it started to fail there too.

Sajjad: So yes, it does solve some edge problems, but it's still based on the same transformer architecture backbone — that predictive nature is giving you one level of determinism, but those deterministic answers can still be wrong.

Nav: Yeah, I mean, if you have a clear rubric — these are the four choices, pick one — it'll do that pretty well. But it's very use-case specific. If your rubric is clear enough, you could possibly build that in code as well.

The Stock-Trading Example: Classification Is Deterministic, Mapping Isn't

Sajjad: You could, yeah. I think the biggest value I see — that hasn't yet been solved, which Jev promises — is when your input is unstructured. You get a body of text. Say you have a stock-trading Jev, and your choices are only three: buy, sell, or hold. Now you have all kinds of input signals — a company announces quarterly results, hires a new CEO, makes a big deal — and you want it to classify any of these news events into buy, sell, or hold.

Sajjad: It's true it'll only pick one of those three choices. But for what input signal does it pick which one? There's indeterminism there. It might choose to sell the stock even though you reported great quarterly profits, because it can go wrong in associating great profits with holding or buying rather than selling. That's really what people found — we met a good mathematician today, one of the world's leading mathematicians, and he confirmed this. Sunil and I were talking to him about it. He said, no, it doesn't solve the problem — it goes half the way, but it hasn't solved the other half.

Nav: Right.

Jev's Real Use Case: A Fast, Cheap Verification Layer, Not a Primary Answer Engine

Nav: I look at it as another tool in the tool chest — use it in the right way for the right job.

Sajjad: Good point. As long as you build safeguards and verification every time you use one of these architectures to do something — follow it up with some kind of verification step to make sure it did what it needed to do, under the right conditions.

Nav: Right.

Sunil: Sam, you look like you want to say something.

Sam: Basically, it's like shrinking the LLM to a classifier, and the hallucination strings become a wrong label that the graph catches. Something similar to that, I think — there are level 1 and level 2 versions of this, and so on. But I'm really wondering: can we do this at the speed we require? If you can do these things at greater speed, it probably makes more sense.

Sajjad: It's a great candidate, because Jev operates really, really fast.

Sam: That's what I was trying to say — 400x—

Sajjad: 400x faster, 300x lower cost. The economics and speed are really compelling. And the range of choices you can constrain — pick from these three, or these four, or I'll pick from 256 — it won't go beyond that range.

Nav: Sure.

Sam: Exactly.

Nav: The way I look at it, it's a good tool for validation or verification of something another LLM has already produced. In my case, we have a question, a rationale for the question, and reasoning for how the question should be answered. I can give that to Jev and ask: is this question answerable based on this set of steps or not? It's using a second LLM — it's not going back to the original one that produced the answer — and it gives me a clear, very quick yes or no.

Sajjad: Significantly reduced cost.

Nav: Yeah.

Sam: So that's LLM-as-a-judge — when you use an LLM as a judge and you have multiple competing answers, you could feed it into Jev, and it can instantaneously tell you this is good, or something like that. I think there are interesting applications of this as we go down the stack — combining different ways of grounding the truth. We know there are different mechanisms, through Z3, through other means. I'm thinking this could serve as some kind of operator on top, which could help bridge between these different things and combine them. Jev is very powerful — at least level 2 would be much more powerful than level 1.

Sajjad: Hopefully these things will get better.

Nav: The other aspect is, it's not anything so advanced that Anthropic or OpenAI couldn't implement themselves tomorrow. Like you said, it's based on the same overall architecture — they just built a better mousetrap for it.

Determinism's Side Benefit: Lower Token Count, Turning Off Reasoning

Sunil: I think we're almost out of time — anything else you two want to say?

Nav: I just wanted to add — determinism leads to other interesting benefits. One is lower token count. Even though your input tokens may be higher, the time spent and output tokens needed will be much less. It can really help reduce token count, because the model doesn't have to rethink what it already knows.

Sajjad: So you can turn off reasoning in those models.

Nav: Yeah, turn off thinking mode.

Sajjad: Yes, absolutely.

Sunil's Petition: “Harness” Should Be Called a “Sidecar”

Sunil: Okay, one thing I want to say — I'm going to start a petition, because I heard this today twice: from you, Nav, and from someone else we talked to today — the word “harness.” I have a real pet peeve with that, because I don't think “harness” is the right name. I think it should be, like you said, Nav, “sidecar” or something like that. I just don't like it — I hope you two sign my petition.

Wrap-Up: How Do You Catch a Wrong Intent That Still Maps to a Valid Path?

Sam: More than the harness — I think there's a bigger question. If Jev could classify requests into fixed intents, and if Neo4j constrains the allowed execution paths — how would you catch a wrong intent that still maps to your valid path?

Sunil: We'll have to cover that next time, Sam. Exactly — we're out of time. Thank you, Nav, for joining us today. If you could hang on for one minute, we'd like to talk to you.