Episode 10 — Why AI Agents Need an Operating System

Beyond governance: the broader aspects of enterprise agents

Sunil Baliga: Token Drop, Episode 10. We had an earlier episode on agent governance, but the more I thought about it, there are a lot of different dimensions to how agents actually function in an enterprise — performance, reliability, and more. Let's kick off today by talking through some of those broader aspects. Sajjad, what's your view?

Sajjad Khazipura: An agent, at its core, is how intelligence gets manifested for a specific business use — essentially a wrapper around an LLM. In a business scenario, it helps to think of an agent the way you'd think of an office worker. When you think of an office worker, you have certain baseline expectations: they're knowledgeable, they're responsible, they're generally correct in what they execute — you expect some real guarantee of correctness and accuracy — and they're timely, in the sense that things keep moving in a business environment.

You should hold agents to those same expectations. Given who we are and what we do, we're naturally focused on correctness and mitigating hallucinations — but that's only one piece of the puzzle. You also want these systems to be timely, both when you ask a direct question and within multi-agent workflows, where the expectation is that agents are communicating with each other at machine speed. But even the best LLM throughput today is on the order of hundreds of tokens per second. We're starting to see new hardware accelerators push that higher, but we're still fundamentally talking about multi-second response times, not true machine-speed communication.

So: timeliness, correctness, and then reliability — how do multiple agents function as a group? It's similar to multiple workers in an office trying to get something done together. If a human worker falls ill or has to take emergency time off, other workers step in to bridge the gap. Likewise, if an agent fails or effectively "falls asleep," how do you complete the transaction it was handling? How do you even discover that kind of failure, and how do you work around it? Do you have real awareness of distributed-systems algorithms and semantics — correctness at scale in a distributed environment, tolerating failures, discovering errors, recovering from them? Distributed databases like MongoDB and Cassandra have been solving exactly these problems for well over a decade, using distributed algorithms and distributed computing techniques we already understand well.

Then there's security on top of all of that. So at a macro level: timeliness, correctness, and the awareness that you're operating in a genuinely distributed environment alongside other agents — which means you need real fault tolerance, recovery, and distributed-computing awareness built in. Those are the broad capabilities I think we need to be mindful of.

The case for an "agent operating system"

Sunil: It almost sounds like what you need is an agent operating system. On a PC, applications occasionally hang — it happens less than it used to, but it still happens, and I just force-quit it on my Mac. What happens when that occurs in an enterprise environment, with an agent instead of an app?

Sajjad: Couldn't have said it better — "operating system" is exactly the right framing, and we're starting to see the industry move in that direction. Inference services like vLLM or SGLang are increasingly taking on capabilities we used to expect from an operating system, and in real-time business applications, they're actually delivering on a lot of that. But there's still a real gap to bridge.

Traditional operating systems operate on timescales of nanoseconds, microseconds, maybe low milliseconds — anything longer than a few milliseconds gets treated as an I/O workload: you dispatch it to a remote system, and that system interrupts you when the job is done. It's not treated as a first-class computational task; it's an I/O offload. But ML and LLM inferencing is actually a genuine compute job, not an I/O operation. So the real question is: how do you give that compute job first-class status?

Sunil: What do you mean by "first-class" — access to the rights and privileges it needs, or something more specific?

Sajjad: More specific than that. Being first-class means being a real participant in the CPU-memory complex and its protocols — think cache coherency: one CPU knows when a piece of data isn't in its own cache but is sitting in another CPU's cache, and the underlying protocol knows how to fetch it from that remote CPU's L1 cache when needed. But when an accelerator is doing the processing, your CPU has no idea what data the accelerator is touching, and the accelerator has no idea what's being modified in the CPU's cache. That cache coherence simply doesn't exist yet between them — though people are actively working on it, experimenting with technologies like CXL and other protocols to make it happen. That's one dimension of being a first-class citizen: being treated on par with the CPU and memory in that complex, genuinely collaborating, cooperating, scheduling, and executing instructions together. That's not happening today.

Agent memory: why persistence matters

Sunil: These agents would also seem to need real memory, the same way an application does. We were talking the other day about an agent that helps people order coffee — you don't want it repeatedly asking "do you want cream and sugar?" when someone's told it five times already; people get irritated by that. So this "operating system" needs some mechanism for agents to hold persistent memory.

Sajjad: Exactly — and memory, the way we experience it as humans, is structured into hierarchical tiers. There's short-term, episodic memory: if I mention something to you right now, you'd expect me to remember it a moment later. But if you asked me 20 days from now whether you'd ordered one spoon of sugar or two, I might genuinely not remember. So there's long-term memory, short-term episodic memory, and session memory that's only relevant within a given interaction. Memory itself needs to be classified into tiers.

How you organize that is an open question — one idea I keep coming back to is extending the knowledge graph itself. We already talk about storing data in the graph; can you extend that graph into an "action space," where some of the transactional state agents generate is also stored in the graph, and that state then serves the agents as they transact with each other?

Sunil: A multi-dimensional graph.

Sajjad: Exactly — a multi-dimensional graph extended into the action space. We haven't fully fleshed this out, but it's something worth developing further.

Two points of failure: query construction and grounded response

Sam Pooni: Sajjad's hit on some really important points here. Just having agents doesn't actually solve the problems agents introduce. Take the failure surface Sajjad mentioned: say a bot makes a query and gets back some data — structured or unstructured. There are really two distinct parts to that equation where failure can happen. First, the natural-language question has to be processed and built into an actual query. Second, once the agent (or the underlying LLM) queries the database and gets a result back, that result has to be converted back into natural language. Both of those steps are places where failure can occur.

If the process isn't grounded in some verified source of truth, the returned data can simply be wrong. And if you make multiple calls, or have multiple agents responding to conceptually the same request, they can genuinely return inconsistent answers.

Sunil: Are you talking about a chain of agents — one agent passing an answer to the next, and errors compounding as they go?

Sam: No, actually the same agent, queried multiple times. Because there's an inherent translation happening at each step, if that translation isn't grounded in something reliable — an ontology-based system, or something otherwise verified — the answer you get back can genuinely differ from one query to the next. We've seen this happen in real agentic systems.

Sajjad: Which is exactly the correctness-guarantee problem.

Sam: Exactly. The second thing worth flagging is what Sajjad raised about working and short-term memory. Imagine a conversation where every time you come back, the agent effectively asks "who are you? What do you want?" — that's genuinely irritating. You want it to retain enough context that next time it recognizes you and picks up where things left off, rather than starting from a completely blank page every time, which is what happens with most current query-based interactions today.

Memory as a first-class reliability risk

Sam: So there needs to be a real notion of working/short-term memory, plus long-term memory — which I'd break down further into episodic, semantic, and procedural memory. Bridging all of that together turns this into a genuinely different problem. Why is memory a first-class reliability risk? Because there are several distinct modes of "poisoning" that can occur. At minimum, three failure modes: memory poisoning, tail context (context that lingers and distorts later reasoning), and cross-agent contradiction. These aren't minor edge cases — they fundamentally shape whether a multi-agent system actually works. It's not as simple as saying "I've wired up a bunch of agents in Java and they'll just work."

The other piece is orchestration patterns. Sajjad touched on this earlier — there are several common patterns: sequential, concurrent, handoff, group-chat style, orchestrator-led, and more. Each of those patterns brings its own complications. And there's a real impedance mismatch here too: traditional systems are built around a known, tight order of magnitude for response time, but these agentic systems can take seconds to respond. That mismatch matters.

Sunil: So where a traditional operating system might expect millisecond-scale response, this is operating on a multi-second scale.

Sam: Exactly — an impedance mismatch. And this isn't a brand-new problem; distributed systems have wrestled with versions of it for a long time. I've worked with Paxos, and with distributed databases like Riak and Cassandra — all of these rely on eventual consistency and are shaped by the CAP theorem. The same underlying problems resurface in multi-agent scenarios: failures happen, and you also have to think about transactional boundaries. Every layer in the system does something, and ideally each layer should be semantically correct enough that you can define real transactional boundaries around it. You can end up with multiple smaller transactions grouped into a larger one — and when something fails, part of that larger transaction needs to be rolled back. So the real question is: how do you build that kind of transactional boundary and rollback capability into agentic systems? How do you tie all of this together — something like the distributed-transaction "saga" pattern — because this isn't just an orchestration-control problem.

A real-world timeout story

Sunil: This connects to something a friend told me recently — he was calling an LLM via API from his application and had set a 30-second timeout, which based on his past experience felt generous. But he was still hitting timeouts, because the LLM — a popular one — would get busy and occasionally take longer than 30 seconds to respond. He had to widen his timeout window just to account for that.

Sam: Exactly the kind of thing we're describing.

Should agentic systems be treated as compute — or as a database?

Sajjad: That actually raises a more fundamental question: should we treat these systems as computational systems — which implies you'd expect extremely low latency, similar to how a CPU processes data within clock cycles, moving values through registers and out to memory for something else to consume? Or should we treat this more like a storage surface — a passive database: you send a query, and it eventually gives you a response, like a passive database lookup. If you treat it like a database, then response times of one or several seconds are perfectly reasonable.

Sunil: Are you saying that because the agent's own native compute is limited — it's mostly offloading the actual "thinking" to the LLM — it behaves more like a database doing data manipulation via the LLM, rather than doing heavy compute itself?

Sajjad: Exactly, Sunil. You can treat it either as a passive database — you send a query and wait for a response — or as an active computational system genuinely working on real-time data as it streams in. If you're a trader at a hedge fund working off tick data on the New York Stock Exchange, milliseconds and microseconds matter enormously; you're processing that stream as it arrives. If you wanted to use LLMs in that kind of use case, there's still a long way to go in terms of performance and integrating them into truly low-latency processing stacks — which would require cache-coherent, shared memory across GPU and CPU. People are actively discussing those architectures, but nothing like that is practically available today. Right now, an LLM is still treated like an offline database: you send a query, it responds in its own time, and that's acceptable as long as it's accessible to your application. But if you want to participate in genuine real-time streaming signal processing, you need another significant step up in performance and capability.

Two halves of the problem: orchestration pattern vs. memory architecture

Sam: If you really break it down, there are two separate halves to this problem. One is the orchestration pattern — which determines who talks to whom (A talks to B, and so on). The other is the memory architecture — which determines what each agent actually knows and trusts. Those are genuinely distinct concerns. A multi-agent system can have excellent orchestration but poorly governed, self-written memory — exactly what Sajjad was describing — and it will still drift. When that drift happens, it leads to cross-agent contradiction, and this happens constantly in agentic systems today.

What keeps a sequential pipeline, a parallel fan-out, or an orchestrator-led swarm coherent is a shared, verifiable semantic layer — the same "verifiable" concept we discussed earlier. Without it, even a well-designed orchestration pattern will eventually run into trouble, because the memory layer is where drift creeps in. That takes us right back to distributed-systems territory.

Sunil: What kind of drift are you referring to, specifically?

Sam: I mean the different memory types and how agentic systems communicate with each other. When agents write back to a shared system of record, if something gets written incorrectly — due to a processing error, or because it's subtly drifted from the actual underlying facts — that drift compounds over time. This is the same kind of compounding-failure pattern we've seen in other systems. Eventually, you end up having drifted quite far from the original source of truth.

Sajjad: It really comes down to this: if you have two agents working on a shared task, each operating on its own separate state, you can get drift if the tasks aren't mediated effectively. But if they're working against genuinely shared state, you open yourself up to deadlocks, livelocks, and race conditions — and even there, you need correct orchestration to guarantee the correctness of those transactions.

So you're choosing between shared state, which requires coordinating reads and writes carefully, or distributed state, which itself splits into two cases. One: you don't actually care about staying in sync — you maintain your own state, do your job, and all that matters is that when someone asks you a question, you give them an answer; you're not duplicating or tracking someone else's state, in which case drift isn't really a meaningful concept. Two: you and another agent are both maintaining state, and if I ask you something and your answer differs from the state I'm independently tracking, that's a clear sign of drift. In that second case, you again need real correctness guarantees and sequencing — regardless of which scenario you're in, you genuinely need distributed-computing algorithms to coordinate these transactions effectively.

This is exactly why people talk about Paxos and Raft so much, and why distributed databases are designed the way they are. In the most rigorous case, you have a distributed lock manager: you assert a lock, write to a location, then release the lock — and sometimes a process dies while still holding that lock, so something else has to detect that and perform lock recovery. None of this is easy — databases like Cassandra, MongoDB, and Oracle RAC have been solving exactly these problems for at least a couple of decades. If you need that level of correctness semantics, it's genuinely hard to build from scratch.

Push consistency down to the database layer

Sajjad: My view is that we should push this responsibility down into the database layer, and treat agents more like applications sitting on top — agents can stay relatively unaware of the deep distributed-systems complexity, as long as the underlying database layer manages state and consistency correctly. That said, you can't ignore the fact that there's shared state that genuinely needs to be managed.

Sam: Exactly — the shared, verified semantic layer. This matters especially with certain orchestration patterns — say a DAG that starts sequential, fans out, and then merges back together. At those merge points, state is genuinely being shared across agents, and that's exactly where you have to be careful. If you're doing an aggregation and different agents are each contributing part of the combined state, what happens if one agent doesn't respond in time? What if one of the contributions is simply wrong? These are classic problems in everyday distributed programming, but they show up with much higher stakes on the agentic surface.

Sajjad: That's exactly right — which is why it's so critical to deeply understand the actual business use case: how the business problem decomposes into a technical solution, what levels of abstraction are appropriate, and — if you don't want to own that complexity yourself — how you delegate correctness guarantees down into the database layer while still keeping the promise of overall correctness intact.

Wrap-up: verify against evidence, not inherited assumptions

Sunil: We're about out of time — Sam, anything you want to close on?

Sam: I think that's a fair summary: any downstream agent needs to verify against actual evidence, rather than simply inheriting an upstream agent's guess. That's really where the core problem lives, and the industry is still actively working through it — across orchestration, memory, and everything in between. We'll get there eventually, but these are the real problems present in agentic systems today.

Sajjad: Agreed.

Sunil: Great — thanks, guys, we'll talk soon.

Sajjad: Thank you.