Episode 3 — Beyond A2A and MCP: Why Agent Governance Needs Its Own Layer

Setting up today's topic

Sunil Baliga: Week three of Token Drop. The past couple of weeks we covered AI hallucinations and then how to mitigate them. Today I wanted to talk about something related: now that agents are built on top of these LLMs, how do you actually make sure they do their job properly and don't deviate from what they're supposed to do?

There's been a lot of industry discussion around protocols like A2A and MCP, but to me those are really just the plumbing for how systems talk to each other — they don't address the higher-level question of how you keep agents within bounds, actually doing their job. So I wanted to dig into that today: what does the industry need to ensure agents stay on the straight and narrow? Sajjad, I'll hand it to you first.

Constitutional AI as a starting point — but not the whole answer

Sajjad Khazipura: I think about governance at several levels. First, credit where it's due to the LLM providers themselves — Anthropic introduced something called Constitutional AI, where they defined a set of broad principles their models are expected to follow and comply with: privacy protection, refusing to give harmful advice. If you ask an LLM how to build a bomb, it'll refuse. So a meaningful amount of this is already being baked directly into the models — alignment with human values and similar principles. We're not inventing this concept from scratch; a lot of the underlying tooling already has some of these principles built in.

That said, at the end of the day, LLMs are still next-token prediction engines, and they can make mistakes. Those mistakes ripple through workflows — and as they cascade, the errors can actually amplify and diverge, creating significant downstream impact. One of our colleagues, Nafisa Hussain, actually wrote about exactly this — how errors propagate and compound as they move through a workflow. That's a classic failure pattern.

Governments are taking note too. California passed Assembly Bill 316, which holds anyone offering AI services liable for misbehavior by their agents — you can't simply say "the AI did it" and walk away from accountability. Companies are increasingly responsible, accountable, and legally liable for what their agents do. And once real liability enters the picture, the bar for what counts as "well-behaved" system behavior goes up significantly.

A quality-management parallel

Sajjad: My own background is in consulting — I worked at WIPRO, where we had a real quality management system in place: industry standards like ISO 9000, ISO 27001, and eventually Six Sigma. Six Sigma, to me, meant you were allowed at most about 3.4 defects per million opportunities to make a mistake — meaning the overwhelming remainder had to succeed. That's a genuine gold standard for quality. GE popularized Six Sigma, and several major Japanese manufacturers — Toyota, Nissan, Honda — implemented similar quality principles and practices.

What does that mean for AI? I think Anthropic's "Constitutional AI" is a genuinely elegant term for the same underlying idea. As you extend that concept more broadly into agentic systems — where agents collaborate to complete tasks — the same governance requirement applies regardless of scale: a single agent performing a single task, multiple agents each performing separate discrete tasks, or multiple agents collaborating together on one larger task. In every case, you need a governance layer that ensures these systems stay well-behaved. I'll pause there and let Sam weigh in on where he sees the state of agent governance today.

Governance as an enabler, not just compliance overhead

Sam Pooni: The core insight everyone seems to be converging on is a real shift happening in 2026: moving away from viewing governance purely as compliance overhead, toward recognizing it as a genuine enabler. Mature governance frameworks actually increase an organization's confidence to deploy agents in higher-value scenarios — creating a virtuous cycle between trust and expanded capability.

What makes multi-agent systems particularly interesting here is that they introduce genuinely unique governance challenges. Multiple AI agents interacting, communicating, and making interdependent decisions creates emergent behaviors that are much harder to predict and control. That makes proper governance even more essential — organizations need to extend their governance frameworks to explicitly cover agent-to-agent communication, coordination mechanisms, and collective decision-making processes. That layer is largely missing today.

Sunil: Here's a concrete example I keep coming back to: imagine a generic agent that can take action based on a request from me or a request from you, Sam. My access to certain data might be different from yours.

Sam: Correct.

Sunil: So if I ask the agent to retrieve something, the governance layer needs to make sure it doesn't access data I'm not entitled to and hand me an answer built on data I shouldn't actually have access to — or vice versa. And that's just the single-agent case. An agent could be clever enough to recognize "I don't have access to this myself, but maybe I can get around that by asking a different agent to fetch it for me instead." That really seems to be the heart of what this governance layer needs to prevent — the "thou shalt not" layer.

The three-layer governance pyramid

Sam: Building on what Sajjad said — I think of this as a three-layer pyramid. At the top is policy: the what and why. The middle layer is rules and guidance, which translates that policy into concrete constraints and interpretation — call it control translation. The bottom layer is procedures: the exact operational steps, the actual execution.

Sajjad: That's spot on, Sam — exactly right. There are policies guiding behavior at the top, just as — and I generally try not to anthropomorphize AI, but in this case it's hard not to — any collection of individuals, whether a family, a company, a broader society, or even a country, is governed by some kind of constitution: written or unwritten rules about what's allowed and what counts as good behavior. Within that, individuals are assigned roles — in a family, at work — and each role carries specific responsibilities, accountabilities, and entitlements: what you're allowed to do, and what you're not.

The same structure applies to agents. They need to be governed by policies, different agents need clearly scoped roles, and there need to be audit trails — everything an agent does should be logged for retrospective and post-mortem analysis. And you need real overrides: if an agent is performing something critical, like transferring a large sum of money between banks, you want genuine circuit breakers — points where the agent pauses and says, "I'm about to execute this, but I need a human to review it first." That's really a key difference between how we design microservices and how we need to think about agents.

Why agents are different from microservices: agency

Sunil: If I think about this from a laptop analogy — a laptop runs a bunch of applications, and the operating system enforces policies: this application can't access that application's memory, and so on. Is that roughly the analogous idea for AI — you need an "operating system" of sorts to enforce policy across AI agents?

Sajjad: Exactly right. In an operating system, you have threads of execution; at the application layer, you have microservices. The critical difference between those constructs and AI agents is that agents have agency. Whatever an agent chooses to do isn't entirely hardcoded or hardwired by the engineer who built it — agents can genuinely think for themselves, within limits. The more useful an agent is, the more you naturally delegate to it: the ability to perceive its environment and incoming signals, plan a response, decide which tools to use to execute that response, and then actually go carry it out.

Compare that to an operating system or microservice, where a developer explicitly defines exactly what it can and can't do — how locks get acquired, in what order, under what conditions. Most of that is hardwired in advance. In an agentic environment, agents make many of those decisions dynamically, based on the specific situation — and that's exactly what makes them so much harder to govern. Since agents depend on an LLM to interpret whatever they perceive, plan their execution, decide which tools to call, and figure out how to execute those actions, they're effectively asking an LLM to guide nearly every step. And because LLMs can make mistakes, that creates an environment genuinely ripe for all kinds of failure.

More capability, more surface area for risk

Sunil: It seems like as more of these protocols emerge, agents gain the ability to do more and more independently. It's a bit like the early days of USB ports on laptops — before they existed, you didn't need to worry about someone plugging in a rogue device, but once they did, that became a real risk. At one previous company I worked at, they physically locked down USB ports so people couldn't plug in unauthorized devices. It seems like as the number of protocols expanding what agents can do keeps growing, the need for this governance layer becomes correspondingly more important. With great power comes great responsibility to actually enforce these policies.

Sajjad: Couldn't have said it better myself — exactly right. With great power comes greater responsibility, because you're granting agents real autonomous execution ability, which means they now need to operate far more responsibly, and someone needs to be actively watching over that.

New frontiers: agent-generated UIs

Sunil: It was interesting — Abby mentioned recently this emerging capability where agents can generate a UI on the fly specifically to collect information from people, rather than relying on a predefined interface. The agent essentially defines the interface itself, dynamically.

Sam: That's A2UI — we've actually been discussing this. It's a genuinely interesting emerging trend, and it can generate interfaces tailored to different device types.

Sunil: So you could end up with an agent generating a UI specifically designed to extract information it shouldn't have access to — which again points back to needing some kind of governance layer, an "operating system" equivalent, to prevent exactly that.

A real example: how governance rules get lost through summarization

Sam: Here's a very real behavior I've personally observed. When you're doing prompt engineering, you give the model a set of rules — for example, "do not access the production system." You hand that to the LLM, and it seems to work fine initially. But over time, as the conversation or context grows, the model starts summarizing earlier content to manage length — and somewhere in that summarization process, the fact that you explicitly said "cannot access the production database" quietly gets lost. Eventually it comes back and says something like "we need to optimize database usage" — and once that summarization has happened, the full original meaning is gone. Suddenly you discover the LLM actually accessed the production system, and you genuinely don't know why. Preventing that requires real structured policy — the intent layer — translated down through rules and guidance (the control-translation layer) into actual operational execution underneath. You need all of these layers working together.

Sunil: Let me make sure I follow — if I have a database and multiple agents are trying to read from or write to it, is that a concrete example of exactly this kind of resource-access governance problem?

Sam: Yes — anywhere you have agent-to-agent communication, and especially anywhere that communication crosses system levels, you need genuinely tight coordination. Current protocols help somewhat, but not nearly enough on their own — you still need that governance pyramid: policy, rules and guidance, and procedures, all working together, so the outcome is deterministic and predictable. That's exactly what enterprises actually want — not some unpredictable combination of behaviors where something quietly breaks.

Wrap-up: governance as the missing piece

Sunil: I think we've covered the topic well. Anything else either of you want to add?

Sajjad: I think this ties in nicely with our earlier conversation on reliability. Governing an agentic system — whether it's a single agent doing a single task, multiple agents each doing separate tasks, or multiple agents collaborating on one large task — is genuinely critical. Governance, broadly, provides oversight over what agents can and cannot do, what access and entitlements each agent has and doesn't have, and leaves behind a real audit trail of everything that happened — so if something goes wrong, you can trace back and understand exactly what occurred.

Sunil: "Operating system" was just an analogy on my part — there's probably a broader term that fits better than that.

Sajjad: Right, which is why I keep gravitating toward human-society analogies instead.

Sunil: Right — like you said, a constitution.

Sajjad: Exactly.

Sam: I think we've covered this in real depth — touched on pretty much every important angle. This governance layer is genuinely missing across the industry right now, and it's where people need to focus. It's also not something the whole industry has fully recognized yet, though more organizations are starting to notice the gap. Hopefully it gets addressed soon.

Sunil: Really interesting conversation this week. Thanks, everyone — we'll talk to you all next week.

Sajjad: Thank you.