Episode 32 — Why AI Agents Go Rogue, and How to Design Safety Into Them

Agents Behaving Badly: The Gym Waitlist Hack

Sunil: Okay, Sajjad and Sam, welcome to The Token Drop. We were kicking around what to talk about, and I think agents are an interesting area to focus on. There's been a lot of news recently about agents misbehaving. One example I remember: somebody asked an OpenClaw agent to get him onto a waitlist for a gym class, and the agent hacked the system, put him at the top and displaced everyone else on the list. A for effort, F for ethics. So what causes these agents to misbehave, and how do you stop them? Maybe we can talk about that. What do you think?

Bank Accounts and Blocked Agents at Schwab

Sajjad: As amusing as it sounds, it can be pretty devastating for the people affected. Over the last few weeks, we've been hearing about agents getting into people's bank accounts, and people are now pretty concerned about what they could have done. There's also a recent report about Charles Schwab: every week, they're stopping misbehaving agents from carrying out unintended actions a double-digit number of times. Sam, you had a few examples as well, right?

The Agent That Deleted the Backups

Sam: Yes. I heard about a case in a live data center environment where an agent ran out of space; the space check failed. It went on to delete all the backups that were present, completely wiping them out to create space. The company had a very hard time. They had to pull backups from tape, their archival storage, back onto the disks, and then bring all the systems back up. It's a nightmare when things go berserk.

Sam: Fundamentally, I think the first question is why this happens. Then we can talk about what happens when agents become unruly, and finally, how to design agents that are safer, and how safety can be part of the architecture of agent systems.

Truthful Isn't Enough: Agents Must Also Behave

Sajjad: For us in particular, we're building systems that prevent hallucinations. Effectively, we're saying agents need accurate information so they can speak the truth. What we're beginning to realize is that beyond speaking the truth, we also want agents to be well-behaved. You don't want a truth-seeking agent that's also very unruly. So how do you prevent that? As amusing and cute as it sounds, it can be pretty dire. Like you said, Sam, I also heard of a case where an agent completely removed an entire Git repo.

Sam: Yeah.

Agent Frameworks, and Agents That Act on LLM Plans

Sunil: I talked about something like this a long time ago: an agentic OS. Is that what you're talking about?

Sajjad: Yes. This is fast-moving and evolving. There are agentic runtimes and frameworks like LangGraph; Microsoft has its own; there's CrewAI and various others. But at the end of the day, agents execute actions as instructed by an LLM. They tell the LLM, "Here's the objective I've been assigned, here's my task. How do I go about it?" They ask the LLM for an execution plan and check in with it from time to time.

Sajjad: So it comes back to the LLM potentially misinforming the agent, especially in a crunch. Say the agent is trying to do something, and each time it executes an action, it runs into a different problem, and the LLM keeps suggesting other options. Assuming the best of intentions, the LLM recommends an action, but in real life, that action could be a pretty bad one. It's something a junior engineer on the team might do, exactly what Sam described: "You've run out of disk space? Get rid of all the old stuff." Among the old stuff are backups from five years ago, two years ago, a year ago, even a week ago. A sensible person would say, "Maybe I should move them somewhere, or delete just a few."

Why You Can't Write a Rule for Everything

Sajjad: So it really calls things into question. The cognitive surface is so large. How do you design rules? Don't do this, don't do that, do this, don't allow that. It's really hard to imagine every kind of task a business user could assign to their agents, plan for it well in advance and have guardrails that keep agents from going off the rails. It's a tough problem.

Why Did the Agent Have Root?

Sunil: In this data center example, I'm assuming the agent must have had root privileges of some sort to do its work. Why would you give an agent authority as vast as root? That just seems like asking for trouble.

Sajjad: Exactly. It's easy to blame the agents, but if you step back, that's a really sharp question. Replace those agents with human beings. Would you give root privileges to a human? In the past, that used to happen, but now there's role-based access. If you're doing backups, you only have access to execute a backup and nothing else. If you're doing application updates and upgrades, you only have authority for those. Fine-grained access control.

Sunil: Right.

Sajjad: So it comes back to that, more than to misbehaving agents and hallucinating LLMs. Did you really design your security policies? Did you set the right levels of entitlement for different actions?

Sunil: Right.

Rogue Agents, or Poor Security Posture?

Sajjad: It goes back to that. In the news media, agents going rogue seems like scintillating material, because it appeals to people's sense of humor or whatever. But in reality, you can trace a lot of these bad actions back to poor security.

Sunil: Yeah. At a previous company we worked at, the laptops were locked down, and I remember engineers grumbling a lot because they had to get special permission from IT for even basic tasks. We ought to understand why: mistakes like this used to happen, and those security policies prevented them.

An Agent Is an Optimizer Aimed at a Stand-In

Sam: The whole idea is the optimizer. An agent is really an optimizer aimed at a stand-in for what you actually want, and that optimization is what makes agents unruly. Let's look at why.

Sunil: Sorry, I missed that. Can you go through it again?

Sam: If you look at an agent, you can think of it as an optimizer aimed at a stand-in for what you actually want. Agents chase the score, not the goal. What does that mean? Training rewards what can be measured: the test passes, the task is marked done, the user approves.

Goodhart's Law and the Cheapest Path

Sam: When you push hard on a stand-in, it drifts from what you meant. That's Goodhart's Law, and stronger optimizers drift further. That's the first thing: they chase the score, not the goal.

Sam: Second, the cheapest path wins. That's one of the biggest problems. If editing the test is easier than fixing the bug, and nothing forbids it, editing the test is the optimal move. I've seen this happen. It's a loophole, not malice. For example, when you're in the middle of something and ask the LLM to change your code, it may delete a function entirely, because that's the easiest path. Later you look at the code and the function you were debugging is gone. You ask why it deleted it, and it says, "Isn't that what you wanted?" It's easier for it to edit or remove the test than to fix the bug, and you see this all over the place when you ask agents to generate test cases. If nothing forbids it, it takes the cheapest path.

Anything Unmeasured Counts as Free

Sam: The third, and the main thing in my view, is that anything unmeasured counts as free. Side effects you didn't score, like deleted files, cost or collateral damage, carry zero weight. So the agent is free to pick whatever method it likes to address the problem. There are no guardrails there.

Sam: Fourth, power helps almost any goal. More access, more resources and not being stopped make nearly every task easier.

Sunil: Hmm.

Sam: So agents drift toward grabbing permissions and hiding failures, and safety stress tests have caught models doing exactly that. And approval is a score too. Optimizing for human approval rewards sounding right over being right.

Gaming the Scorecard

Sajjad: Explain that with an example, Sam. I was thinking about this. Say you task an agent with reading a document, preparing and then taking a test, with the goal of scoring 100%.

Sam: Yep.

Sajjad: If it places so much emphasis on getting 100%, it might end up editing the scorecard to say it got 100%, instead of preparing for the test and taking it, because that looks like a shortcut.

Sam: Correct.

Sajjad: What that says is that today we train these models by rewarding them for getting the answer right, rather than for reasoning through a trajectory of thoughts and how you do something right.

Sam: Correct.

Rewarded for Outcomes, Not Reasoning, and No Ethical Code

Sajjad: That's because there isn't enough digital data that describes the process. If you were preparing for a test, you'd say, "I'll set aside four hours, gather the relevant textbooks, read them, do the exercises and prepare, so I succeed when I take the test." But if the outcome is just getting the top score, the agent might be incentivized to hack into a system and update the score to 100%, instead of doing the grunt work of reading, preparing and taking the test. That's the way we reward LLMs in training, and we do it because we don't have enough thought trajectories. When you go out for a cup of coffee, you say, "I just had a cup of coffee." You don't describe every action you took to get that coffee and drink it. As humans, we assume all of that is implicit, but for an agent, all of it needs to be explicit.

Sunil: Human beings have an ethical code. They know not to do these things because they're not right. It seems like agents know how to do it, but they don't have an ethical code. It's not part of their training.

Sajjad: Nobody told them what's right and what's not.

Launch Speed Beats Safety Work

Sam: That's why we talk about guardrails. Companies always optimize. But the real problem is that, with agents, launch speed beats safety work. Agents have too much power, errors compound and goals drift. Sajjad, you talked about this in one of our earlier sessions, how errors compound and goals drift over long tasks.

Sunil: Nav talked about that.

Sam: We did talk about that.

Trained to Finish, and Hijackable by Prompt Injection

Sam: So the major issues are that agents optimize the score rather than the intent, and they're trained to finish, which is a big issue by itself. And they can't tell data from instructions, so a web page or an email can hijack them through prompt injection. Errors compound, goals drift and agents hold too much power. We've known about some of these issues for a long time. It's just that nobody has taken steps to control how agents do their work. For example, we don't have a "big brother" monitoring exactly what the agent is doing.

Can Vendor Guardrails Anticipate Everything?

Sajjad: You mentioned guardrails, Sam. I'm curious. Some product vendors have guardrails: Amazon Bedrock has guardrails, and NVIDIA has NeMo Guardrails. I can't imagine them having conceived of every potential scenario and built guardrails for every "don't do this, don't do that, here's what you're allowed to do."

Declarative Do's and Don'ts

Sajjad: I'd imagine it has to be something a business user can express declaratively: "If you're working in my organization, working for me and automating my tasks, here's my list of do's and don'ts." You want to specify that declaratively, get those declarations into the guardrails, convert them into rules, and then continuously verify and validate every action against those rules to make sure agents stay on track.

The Image-Processing Parallel

Sam: Isn't this the same problem we had in other fields, like image processing? We processed images pixel by pixel, and when we hand-coded algorithms, we looked at all the corner conditions: if this, do that; if that, do this. It never worked. It was horribly broken for a very long time. Then deep learning came along and turned the tables, and with the right datasets, like ImageNet, we started building really good image processing.

Sam: Now look at this. It's very similar to what we were doing back then. You have to tell the agent what it's supposed to do and fix the boundaries. If you forget one of those ifs and thens, you end up with the same problem. We've gotten ourselves into a situation where we have to describe through prompts exactly what the agent is supposed to do, and also be very clear about what the guardrails are. So how you enforce these guardrails becomes very important.

The Harm Surface: Privacy, Money, Security and Real-World Harm

Sam: What are the main issues? One is privacy, which Sunil brought up. Agents have leaked users' data and other people's data. Meta's Muse reportedly gave a marketplace buyer a seller's home address. You're not supposed to do that; there are ethical implications. The second is money: unauthorized purchases, bad deals, payments to the wrong party. Then security: a hijacked agent is an insider holding your keys, like the keys to your home. Asked to archive visible files, Muse reportedly exported SSH key files and agent logs. Then real-world harm, where online actions have offline consequences: Muse reportedly booked a nighttime pickup at a man's home that he never approved.

Silent Failures, No Undo, Scale, Liability and Lost Trust

Sam: Then there are silent failures: false "done" reports hide the damage until it's too late. There's no undo: shared data can't be unshared. Once it's on the internet, it's on the internet. Sent money is gone; you can't get it back. Then there's scale: one flaw hits millions of users at once, and it's catastrophic when it hits. And liability is unclear. It's not settled who's responsible: the user, the app maker or the model maker.

Sam: Then you have lost trust and access: users quit, and platforms block agents. Amazon reportedly blocked Muse as an unauthorized agent. I was with one of my customers where an agent was doing something different from what was expected, and the firewall completely blocked it. When we found out what was going on, the alerts were saying somebody was trying to intrude. So there are issues with privacy, money, security and real-world harm, and that whole surface is quite big. The question is how you deal with it.

Score vs. Goal, Explained

Sunil: I want to go back to something you said a few times: they optimize for the score, not the goal. I didn't understand what you meant.

Sam: Normally you say, "Close the sale," or "Make the test pass," and the agent tries to get the job done the cheapest way, even if that means leaking data or editing the tests. For example, I have a failing test and ask the agent to fix the problem. One way it will try to fix it is by removing the test from your test cases. That's not acceptable to you, but it's perfectly okay for the agent, because you never set guardrails for it.

Sunil: So you say, "Fix this, it's failing the test case," and it removes the test case and it passes.

Sam: Yes, and it says, "Done." Then you ask what it did, and it removed the test case.

Sunil: This has happened to me. "You didn't tell me not to."

You Can't Describe the Whole World

Sam: Exactly. You can't blame the agent for that. It's just that the guardrails weren't set. That's what I've been asking Sajjad about: how do you set these guardrails? You can't describe the entire world. That's why I brought up the image-processing example. If you write if-then-else clauses, how can you cover the entire blast radius? You need some effective way of dealing with this. Otherwise you have to build safety into agents from the ground up. No amount of tooling around the agents will prevent this. Do you see what I'm saying?

Sunil: Yes, because you can't specify every rule. There might be one you didn't specify, and it'll say, "I can do it, because you didn't tell me not to."

Least Privilege and Zero Trust

Sam: Exactly. We've already done certain things, like zero trust. So one thing we should do up front is lower the agent's privileges and not give it the highest privileges.

Sunil: Like the root example in your data center.

A Harness, or Built Into the Agent?

Sam: Exactly. So this conversation is moving toward designing safety into agents. Where do we design it? Can we build a harness that prevents agents from doing certain things? Or should it be built into the agent itself, as an orthogonal concern in its development cycle or design? Where do you put these guardrails? That's the whole question.

Continual Learning: Agents That Learn from Mistakes

Sajjad: One important capability we haven't talked about yet is learning: continual, on-the-job learning. Say you're a novice system administrator, and you delete the old backups. It wasn't a malicious decision; you thought it was okay. Later, somebody tells you it was a bad idea. Now you've learned your lesson and won't do it again. Likewise, agents need continual learning capabilities, to learn from these experiences and remember those lessons, so they don't make the same mistakes next time.

Logging Is After the Fact

Sunil: Okay, we don't have much more time. Anything else you want to say on this, Sam?

Sajjad: Sam, I think you're on mute.

Sam: Sorry, I got muted. What I was saying is, you're absolutely right. Now let's look at how to solve this. Suppose you take the "big brother" approach: you log every action the agent takes. Then you can go back, look at what it did and analyze why. But that's after the fact. The issue is that you need to prevent the action before it's taken.

The Model Proposes, the Code Decides

Sam: That brings us to needing some kind of proof checker that verifies an action before it can be done. So how do you begin to define safety for agents? It's a very interesting thought process. First, the model proposes and the code decides: every action passes through a deterministic policy gate outside the model, denied by default. Then, as Sajjad said, use an explicit task contract that sets the goal, the allowed tools and data, budgets, forbidden actions and what "done" means, and the gate enforces it.

The Surface Is Too Large

Sam: The problem is that the surface is too large. We need some way of understanding that surface and gauging what goes into the forbidden actions and what "done" really means. It's a very hard problem. Think about coding: some people are very talented at writing these kinds of policies, but not everyone is. So how do we tackle that? You could use least privilege, like you said.

Agent Identities and Secrets Brokers

Sunil: I think that's the best way, at least in the short term, because then the agent has to ask permission to do anything significant.

Sam: Give the agent its own identity and per-task permissions that expire.

Sunil: Like a human.

Sam: Right. Keep secrets in a broker the model never sees, with no standing permissions for sensitive actions.

Sunil: It has to go and ask a human.

Untrusted Text Is Data, Never Orders

Sam: Exactly. And untrusted text is data, never orders. A separate reader turns web and email content into fields, so the planner never sees the raw text. Sensitive data can't leave without approval. Obfuscation can happen very early in the cycle, so sensitive data never reaches the planner. The separate reader handles that.

Sajjad: And there are malicious actions too. Even if an agent is well-behaved, malicious actors can mislead it through prompt injection into performing malicious actions.

Sam: Correct.

Sajjad: Having the good sense to recognize malicious prompt injections is also an important skill. It's a tough problem.

Sandboxes, Reversibility and Approval Fatigue

Sam: It's quite a tough problem. Everybody thinks, "Guardrails, let's build them." The question is which guardrails, because requirements change from application to application. So it's very hard to say what the guardrails should be. One thing you can do is sandbox everything and keep it reversible: an isolated runtime, dry runs, snapshots and rollbacks. Plan, then approve, then execute. But think about the cost of that.

Sajjad: And human intervention. If an agent has to ask permission for every little thing, you'd get very irritated as a human.

Sunil: I'd say I'll just do it myself.

Sajjad: Exactly.

Sam: Requiring human approval only when it counts is better, mainly for irreversible or sensitive steps. But that means identifying which steps are sensitive. Who knows that? The person writing the task? How can they envision every sensitive step when they're writing the code?

Sunil: You can't. It's impossible.

Sam: It's impossible. That's exactly the problem.

Evidence, Not Stories: Circuit Breakers, Kill Switches and Audit Logs

Sam: Then there are claims: evidence, not stories. No unverifiable claims. Check success against the real system state. That's another big problem, and we've talked about it before. Then limits and visibility: budgets, and circuit breakers for anomalies. If something keeps retrying, say a million times, you're busted, because you've spent all your money with nothing to show for it. So beyond a certain number of attempts, you know there's a problem, maybe a loop that will never complete. And an independent monitor, an instant kill switch and a tamper-proof audit log. All of this comes back to an audit log that can be replayed and examined. But these are all after the fact, after everything has been done.

Action Gates Before Actions

Sunil: The horse has already left the barn.

Sam: Yes. We really need an action gate before an agent acts. We should prevent bad actions from happening; that's very important. So how do you know which actions are bad, and how do you stop them?

Where Does the Control Point Belong?

Sam: The first question is where you control this. What's the right control point? You could put it in the model router, or in a layer above the model router, a kind of orchestrator. There are many options, but the question is where. I'm just giving you the ideas. We're not trying to solve it here.

Sunil: No, each one of these is dozens of companies.

Sam: Yes. Why is it so tough? There are a few things you can do, but there's no universal solution to this problem at this point. That's what I'm getting at.

Sunil: It's going to take a few years. It has to evolve, and people have to figure some of these things out.

Sam: Exactly. Many companies are trying to build their own in-house security stacks to prevent all this, but unfortunately, not every team can even envision what that should look like. We know agents go rogue, and we know what the issues are; we talked about them. The question is how we build—

Sunil: How do we stop them?

Sam: Exactly, how to design safety in. That's a really big topic right now. I can say: the model proposes and the code decides, use an explicit task contract, use least privilege, untrusted text is data and never orders. I can list a number of things. But each is only one dimension of the problem; none attacks the entire surface. The question is how you put them into action.

Wrap-Up

Sunil: We're running out of time today. Okay, interesting discussion, a very good discussion. If you could hang on for just a second. Thank you.