Token Drop Podcast · Episode 33
Episode 33 — Running a Fleet of AI Coding Agents: Specs, Review and the Human Bottleneck
Also available on YouTube · Spotify · Apple Podcasts
About This Episode
What happens when a fleet of AI agents can take a feature from requirements to production faster than any human can read what it produced? Angus Dunn, founder of Captomi, joins Sunil, Sajjad and Sam to share what he has learned running coding agents for clients, including in regulated industries. They get into why the spec is now the product, why humans have become the bottleneck, how to stop agents from gold-plating code and fixing tests instead of bugs, what auditors want to see, the agent that ran up a $20,000 bill in three days, and what all of this means for senior and junior engineers.
Episode Summary
Sunil opens from DaaX's new office, running on hotspots until the internet is connected, and introduces Angus Dunn, founder of Captomi, whom he met at a Tech Week event in San Francisco. Angus explains that Captomi provides a fleet of coding and development agents that take a client's requirements through design, coding, testing, staging and production. Clients want two things: speed, to ship before anyone else, and quality, which is much harder to know when eight or ten VMs of agents are implementing requirements faster than anyone can track, especially in regulated industries like financial services.
Sajjad raises the human bottleneck: agents produce more than he can read, comprehend and approve. Angus argues the answer is narrowing context. Feeding the spec agent your current product capabilities, codebase and users produces a far more relevant spec, broken into chunks a person can decide on. One client even ran out of features for the agents to build, because with enough tokens the limit is no longer compute but the team agreeing on what to deliver. Angus sees the product manager and engineering roles merging into an "AI product builder" who prototypes directly with agents and hands off to an architect for technical design.
Sunil recalls a previous guest's point that errors compound when agents are chained. Angus agrees and gives an example: a coding agent paired with an over-conservative review agent can spiral into 70,000 lines of tests for a few hundred lines of feature, so a human must decide where to stop. Sajjad adds that agents are trained to build full enterprise architecture even for a simple test. Answering Sam's questions on reviewing a 2,000-line diff, Angus describes spec-driven development: clear requirements, acceptance criteria and an explicit out-of-scope list; tests written from acceptance criteria first, going red to green; and agents held accountable to those criteria rather than humans reviewing every line, with a report of anything extra they built and lessons fed back into future specs. Tests run in scoped batches to avoid blowing the context window, and new tests are based on requirements, never the code, so agents can't make a failure disappear by editing the test.
For regulated clients such as an asset manager, Angus says every answer must link back to a source of truth, with a confidence level and every contributing source shown so a compliance officer can verify it, and responses logged alongside their sources. Revisiting last week's story of an agent with root access deleting backups, Sam notes some system work genuinely needs root. Angus says Captomi's agents never touch production databases, but in development an agent once wiped all the dev data while cleaning up rows, and a fleet of agents once burned through about $20,000 of tokens and compute in three days. The episode closes on Sam's question about what happens to senior and junior engineers. Sajjad sees a glass half full, with software still to be built for nearly every sector, and compares AI to the compiler that freed engineers from assembly. Angus, who no longer writes code after 20-plus years, advises seniors to focus on design and architecture and juniors to learn programming basics plus design patterns, architecture and infrastructure.
Chapters
- 0:00 — Welcome, a new office, and guest Angus Dunn
- 1:14 — A fleet of agents, from requirements to production
- 2:46 — What clients want: speed, then quality
- 3:54 — The human bottleneck: reading what agents produce
- 4:44 — Narrowing context for better specs
- 6:09 — Specifying precisely in natural language
- 7:05 — Running out of features, and the team decision
- 8:58 — Human bottlenecks and the future of requirements
- 10:13 — Product and engineering merge: the AI product builder
- 13:03 — Chained agents and compounding errors
- 14:24 — Over-conservative reviewers and 70,000 lines of tests
- 15:46 — Agents that build enterprise architecture for everything
- 17:41 — Can a human review a 2,000-line diff?
- 18:44 — Spec-driven development: acceptance criteria and out of scope
- 20:03 — Hold agents accountable instead of reviewing every line
- 22:16 — Context windows and running tests in batches
- 23:45 — Agents that fix the test instead of the bug
- 24:59 — What auditors want in regulated industries
- 27:39 — Revisiting the root-privileges story
- 29:09 — Agents gone wrong: wiped dev data and a $20,000 bill
- 31:07 — What happens to senior and junior engineers?
- 32:04 — Glass half full: there's still a lot of software to build
- 34:01 — AI as the new compiler
- 34:55 — Angus's advice for senior and junior engineers
- 36:30 — Wrap-up
Full transcript
All opinions expressed are those of the individuals themselves, not necessarily of any organization they work for.
Welcome, a New Office, and Guest Angus Dunn
Sunil: Okay, welcome to The Token Drop. We here at DaaX have moved into a new office. We don't have our internet hooked up yet, so we're working off hotspots. Forgive us if the video or audio is a little choppy; hopefully that will be fixed next week. Today we have a special guest, Angus Dunn, the founder of a company called Captomi. I met Angus a few weeks ago at a Tech Week meetup in San Francisco, got to chatting with him, and found what he had to say really interesting, so I thought he'd be a good guest for our podcast. All opinions expressed are those of the individuals themselves, and not necessarily of any company they work for.
Sunil: So, Angus, you have a lot of experience using a fleet of agents to implement solutions for customers. Maybe you can share some of your experiences with us and the audience: the good, the bad and the ugly.
A Fleet of Agents, From Requirements to Production
Angus: Well, first of all, thank you, Sunil, for having me on the podcast. We appreciate the opportunity to talk to the team here, and to the audience. Regarding the fleet of AI agents we provide for our clients, let me give you a bit of background first: what it's about and what it does.
Angus: Our clients are mostly software companies, or businesses looking to roll software out into production for their users. We provide a fleet of coding and development agents to help them build that software fast. One very common theme when I talk to clients is that they really want to leverage frontier models like Claude Code and Codex to boost the productivity of their engineering teams, and to develop features fast and get them into production. That's where the fleet of AI agents comes in. Our agents take the product specs and requirements from the business, then do the design, coding and testing, deploy to a staging environment, run through testing there, and deploy to production. It takes things all the way from a simple prototype idea and requirements to production for the users.
What Clients Want: Speed, Then Quality
Angus: Some of the interesting things I've learned from our clients: number one is speed. They really want to get it out before anyone else does, and speed is a very critical factor when they think about using AI to develop code. The second is definitely quality. Does it meet the requirements? That's much more difficult, because we're talking about agent speed. With a fleet of AI agents, you might have eight or ten VMs running multiple agents at the same time, implementing so many requirements in such a short time that you can't even tell where the quality stands. That's one of the challenges we face with clients, and we do have solutions to make sure the quality is there, along with scalability, performance, and even that the AI's responses fit within benchmarks and guardrails, especially in regulated industries like financial services.
The Human Bottleneck: Reading What Agents Produce
Sajjad: Angus, thanks a lot for describing what Captomi does. One question I had, and this is something I'm challenged by: it's one thing to ask Claude Code, or any other agent for that matter, to think through a problem and come back with a solution. But the challenge for me, personally, has been simply reading the volume of content it produces.
Angus: Yes.
Sajjad: You're trying to make decisions fast and move fast, but I'm finding I'm limited by my human ability to read, comprehend, review, understand and then give the go-ahead. That's been my challenge. How does Captomi address that?
Narrowing Context for Better Specs
Angus: First of all, we have spec agents that write the specifications, but that relies on one thing. When you just ask Claude Code to vibe code a product or a feature, it goes all the way out, figures out what needs to be done and gives you a huge volume of information to digest. That doesn't help, because you don't know what the end product will look like. What you need to do is narrow down the context to what's specific to your business: What are your current product capabilities? What does your current codebase look like? What are your requirements, and who are your users? Feeding that context into the specification produces a much higher-quality spec that you can actually use, because it's related to you and your business.
Angus: You also need to be able to understand it fast, and that's where the specification agent comes in. It digests the information for you so you can consume it in chunks, rather than a whole wall of text where you have no clue what Claude is talking about. Then you can make informed decisions: you want point one and point two, but not point three.
Specifying Precisely in Natural Language
Sajjad: Yes, that's largely true. In fact, you hit on a really interesting point: how do you specify accurately? We're no longer communicating in code; we're communicating in English, or any other language for that matter.
Angus: Yep.
Sajjad: And the natural language we use to communicate may not be exact or precise. When you're specifying something in a conversational mode, it's hard to be precise, accurate and comprehensive. These agents' ability to get things done is a function of your own ability to specify exactly and accurately. So what are your learnings in that space? What can you share with us?
Running Out of Features, and the Team Decision
Angus: I want to emphasize another point you made, which is pretty valuable and actually happened to my client too. Because AI agents are so capable, one of my clients said, "We don't have enough specs for the AI agents to work on. We ran out of features." That was so true, because it really comes down to compute. If you pay AWS or GCP and have enough tokens, you can crank out a lot of things fast.
Angus: Now, back to the human part. At the end of the day, the features you want Claude Code or agents to develop come down to a decision humans need to make, and most likely not a single person but a team. That discussion within the team needs to happen first, because you need to agree on what will be delivered. Whether you capture it in a meeting transcript, emails or Slack messages, any of the AI models can summarize it into a really clear picture of the outcome. Then you can feed that to an AI specification agent, mine or Claude Code directly, and get a spec that's very specific to the agents you want to use. That drives the outcome you're looking for. One saying I find really interesting: when humans talk to humans, we need simple ways to understand and communicate. Agent to agent, we don't really need to care, because they'll talk among themselves anyway. That's a completely different spec. Let them deal with it.
Human Bottlenecks and the Future of Requirements
Sajjad: Yes, indeed. I'm just calling out what I think are potential bottlenecks: the human ability to express accurately and precisely what you want built, and then the human ability to review. Now you're limited not by machine speed, but by human comprehension and the human speed of review and approval. That's one area.
Sajjad: The other area, speaking of specifications: in the past, product managers wrote out a long list of functional requirements, and from those you'd derive performance, scalability, security, cost and various other requirements. How exacting do these requirements need to be? And what are you seeing at the frontier? Are you seeing auto-formalization capabilities, formal verification techniques and so on?
Product and Engineering Merge: The AI Product Builder
Angus: You bring up a very interesting point. As you said, product managers used to come up with the functional requirements and hand them to engineering, and then we'd write the technical spec, analyze it, and work out the security issues we might encounter, scalability, performance and the functional capabilities, and how it all fits into the existing code. Then there'd be another round of architectural review of how it fits into the overall architecture of the codebase and product. A lot of that has been short-circuited, I have to say. I've seen it multiple times: run Claude Code or agents through the entire codebase and infrastructure, and they do a very thorough analysis of security and scalability, plus an architectural review. It's pretty solid, and it keeps getting better as newer, more capable models come out.
Angus: The more interesting part is: what is the role of the product manager now? What's the role of engineering? I think the roles are merging. A product manager no longer just writes functional specs. A product manager can build a prototype, taking the existing codebase and product capabilities and quickly running them through Claude Code or AI agents to visualize it, even test it, and see whether it's what he or she expects. And he or she can quickly change and tune it. Once that's narrowed down, it goes to a second round, where a more technical person, maybe an architect or a senior engineer, applies the technical design, making sure it fits the existing architecture, or a new architecture and infrastructure. That has been compressed significantly.
Angus: I've seen companies, including clients, merging the two roles. They don't call it product and engineering; they call it the AI product builder: someone with engineering knowledge who takes on product management. Combining them is a lot more effective, because you no longer waste time on communication. One person or one team decides what the outcome and the product will be, and the productivity of the entire team increases significantly.
Sajjad: Yes.
Chained Agents and Compounding Errors
Sunil: Here's a question. You talk about these fleets of agents, one agent talking to another talking to another, and that reminded me of something Nav Mathur talked about on one of our previous podcasts. When you chain these agents together, the error rate multiplies out. If one is 95% accurate, times 95% for the second and so on, by the time you get to five, I think he said you're down to fifty-some percent accurate.
Angus: Yes.
Sunil: So how do you account for that?
Over-Conservative Reviewers and 70,000 Lines of Tests
Angus: Yes, that happens a lot, I have to admit, and 95% might even be an understatement. Let me give you a solid example I've seen many, many times. Say you use one agent to do the coding and another agent or model to review it. The coding agent listens to the review agent and adjusts accordingly. Initially that sounds pretty good, because you have independence, a kind of jury or judge reviewing the feature and driving the appropriate changes. But what if one of them is over-conservative? The original coding agent goes down a rabbit hole, refining and adding more tests, and you might end up with 70,000 lines of test code while the actual feature is a couple hundred lines.
Angus: If you don't stop it, you get into that rabbit hole. That's where you need to make the cut, and that comes down to a human decision. You'll probably do 80% and leave the last 20%: the 80/20 rule, simple as that. That's a major learning on this journey of using AI to code for you, or even review. It's not easy at first, but having worked with so many clients, we now have patterns and learnings captured that minimize it. But we can never make it 100% accurate. That's inherent in all the AI models we use, and in all the agents too. So there will always be a human in the loop to make the judgment call: this is where we stop. And that's different for each company, each product and each feature.
Agents That Build Enterprise Architecture for Everything
Sajjad: I've seen that happen all the time. These agents are trained to be exceedingly good engineers. No matter what task you give them, they go back to enterprise architecture: we need to build for security, upgradability, manageability, supportability and everything else. In a very short time, even though you're trying to do something small and simple, you end up with boatloads of code. On the face of it, it looks good and functional, but when you reflect on it, is this what I wanted? Maybe not. I just wanted something really simple to test or verify something, and now I have a boatload of code. These agents are trained not to leave any gaps, so they cover what they do as exhaustively as they can and build a very complex enterprise architecture.
Angus: That is so true, and it's one of the major learnings across all my clients. How do we stop all this slop? When AI agents do the coding, there's so much of it, and some of it is completely unnecessary, wasting time and compute. That learning becomes a feedback loop to the agent itself: when it detects this happening, either it invokes a human to act on it or, if it's smart enough, it stops itself.
Sajjad: Yes.
Can a Human Review a 2,000-Line Diff?
Sunil: Sam, you look like you've been wanting to say something.
Sam: Yeah, let me ask a couple of questions. You're talking about review across the entire SDLC. Can a human meaningfully review a 2,000-line diff? I've seen this myself: Claude generates a 2,000-line module. How can anyone meaningfully review that code in ten minutes? So do you review the code, or review the agent's plan before it writes anything? And how can you make that more focused? I've heard about spec-driven development; can you elaborate on that? And do you have a second model review the first model's code? What does it catch that humans miss?
Spec-Driven Development: Acceptance Criteria and Out of Scope
Angus: Very good questions. You're absolutely right that the fleet of agents we provide models the SDLC, so it has coding and review, and continuous review to make sure everything ties back to the spec. I'll answer in two parts. The first part is the spec itself, because that's where all the building and coding starts. The spec must be really solid and clear. By clear I mean you have not only the requirements but also the acceptance criteria: what's acceptable. You also need to state what's out of scope, the things you're not going to deal with, so the agent that picks it up doesn't build something out of scope. That's extremely important; otherwise you'll have a group of agents building 70,000 lines of code for you. From the acceptance criteria you get the corresponding test cases. A typical workflow for us is to implement the test cases first, according to the acceptance criteria, and then go from red to green. That's pretty typical for most AI-assisted development anyway, so that part doesn't change.
Hold Agents Accountable Instead of Reviewing Every Line
Angus: The second part is review, and you're absolutely right: there's no way humans can review 70,000, or even a thousand, lines of code. It's just too much. This is my personal take, but I don't think humans should be reviewing agents' code line by line, because we just can't keep up. We become the bottleneck, and we shouldn't be. What we can do is make sure the agent is accountable for what it delivers. How do we measure accountability? Against the acceptance criteria and the requirements: does it go green on each of the acceptance criteria? That's how you verify it.
Angus: If things are green, you also need to do one more thing: ask what extra things it did. It will scan the code it produced and tell you. That's what you need to focus on and review. If the extra work is more than you need, that's probably not a good sign. Maybe it's something you missed in the spec in the first place. That becomes a learning: go back to the original spec, or apply it to future requirements. So I'd go one step beyond review, because review is good for that particular feature, but you also want learnings to apply to future development. That's where benchmarks and learning come in. A benchmark means: if I implemented the same thing using a different approach, would the outcome be better? You probably want to run the same thing twice. I know it costs a lot more tokens, but it gives you a benchmark, based on a judge and a rubric, for gauging which outcome is better. That feeds the learning. For learning, we use memory and GitHub issues; some clients use Jira or store their learnings in a database. Those are what you apply when you write specs for future features.
Context Windows and Running Tests in Batches
Sam: When you generate tests, that becomes a very interesting problem. Say there's a 2,000-line module and you're generating tests for it. Your context window is limited. Some modules are extremely complex, and you want a full integration suite, and boom, in the middle it fails because the context window has blown, and all it returns is the test stubs, which I've seen.
Angus: Yeah.
Sam: In my case, the pytest file comes back with no content behind it. How do you put guardrails around that?
Angus: That one is relatively simple to address. I wouldn't say simple, but yes, it's one of the challenges we've seen. We have a lot of tests to run: unit tests, evals, end-to-end tests, journey tests, different sets of tests. The way we run them is that each feature is scoped, so it has a specific set of tests to run. It's never one big batch, because even a full batch of scoped tests would still run out of context. We run them in batches, so we know what fails and where. One of the challenges with running tests is that they burn a lot of tokens too. I'm not sure if you've encountered that.
Agents That Fix the Test Instead of the Bug
Sam: The more important thing is that the agent can edit those tests to make them pass. When your CI/CD pipeline is running and you're doing root-cause analysis, and in fact I talked about this last time with Sunil and Sajjad, when you show it a problem, it makes it go away by fixing the test rather than the real problem. How do you deal with that?
Angus: That is so true. During the coding process, we run the tests as we code, in small chunks. And one thing before I forget: whenever we touch an existing feature, there should already be tests running against it. If we change the feature, we change the tests to match the new requirements, and the changed tests must pass; the previous tests become the red tests going from red to green. If it's a brand-new feature, the tests cannot be based on the code. They have to be based on the requirements and the acceptance criteria. That's extremely important. If the tests are based on the code, that's when you get agents fixing the tests to match code that has gone wrong.
Sam: Right, an agreed-upon interface.
What Auditors Want in Regulated Industries
Sajjad: Since you operate in regulated industries, what artifacts would auditors like to see? Now code has been built by AI, tested by AI and deployed by AI. Are auditors very concerned, and what do they want to see to convince themselves that this is a compliant system?
Angus: Let me give an example, and then a more general answer. One client we work with is in asset management. A user asks, "What's the performance of this investment strategy?" For a high-dividend-growth strategy, say, what are the numbers? Is it 5% quarter over quarter, and what caused that 5% increase or decrease? That information has to come from somewhere; it can't be something a model generates. So we have a vault where we store the information, and whenever a user asks a question, we always find the source of truth and link to it. If we're looking for that 5%, we know which document or database it came from, and we have a direct link to it. In some cases, for every single answer, we need to provide the source and how confident we are. Because it's a regulated industry, an exact match is great; everyone trusts it, because they can go directly to the answer. If it's a possible match, we need to work out where the answer comes from. Is it from multiple sources? If so, the model has compressed or summarized them, so we show all three sources, and a human can go to those three sources and review them. That's what the compliance officer is looking for.
Angus: The more general answer is: always find a way back to the source of truth, record it and show it to the end users. And of course you keep audit trails behind the scenes: you log the response, along with the source of truth associated with it and when the answer was generated.
Revisiting the Root-Privileges Story
Sunil: It was interesting: last week we were talking about this, and Sam, I think you gave the example of an agent with root privileges that wiped out all the backups.
Sam: Basically, it ran out of disk space, and the agent decided the wise thing to do was delete all the backups to create space.
Sunil: And I was asking, why would you give root privileges like that to an agent? I'm still not sure why you would.
Sam: At the system level, some of the things you do really require it. Technically, as a backup user, you wouldn't need it. But if you're looking at it from a system perspective, analyzing memory, say, you still need system privileges to push data to the dashboard and look at what's going on. Some of these really require root privileges on Unix systems.
Sunil: Yeah, you're right.
Sam: At that point, it's inevitable: based on the goal, it can over-optimize.
Sunil: Some jobs can't be done without root, I understand.
Sam: Exactly.
Agents Gone Wrong: Wiped Dev Data and a $20,000 Bill
Sunil: My question back to Angus: have you seen anything like this? Have any of your agents gone absolutely berserk and done something you didn't expect? Got a good story for us?
Angus: Absolutely. I have multiple good stories. But before the interesting ones, let me give you the boring one. None of our AI agents can touch a production database. Period. They can't update anything or delete anything there, especially delete. We don't give that access, because it just doesn't work, and none of our clients do that either.
Angus: The interesting stories happen in the development environment, because we do give root access there. In my own experience, I had an agent that set out to clean up some rows, and it cleaned up the rows and all the development data too. Wiped it. It was starting from a clean slate because columns had changed and it needed new data to populate and connect everything up. So that did happen. The second one: a fleet of agents was building so fast and spinning up so much that it consumed enough tokens and compute to cost us, I think, about $20,000 in three days. I was like, wow, how did that happen? So we had to put in a lot of guardrails to make sure it doesn't happen again. We learned our lessons the hard way, too.
What Happens to Senior and Junior Engineers?
Sunil: We're almost out of time. Sam, we have just a couple more minutes.
Sam: If agents write 90% of the code, what does a senior engineer's day look like today, in your view? And how do junior engineers learn to code when agents do the coding? Those are two different questions: what happens to senior engineers, and what happens to junior engineers? This is a bigger problem for us. Think about the semiconductor industry: we sent everything to China, and now we don't produce much here in the U.S. There's a cost to everything we do. So what's your view?
Sunil: Sam, if you're really elevating yourself by not having to write the code, why do you need to know how to write code?
Glass Half Full: There's Still a Lot of Software to Build
Sajjad: Sam, what I've seen is that it's like viewing the world as fairly limited, looking at the glass as half empty. From a more optimistic perspective, the glass is half full, and I can fill it up even more. Think about it: do we have software for every use case, every application, every sector of human activity? The answer is no. Somebody needs to go out and build all that software and all that automation, and you're going to need people, machines and architects. If you see yourself as a limited-scope company with one product, thinking, "Now that I have AI, I can cut my engineering team in half," that's a very limited way of looking at it.
Sunil: Two hundred years ago, I probably knew how to grow wheat. Now I can't even grow tomatoes without them being eaten by squirrels. But I don't need to; that's the point. So why do I need to know how to code?
Sam: This is where a lot of people who have been coding for years have a problem, because they're experts in that domain. Having spent 30 years building stuff, they know how to build really solid code, and when AI produces code, they get very nervous. I've seen this with a lot of senior people. They say, "How can I believe this thing is doing what it's supposed to do?"
AI as the New Compiler
Sajjad: But it's the same thing, Sam. Remember, in the past people wrote assembly.
Sam: Correct.
Angus: That's a good one.
Sajjad: Then compilers came along, and engineers were told they didn't need to know assembly, just a high-level programming language. Imagine how confused those engineers must have been: "I can't optimize the register movement of my data anymore; the compiler will do that. Will it do a good job? How do I trust it?" But eventually we all learned to use compilers.
Sam: Exactly, it's very true. And now Angus is telling people it's spec-driven development.
Sajjad: Right. AI is the new compiler: it takes a spec and generates code. It's another tool in the armory.
Angus's Advice for Senior and Junior Engineers
Angus: I agree. I've been coding for 20 or 25 years, and these days I don't code anymore. The way I see it, my abilities haven't gone anywhere. My past experience helps me understand what I can do with these frontier models and coding agents to achieve the same goal, which is delivering features faster. My role evolved, along with my responsibilities and my tasks. What used to be typing C, C++, Java or, these days, TypeScript, I no longer do by hand. Now I can focus on the design and the technical side, and on how to scale it and build more features faster.
Angus: So for senior engineers, to your question, Sam, I'd recommend upskilling: instead of focusing on writing the code, learn more about design and architecture, and how to apply different architecture and design patterns to the problems you're solving. For junior engineers, I'd recommend starting with the basics of a programming language; you need that anyway. Beyond that, start learning technical design, design patterns, architecture and infrastructure, and with the help of frontier models you'll be able to catch up really fast.
Wrap-Up
Sunil: I think that's an excellent way to end, and we've run out of time. Thank you, Angus. If you could just hang on a minute. Okay, great.