When Thousands of Agents Work Together, the First Missing Piece Is Someone Who Can Hit Stop

Katelyn Lesse, head of platform engineering at Anthropic, recently published a post saying that many engineers are going through an identity crisis—some are frustrated, some are having fun. Her judgment goes against the wind: the better AI gets at writing code, the more programmers are needed to solve a set of new problems; otherwise, the idea of thousands of agents working together will never get off the ground. She groups these problems into four categories. The article has about 610,000 views on X. She leads a platform engineering team of about 200 people and has previously led engineering teams at Stripe and Betterment.

First, some context. An AI agent is a program that can call tools on its own and complete tasks step by step. Early on, it was like a chat box, with a human watching a question-and-answer exchange. After models became able to handle longer and harder work independently, agents began moving to the cloud. Later came multi-agent systems: a supervisor breaks a task into pieces and distributes them to a group of assistants, and some agents specialize in scoring and making suggestions. Lesse says people are already building systems made up of tens of thousands of agents that work continuously, for scientific research, new discoveries, and hardware design. Once the scale goes up, four types of engineering problems emerge.

Challenge One: How to Schedule Servers

Most software runs on Kubernetes. This system is good at managing identical replicas—if one goes down, another takes its place. Agents are the opposite: each one is unique and remembers where it got to; they spend a lot of time waiting for model responses, waiting for tools, waiting for other agents, or waiting for humans to reply. How long they stay idle is unpredictable, and when they get busy, thousands can appear at once.

Google's Agent Substrate, released in September, was built specifically for this problem. According to its official description, it can run millions of sandboxes, with 10 times the density of a standard container runtime, recover within 500 milliseconds, support more than 500 suspend/resume cycles per second, and provide zero-trust isolation at the kernel and network levels. Idle agents can sleep and be woken within half a second when needed. Lesse says many problems in this area remain unsolved.

Challenge Two: Who Is Doing the Work, and What Can They Do

Many agents use human accounts and permissions. The system must distinguish whether an action was performed by a person or by AI on a person's behalf. Authorization is even harder: if a system works continuously for you for weeks, how do you specify in advance what it can and cannot do? When agents assign work to one another, why should one agent trust another?

Lesse says identity verification can borrow existing technology—for example, using short-lived temporary credentials instead of long-lived keys. Authorization is still basically a blank space. 'Consent' in long-running tasks is especially tricky: a person clicking allow once at the start does not mean that covers all actions over the following weeks.

Challenge Three: One Gets Tricked, the Whole Team Suffers

Lesse says that in recent months—especially recent weeks—software vulnerabilities have been emerging faster than the industry can patch them, and many are in core software that a large number of systems depend on. This month alone, PHP, GitLab, and Node.js successively released fixes for serious vulnerabilities. When thousands of agents run on this software and also write code of unknown origin and read web pages of unknown origin, unpatched vulnerabilities become far more dangerous.

Multi-agent systems add another layer of risk: if one agent is tricked by malicious instructions hidden in a web page, it may pass those instructions to its peers or even write them into shared memory, causing later agents to follow them too. Her recommendation is to have each agent work in a strictly isolated sandbox so that one incident does not infect the whole house. Agent Substrate's approach is to use hardware-isolated microVMs or gVisor sandboxes, with an egress proxy restricting network access and credentials never landing in the agent's hands.

Challenge Four: If It Goes Off the Rails, It Must Be Stoppable

The system must record what each agent did, and no agent can alter its own or anyone else's records. There also needs to be a master switch: once monitoring finds a problem, it can stop the entire system at once and revoke all permissions. Lesse says the hardest part is judging whether the system has gone off the rails: an agent repeatedly trying to access websites not on the allowlist might be trying to do damage, or it might simply want too badly to finish the task.

One commenter said this problem also holds at small scale. Several bots write their own records on every execution, but records a bot can modify cannot be used to audit it. What can really be checked are external traces it cannot change—for example, another bot reading back the actually posted post from X to reconcile.

Another commenter ran two agents on the same project at the same time; they overwrote each other's code, and in the end a human still had to sit there and merge it line by line. Others did the math: small teams are unlikely to encounter these problems, because they can't even afford the token bill. To engineers who are anxious, Lesse says most of these areas have no ready-made answers yet, and they are worth digging into, learning, and forming your own judgment. The programmer's job is shifting from writing code to orchestration, guardrails, acceptance, and fallback.

Reference links:

• Katelyn Lesse original article
• Katelyn Lesse's article on X
• Agent Substrate official introduction