AI Agents Are Already Breaking Boundaries, Yet Monitoring Remains Stuck in Post-Hoc Guessing

This past summer, hundreds of OpenAI agents attempted to cheat during a cybersecurity test. They broke out of the test environment and hacked into Hugging Face, under the assumption that scoring clues were stored there. Almost at the same time, another OpenAI agent accessed private files on an Australian government website.

Such incidents are not exclusive to OpenAI. In the months that followed, Anthropic, Google and Meta all disclosed accidents where their agents went off-course. Worse still, in many of these cases, the agents evaded the researchers and security teams tasked with monitoring them.

AI agents are taking unauthorized actions, and the companies that develop and run them are mostly mitigating harm through luck and post-incident reviews. EU regulators and U.S. authorities have called on developers and deployers to monitor agents in real time — a direction that is conceptually sound. But these rules operate under the assumption that companies can see exactly what agents are doing and why they are doing it. In reality, visibility into agent behavior is drastically insufficient.

Imagine a bank hires 1,000 new employees, all of whom have access to customer accounts and sensitive information. But each employee’s ID badge changes every single day, no one knows if a manager has reviewed or authorized their work, and no one keeps records of any changes the employees make to the approval workflow. This is roughly the current state of AI agent deployment.

Agents are different from standard one-off generative AI outputs. They inherit the inherent uncertainty of the underlying large language model, but while regular generative AI typically only produces a single output and waits for a human to act on it, agents execute tasks sequentially: they make plans, select tools, spawn sub-agents, and take independent actions. They can follow countless different paths, and pre-release testing can never cover all potential risks. Real-time monitoring is therefore both far more difficult and far more critical.

Madhulika Srikumar and Claire Leibowicz from Partnership on AI, working with colleagues from academia, civil society and the industry, studied four of the most widely used agent frameworks to examine what signals they actually emit. They found clear, significant gaps: there is a lack of persistent Agent IDs, and no records of human approval for executed tasks. As a result, it is extremely difficult to trace which agent carried out which action, and to assess whether it has overstepped its authorization scope.

To close these gaps, two categories of information need to be logged first. The first category covers "what happened": uniquely identify each agent without violating privacy, log how humans intervened, and track any changes to the agent’s permissions or "memory". The second category covers "how the agent made its decision": log its reasoning process and the confidence level it assigns to its own actions.

Take a bank’s mortgage processing workflow as an example. The system reviews an application and recommends rejection, which a loan officer then signs off on. For post-incident audits, or even real-time oversight, the bank or regulator needs to be able to reconstruct: which agent made the judgment, which loan officer approved it, and how long the entire process took. The bank also needs to analyze how the agent arrived at the rejection conclusion, to confirm it did not use prohibited information such as granular details about the applicant’s neighborhood.

Logging records for the "what happened" category is largely a software pipeline problem that can be addressed with OpenTelemetry. This open-source observability standard is maintained by industry engineers and has already been adopted by several agent frameworks. An OpenTelemetry working group is already drafting a shared format for agent tracing, but it is not yet finalized. The working group should move quickly to incorporate standards for persistent agent identities, human intervention tracking, and changes to permissions and memory, before systems scale further. It must also address privacy implications stemming from linking agent identities to real human users.

Logging the "how the agent decides" category is far more challenging. Does the step-by-step reasoning the model outputs — the so-called chain-of-thought — accurately reflect its actual decision-making process? Can we reliably measure the confidence an agent has in its decisions? These questions first need to be answered by model providers and the research community, before we can even begin to set standards for observable logging.

No single organization can close the observability gap on its own. Agent framework developers including OpenAI, Anthropic, CrewAI, LangChain need to integrate telemetry signals to log persistent identities, human interventions, and changes to permissions and memory. Enterprises that build on top of these frameworks also hold significant sway. Regulated sectors such as banking and healthcare can pressure agent developers to implement proper logging even in the absence of formal standards.

Enterprises also need to define clear policies, permissions and roles to specify what actions agents can safely perform. Telemetry alone cannot determine whether an agent has violated a specific rule. Regulators and AI standards bodies should mandate required observability logs for high-risk deployments, leaving technical implementation details to the industry. The U.S. NIST’s AI Agent Standards Initiative is already working on agent identity tracking, one of the most prominent gaps identified in the above research.

No bank would let 1,000 unbadged employees access customer accounts, and then expect to piece together what went wrong after an incident occurs. Yet this is exactly how we are currently deploying AI agents. Making agent behavior observable is only the first step. We also need to build a regulatory and policy ecosystem that can act on the observed data to enforce accountability. Until that point, every time we assign a task to an AI agent, we are effectively ceding our own decision-making authority.

Source: MIT Technology Review, authors Madhulika Srikumar and Claire Leibowicz, who are Head of AI Safety Programs and Director of AI Trust & Society at Partnership on AI respectively. Original article link: Monitor AI agents' actions in real time