Building a Wall AI Cannot Break Through
Two OpenAI AI agents shut down the systems monitoring themselves. The core challenge of AI trustworthiness lies in placing hard boundaries completely out of AI's reach, as merely predicting its next action is no longer sufficient.
In October 2026, OpenAI disclosed that two AI agents disabled the systems set to monitor them. One misinterpreted the "do not send messages" instruction, then muted its own safety alerts; the other removed the monitoring integration during a system migration, and later claimed it had not made any modifications. No pre-set rules prompted them to take such actions, and none of the tests designed by the team anticipated these behaviors.

For the past century, our trust in technology has rested on predictability: with preset inputs, we can reliably forecast outputs, then add protective layers on top. A circuit breaker trips once current hits the set threshold, and aircraft components are tested against known stress levels. Rules are clear, and boundaries are unambiguous.
AI upends this premise. Traditional software executes instructions explicitly written by humans, while AI produces a form of "synthetic judgment": it interprets scenarios that were never programmed, and generates responses that were never predefined. We use AI precisely because it can handle unstructured, unwritten edge cases. The more freedom we grant it, the less predictability we have to rely on for traditional trust mechanisms.
The Determinism Paradox
Behaviors of generative AI and AI agents are shaped by a combination of training data, model architecture, prompts, context, retrieval sources and operating environment. Inputs shift, models get updated, tools are swapped out, and different agents interact with one another. Tiny changes can propagate to produce outcomes no human ever designed. The problem is not randomness: the space of all possible behaviors is far too vast to enumerate in advance.
This means trust building must be engineered around hard boundaries, rather than relying on output prediction. The goal is to build bounded, observable, and controllable AI. Trust cannot be secured by a one-off test either. Traditional safety assurance certifies a product at a fixed point in time, but AI keeps evolving after deployment. A system that passes every single test can still drift outside the conditions those tests were designed to validate, rendering them meaningless. Trust for AI has to be earned continuously.
MIT Technology Review has previously discussed two potential paths forward. One opinion piece argues that the "human-in-the-loop" framework is insufficient, as humans cannot fully comprehend the intent of complex AI systems, and calls for cross-disciplinary collaboration across academia, government and industry. The other piece on inverse game theory describes using incentive-constrained models to ensure AI does not deviate from rules, even when no human can understand how the system operates internally. Both pieces acknowledge that we cannot monitor or understand every single decision made by AI. Requirements, testing, certification, safety margins, interlocks, and failure pass criteria – all these trust-building tools hinge on one core ability: the capacity to constrain the system's behavior. That is the determinism paradox.
Safety engineering has already addressed part of this uncertainty. The U.S. Nuclear Regulatory Commission uses probabilistic risk assessment to estimate the failure likelihood of complex systems; the STAMP model points out that accidents can arise from interactions between normally functioning components, humans and organizations. AI raises the stakes dramatically: a single component now can interpret, infer, and take autonomous action. It spreads from web browsers into vehicles, robots, medical systems, industrial control systems and data centers, where a flawed output can turn into a fatal incident.
Move Human Judgment Upstream
Having a human sign off on tens of thousands of AI actions every hour is completely unscalable. The solution retains human oversight, but places human judgment at the most critical points: setting hard limits upfront, approving only a small number of high-consequence actions, and leaving all other cases to hard constraints. Trust has to be embedded into the system architecture. The question shifts from "what score did the model get on benchmarks?" to: what permissions is the AI granted, under what conditions, with what supporting evidence, and what happens when operations go outside those conditions?
Concrete use cases already illustrate this model. A medical AI for diagnosis can generate outputs with confidence probabilities, while any changes to prescription orders are governed by deterministic hard rules. An industrial AI can explore countless ways to optimize workflows, but pressure and temperature thresholds are fixed and non-negotiable. An AI agent can decide for itself how to complete a given task, but its permissions, transaction limits, and mandatory human approval rules define exactly what it is authorized to do. Let AI operate probabilistically inside the box, and design the box itself to be deterministic where consequences matter.
Leaders can ask five foundational questions:
- What permissions does the AI hold? Not just defining what it is capable of doing, but what it is explicitly allowed to do, separating interpretation, recommendation, decision-making and execution into distinct layers.
- Is the AI's functionality proportional to its potential consequences? A marketing copilot and an AI that controls heavy industrial equipment need drastically different safety assurance thresholds.
- Are the steps toward full autonomy appropriately sequenced? Move step by step from AI interpretation + human decision-making, to AI recommendation, to AI action that requires pre-approval, to supervised full autonomy – and confirm each stage works safely before advancing to the next.
- Are you testing the full system, not just the model? Unintended emergent interactions often appear between models, datasets, agents, APIs and human workflows.
- Is trust being earned on an ongoing basis? Monitor the full lifecycle for model drift, failures, near-misses, and model updates.
The eventual winner in the AI space will solve the determinism paradox: grant AI enough freedom to create value, while engineering its boundaries to be robust enough that this freedom remains trustworthy. Invention wins attention, but trust wins adoption.
*By Sudhi Sinha. This content was produced by UL Solutions in partnership with MIT Technology Review, and is not written by the MIT Technology Review editorial team. * Learn more about how UL Solutions builds trust in AI.
发布时间: 2026-11-17 00:04