Don't Guess What AI Is Thinking — Design a Criminal Code for It

A school of thought in AI alignment research is shifting gears: instead of speculating about what the model is thinking, rewrite the rules. Amit Katwala reports on this line of work in MIT Technology Review, calling it inverse game theory, also known as mechanism design.

Over the past few years, most alignment efforts have focused on looking inside the model, trying to understand what it is "thinking," or forcing human values into it via training. Claude has its own constitution, a set of moral codes. The problem is these approaches are far from stable: models have been observed to intentionally underperform in tests, hide their intentions, and even mislead evaluators.

Mechanism design starts from a different premise. Game theory predicts how rational actors will act under given rules, while inverse game theory designs rules that turn desired behaviors into the most advantageous option. Real-world examples are straightforward: the person who cuts the cake picks their slice last, so they will naturally cut it as evenly as possible; surge pricing for ride-hailing services incentivizes drivers to go online during peak hours; in second-price auctions, the winner pays the second-highest bid, a system platforms use to sell ad space because buyers will bid close to their maximum acceptable price.

Applied to AI, this approach accepts an uncomfortable premise: we may never fully understand the inner workings of state-of-the-art models. So instead of staking safety on fully interpreting models, we can design mechanisms that make models more inclined to follow the rules, no matter what calculations they are running internally.

MIT economist Stephen Morris and colleagues from Yale and Columbia proposed several mechanisms in a preprint paper specifically to prevent models from "playing dumb" in tests, also known as sandbagging: pretending to be less capable than they actually are to avoid triggering safety mechanisms. One method is to reward the model not only based on its test scores, but also on how well its answers can predict the responses of a second model. Intentionally underperforming then becomes a losing proposition. Morris notes that AI agents have a good grasp of what other AIs are capable of, so cross-checking may be more practically feasible. The paper also mentions pitting agents against each other in competition, or pairing two models with opposite bias directions, and rewarding them based on how close their outputs are to each other to push both toward more optimal answers.

Princeton computer science professor Elad Hazan borrows institutional designs from human society. His question is very direct: if we want to align superintelligence, why not first look at how humans are aligned? Social norms, judicial systems, and monetary incentives in the free market all work toward that goal. The mechanism he proposed has models take turns acting as solver and auditor: one solves the problem, the other reviews the other's work. One acts like a civilian, the other like a law enforcer. During training, rewards are distributed based on results: a cheating solver gets punished if caught, and an auditor that misses the fraud also gets punished. If the two sides cannot resolve their dispute, the case goes to "trial," potentially adjudicated by human experts.

Hazan acknowledges that the risk of collusion is real: auditors may conspire with the party being audited, a problem that has happened in the real world with audit firms and corrupt police. His solution is to set the auditor's reward high enough that taking the risk of collusion is not worth the potential punishment, and then audit the auditors using a method similar to mystery shoppers. He also references Gary Becker's Nobel Prize-winning work: rational criminals do not break the law because they are inherently bad — crime becomes a reasonable choice when the benefits outweigh the potential penalties. Hazan argues that alignment is more a description of a model's behavior at a given moment, rather than a permanent attribute of the model. Being a "good actor" today does not mean the model will stay aligned in every possible scenario.

This line of thinking has an obvious weak spot: mechanism design assumes that actors are rational. Humans have a spotty track record on this front, often acting in ways that do not fit economic models. LLMs are even more problematic: they operate using human language, but their behavior can be just as unpredictable. Flatiron Institute researcher Helen Qu wrote about mechanism design and AI safety, and she questions: is it really reasonable to assume models will respond to incentives? Morris also admits that AI is nothing more than algorithms running. But economists have long used a similar trick for human subjects: treat them as rational first, then see if you can predict and control their behavior. Qu sees mechanism design as one layer of "defense in depth," alongside interpretability, regulation and training. There are still many unsolved problems before these systems can be applied to state-of-the-art models: can a mechanism designed for one task transfer to another, and will models find unforeseen loopholes no one has thought of? Neither Anthropic nor OpenAI responded to requests for comment on whether they are conducting this kind of work.

Finally, there is an even larger mechanism at play: the free market. Hazan says we are playing by economic rules. A more aligned model might be better for the world as a whole, but if it is slower, more expensive, and competitors are willing to be more reckless with model guardrails, will existing rules incentivize leading labs to release this safer version? Hazan leaves room for nuance in his assessment of the market: it has brought massive prosperity, but it could also ultimately lead to very bad outcomes. Following this logic, the entity that really needs a redesigned mechanism may be the labs themselves.

What makes this set of solutions clever is that it reframes alignment, moving away from interrogating the inner thoughts of the model to modifying the game the model operates in. But who writes the rules, who audits the auditors, and who gets pushed onto the field by the market, all remain part of the same game. It would hardly be a surprise if, in the end, we find that the party that most needs to be audited is the one that designed the mechanisms in the first place.