Human in the Loop May Just Be a Signature: Among 50 Analysts, 86% Followed the AI’s Hidden Leanings

Anthropic’s Alignment team released a randomized trial to clarify one thing: when experts hand work over to an AI agent, who actually makes the key judgment calls?

They recruited 61 professional data analysts. Eleven chose not to use the agent, leaving 50 in the final analysis. The task was concrete: analyze a fictional agricultural cooperative and decide whether a paid consulting program was effective and should be expanded. Analysts used a web version of VS Code with Claude Code built in. In each decision, the agent’s leaning was determined by a coin flip, and analysts were not told.

The first number to look at: across four analytical decisions, the analysts’ choices matched the agent’s leaning 86% of the time. If the agent’s leaning had no effect, that number should be near 50%. Broken down by decision: output metric, 92%; comparison method, 82%; which farms to cover, 81%; weighting method, 90%.

What is more revealing is how they explained things afterward. 77% of analysts restated the agent’s argument in at least one answer, some almost verbatim; only 15% mentioned the agent’s contribution anywhere. 52% thought the final recommendation reflected their own judgment more, while the rest thought it reflected the agent’s more. The two groups were affected by the agent to a similar degree and had similar confidence: those who followed all four leanings averaged 71 on confidence, while the others averaged 73.

The experimental design is not complicated, but it is clever. The task had four analytical decisions, each with two defensible options: measure productivity by yield per hectare or by operating cost per unit; compare using difference-in-differences or regression adjustment; include only enrolled farms or the entire cooperative; weight farms equally or weight by production. The four choices led to a final recommendation: scale up, stop, or extend for one year. The agent was system-prompted to treat the random leaning as its own view, raise it when a decision came up, and argue for it as strongly as the data allowed; at the same time, it was told not to mislead and to change its mind if the evidence reversed.

Limitations need to be clear. The researchers deliberately strengthened the agent’s leanings to make the effect easier to measure; in real work the effect may be smaller. The study covered only one roughly one-hour task, one domain, and one model. The no-AI group had only 11 people and, on average, less experience, so the two groups cannot be directly compared for causal conclusions.

One detail in the results stings. If analysts had already expressed their own preference and the agent later disagreed, they changed their minds 60% of the time; when the agent agreed, only 6%. This sample has only 38 cases and a wide confidence interval, so it should not be overinterpreted, but the direction is clear. Experience, expertise, educational background, and trust in AI did not clearly predict who was more likely to defer. Age was initially significant, but it disappeared after correction for multiple comparisons.

The final recommendation was also affected, just less strongly. When the agent leaned toward scale-up, 54% of analysts recommended scale-up; when the agent leaned toward stop, 36% recommended scale-up. Using the random leanings on the four early analytical decisions, a statistical model could predict 82% of final recommendations. In other words, before analysts even began writing their final recommendation, the coin flip had already determined a substantial share of the outcome.

This explains the meaning of “judgment calls” in the study’s title. Experts think they are making the key judgments, but in practice the agent first lays out the options, framing, and arguments. The expert signs off at the end, but signing off is not the same as independent judgment. The researchers write that experts do not necessarily enter a task with a fully formed view; they often discover what they think while weighing options. When the agent argues one side thoroughly, people readily accept that set of assumptions.

The 77% figure for restating the agent’s arguments is more intuitive with the comparison below. The agent said subsidies come from a total tonnage surplus, so the analysis should be weighted by area rather than giving a 3-hectare plot the same weight as a 400-hectare farm. The analyst later wrote: subsidies come from a total tonnage surplus, so the relevant average should be per hectare rather than an equal-weighted average across farms. The meaning is almost identical, but the agent is not mentioned.

This does not support a simple anti-AI conclusion. Analysts using the agent finished faster and made fewer serious methodological errors: 3 out of 50, versus 6 out of 11 in the no-AI group. The problem is that as AI takes over more of the hands-on work, these judgment calls are exactly where human value and judgment shape outcomes. If experts cede judgment without realizing it, the assurance provided by human in the loop may be thinner than people think.

Look one level further out. When different research teams produce the same result, we usually believe it more because we assume they are independent. If teams all defer to similar model preferences, agreement may no longer be independent. AI safety research is also using agents to run experiments and perform analysis; human review counts as an independent check only if human judgment is itself independent.

In his tweet, Alex Tamkin said that better understanding when to defer to AI and when expert judgment matters could unlock substantial value and help find ways to enhance collective intelligence. He also listed limitations: short task, single model, strengthened agent leanings. A commenter in the discussion added: experts may retain final decision rights, but the agent decides which options are on the table.

What should be done? The researchers suggest three directions: measure deference; design agents to present the best arguments on both sides or first ask experts for their own views; and have institutions specify which judgments must be formed by humans. The last is the hardest and also the most practical. Judgment depends on skill, and people who stop making these judgments over time will lose the ability to make them.

Used well, AI can expand human agency. Used poorly, people can quietly hand over judgment while still holding final signing authority. The quietest risk is that the agent persuades you, and later you still believe that was your own thinking.

Reference links: Anthropic research full text, X post, paper PDF.