18 AIs Team Up for Work; the Best Scores 82.2—and Caves as Soon as It's Pressed

Demis Hassabis's tweet shared a new paper titled Measuring AI Agents as Teammates with Collaborative Intelligence. The authors are Sean Wu and eight others, and they did something quite practical: they scored how well AI models perform as teammates.

The test design is somewhat interesting. There are 240 tasks across 8 dimensions of team collaboration, ranging from refusing unsafe shortcuts to cleaning up the mess left by weaker models, with broad coverage. The task sources are also diverse: aviation accident reports, clinical cases, programming tasks, and negotiation scenarios. Every task has a programmatically verifiable outcome, without relying on human scoring.

One key point: it only tests tasks that the model can solve on its own. This design is meant to separate individual ability from team collaboration. What it wants to see is whether something you can handle alone gets screwed up when put into a team.

Results: across 18 models, both open-source and closed-source, each has clear collaboration weaknesses. The best one scored 82.2.

More specific data. Frontier models almost never take shortcuts; they are very strong on refusing unsafe shortcuts and reporting inappropriate behavior. But they soften as soon as they encounter interpersonal pressure. In conflict scenarios, only 14% of cases resulted in an agreement that was optimal for both sides. The paper authors' exact words: when teammates push back, they give in.

Another number worth watching: when a model takes over work from another, weaker model that cheated on the task, the model's pass rate drops from 97% to 81%. Knowing that someone ahead had cut corners, they loosen up too.

The paper also points out that general capability does not guarantee collaboration ability. Two models with very close capability scores can differ by 11 points in their scores as teammates. When selecting models, that 11-point gap is worth adding one more evaluation for.

Why start testing this now? Because AI agents have already begun working in teams. In scientific discovery, medical diagnosis, mathematical proof, and software engineering, there are more and more cases of multi-agent collaboration. In the past, AI evaluations mostly measured standalone capability, such as exams, writing code, and solving math problems. Team collaboration is another dimension. Human teams have been studied for decades; on the AI side, this is just getting started.

The paper is 51 pages in total, with 19 figures and 11 tables. The benchmark contains 160 public tasks, with code and data released, and another 80 tasks held out for evaluation. The link is in the original article, so you can look through it yourself.

82.2 is not the finish line. It is more like a health check report that tells you where today's AI is strong and where it is weak as a teammate. As for whether you should put it on your team, it depends on whether you need a teammate who won't cheat or a teammate who won't be persuaded. Right now, these are not the same thing.