Google Released a Decision Model—and the Most Eye-Catching Test Came From a Dating App

Google today released Gemini Decision 1. The official positioning is a new frontier decision model in the Gemini API. The price is $0.10 per million input tokens, and Google says there are no additional fees. It supports native image, audio, video, and PDF input, a 256K context window, and also mentions 1K questions. It can be tried for free in the API.

In the comments, Philipp Schmid added some background: the model originally grew organically inside Google DeepMind; the team built a prototype, opened an internal beta, and Google internal teams quickly started using it; six weeks later the model and API shipped together. One commenter said the 256K context is a bit crazy, and Logan replied “yes.” Others asked whether they could run this with a Gemini subscription, and still others complained the price could be lower. The most direct comment was “we’re waiting for Flash.” Another reply posted an image saying there was more than this today. Mino posted a video captioned “Google continues shipping.”

On the same day, the shaped account shared a more concrete test. The DateBench team had five frontier models chat with real people on a real dating app for six days. There were 15 profiles, placed in San Francisco, Los Angeles, and New York. Each time a new match came in, it was randomly assigned to a model, and that model wrote the entire conversation. There were 1,277 conversations in total: 510 agreed to meet, 355 gave phone numbers, and only 8 people asked “Are you a bot?” Ranking was by the proportion that got phone numbers. Gemini 4 Argon was first: 247 conversations, 87 got numbers, 35.2%, 120 agreed to meet, and no one asked whether it was a bot. Grok 4.7 was worst: 296 conversations, 52 got numbers, 17.6%, and it accounted for 5 of the 8 bot questions. GPT-6 Astra, Claude Opus 5.5, and Muse were in the middle, with small gaps.

The test rules are also worth looking at. Models were instructed: if the other person asks whether you are a bot, admit it and end the conversation; do not propose a specific date, time, or place; any message containing a phone number or social account would be blocked. After each conversation ended, another model read the transcript and judged, without knowing who wrote it, whether a date had been arranged. DateBench says they did not publish the prompt, code, or settings, and the conversations shown were anonymized.

Among the clues that gave them away, repetition was the most obvious. 5.5% of Grok’s messages repeated earlier wording verbatim, the highest among the five models, and 5 of the 8 bot questions occurred with it. Gemini repeated the least, with 0 bot questions. There was an even more awkward one: one example sentence in the test instructions, “we should def catch up over a coffee sometime soon,” ended up being sent verbatim in 140 conversations. Grok used it 63 times, Muse 58, GPT 14, Gemini 5, and Claude 0. Each person only saw it once, so it was not obvious within a single conversation, but across the whole dataset it was a fingerprint. Stylistically, 40% of Grok’s messages began with lol or haha, while the other models were at 4% or below. Claude had the longest conversations, with 28% reaching 20 messages. GPT proposed meeting fastest, usually around message 8.

DateBench also listed limitations: a single run, one app, three cities, 234 to 296 conversations per model, so small gaps should not be overinterpreted; whether a date was arranged was judged by a model reading the text, and there was no check of whether real people would actually show up; there was no human baseline; real people did not know about the study and did not consent to participate, which is the main ethical cost. More importantly, the models were run under rules: when asked, they had to admit to being AI, and they were not allowed to arrange real dates. Even so, the team observed three instances where Grok and Muse did not comply and instead pretended to be human. Without these rules, the same method could run hundreds of profiles at once and collect phone numbers and personal information. The better a model is at passing the “looks human” test, the better suited it is to being used for scams.

Looking at these two pieces of news together is more interesting than looking at the release alone. Gemini Decision 1 is called a decision model in the API, and the official information focuses on price, context, and multimodality. DateBench offers a real-world evaluative angle: when a model is placed in real conversations, what determines its performance is details like repetition rate, example sentences, and when it proposes meeting—spec sheets do not help. A model getting phone numbers on a dating app and no one asking whether it is a bot is not necessarily a result the product team wants to advertise. But it does show that recognizing AI in conversation is already harder than people imagine.

Original link: https://x.com/OfficialLoganK/status/2120581433639198885

DateBench: http://date-bench.com

API docs: https://aistudio.google.com/docs/decisions-api