Four AI Firms Are Adopting a Shared Opt-Out Signal — Can Websites Block AI Scraping in One Step?

OpenAI, Google, Microsoft and Anthropic have announced they will implement a new standard that lets publishers opt out of AI web scraping. This signal, called the shared policy token, was proposed by Kevin Kelley, Head of Standards Technology at Anthropic, at the IETF, and is added to a website's robots.txt file. The news was first broken by Search Engine Roundtable.

Illustration by Cath Virginia / The Verge | Photos from Getty Images

robots.txt is not a new tool. Search engines, archiving services and all types of web crawlers have long referenced it to decide which content they can access. The pain point that emerged as AI training scraping became more widespread is that websites previously had to set their opt-out preferences separately on each AI company's platform, which was high-cost and prone to omissions. The new solution lets publishers state their preference once in robots.txt, with multiple AI companies reading the exact same unified signal.

The IETF draft defines this signal as a formal training preference marker. robots.txt was originally designed for scenarios including search indexing and web archiving. The report notes that all four companies have pledged to adopt this standard. One critical point to note: robots.txt is a declarative statement, and compliance relies entirely on crawler operators' voluntary adherence. It communicates the website's opt-out intent, but does not technically block scraping attempts at the infrastructure level. Websites will no longer need to fill out dozens of separate forms, but whether the new standard can actually block training crawlers still depends on how each company implements the rule, what scenarios it covers, and how existing previously scraped data is processed.

For developers, two developments are worth following closely: whether the draft will move to formal finalization, and the specific details of how each AI company reads and processes the policy token. For regular users, this is not a flashy gimmick, but it directly impacts whether the content on websites they visit will be scraped to train AI models. The original full article from The Verge is available here.