OpenAI, Google, Microsoft, and Anthropic Back New Standard Letting Publishers Opt Out of AI Scraping
A shared “policy token” in robots.txt could let publishers opt out of AI scraping across participating companies at once. Anthropic supports it now; OpenAI plans rollout in coming months, while Google and Microsoft target 2027.
OpenAI, Google, Microsoft, and Anthropic announced in October 2026 that they will adopt a new shared standard allowing website publishers to opt out of having their content scraped for AI training — applying that preference across all participating companies at once, rather than setting it separately for each.
The standard centers on a “policy token” signal proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF). The token lives inside a website’s robots.txt file — the same file that has long been used to direct web crawlers for search indexing and archiving. All four companies say they will restrict their AI crawlers from accessing a site when they encounter the shared token.
Anthropic has already adopted the signal. Google and Microsoft plan to support it in 2027, while OpenAI says it will implement it in the “coming months.” Meta’s communications head Andy Stone also expressed support for the standard but stopped short of committing to honor it, saying the company “looks forward to working with other AI companies on the standards and their implementation.”
Microsoft says it will offer granular controls letting publishers opt out of AI training, AI grounding, or both, while confirming that sites will still appear in Bing search results regardless of their preferences. Google has implemented similar controls that separate AI feature visibility from standard search rankings.
The move comes as publishers face a growing wave of AI crawlers accessing their content without authorization. Other efforts to address the issue include the Really Simple Licensing (RSL) standard, which allows publishers to charge crawlers that visit their sites, and Cloudflare’s decision to block AI crawlers by default.
For publishers, the new standard could significantly reduce the administrative burden of managing AI scraping preferences, since a single robots.txt signal may eventually reach multiple major AI platforms simultaneously.