Four AI Labs Back a Shared Training Opt-Out Signal

What happened

On October 30, OpenAI said in its official blog that it will honor a common signal in robots.txt, and Google said the same day in its Search Central blog that it plans to support the control in 2027 alongside other providers. The signal comes from an Internet-Draft of the Internet Engineering Task Force (IETF) AI Preferences working group (AIPREF), titled "Machine-Readable Training Preference Signal" and authored by Anthropic's Kevin Kelley.

A site writes two lines in robots.txt — User-Agent: __usage_ai-training and Disallow: / — to state its no-training preference, and can use Allow lines to carve out exceptions for specific directories. Until now, sites had to write separate rules for each provider's crawler, such as GPTBot, Google-Extended and ClaudeBot, with the list growing as new crawlers appeared.

The signal covers training only: it does not restrict crawling, search indexing, or whether content appears in AI-generated answers. It is voluntary: the draft itself is not yet a formal standard, and the IETF page notes that an individual Internet-Draft is not endorsed by the IETF and binds only the providers that choose to follow it. Content already collected into existing datasets or used for training is not removed by it.

Timelines differ by provider: Anthropic has reportedly begun implementing support; OpenAI says it aims to implement the signal in the coming months; Microsoft will start honoring it on February 1, 2027, while adding a page-level training setting and an AI Grounding setting; Google says it aims to support it in 2027. Microsoft also said sites relying on Bing's old NoArchive and NoCache tags must switch by March 1, 2027.

Google said that when a site uses the shared token to opt out of training and does not explicitly mention Google-Extended, Google will treat that as if the publisher had disallowed Google-Extended; sites that specifically allow Google-Extended remain eligible to appear and receive clicks through grounding in Gemini apps. Each provider's existing controls continue to function, so sites can still write rules for individual providers.

Key facts

1. Signal name: __usage_ai-training, written in the User-Agent line of robots.txt together with Disallow: /
2. Draft: "Machine-Readable Training Preference Signal," authored by Anthropic's Kevin Kelley, published October 30, 2026, expiring May 3, 2027 unless updated
3. Announced by: OpenAI, Google and Microsoft on October 30; Anthropic authored the draft and has reportedly begun implementing support
4. Timelines: OpenAI: in the coming months; Microsoft: February 1, 2027; Google: aiming for 2027
5. Scope: Training preference only; does not restrict crawling, search indexing or citation in AI answers
6. Google's reading: Without an explicit Google-Extended mention, the shared opt-out is treated as disallowing Google-Extended; existing provider controls keep working
7. Microsoft extras: Page-level training setting, AI Grounding setting and a DisallowExtended option; old NoArchive and NoCache tags must be switched by March 1, 2027