Publishers may soon be able to set one AI training preference in robots.txt instead of maintaining separate rules for every AI crawler. Google, Microsoft, and OpenAI said Friday they plan to honor a shared policy token called __usage_ai-training.
The idea comes from an Internet Draft submitted by Anthropic’s Kevin Kelley to the IETF’s AI Preferences working group. It is still an individual draft, not an official standard, and none of the platforms supports it yet.
How the shared opt-out works
Kelley’s draft does not add a new directive to robots.txt. A site would write an ordinary rule group under the token. The draft’s example disallows training on the whole site while still allowing the /press-kit/ folder. The working group’s vocabulary defines AI training as changing a generative model’s learned parameters, while models that only classify or rank content are exempt. Existing robots.txt rules are not treated as a shared preference, a wildcard group cannot match the token, and a file without the token simply means no preference. The token also does not control crawling.
Google ties the token to Google-Extended
Google’s John Mueller said that when a site opts out through the shared line and has no specific Google-Extended rule, Google will interpret it
“as if the publisher had disallowed Google-Extended.”
Support is planned for early 2027. Google-Extended governs Gemini model training and grounding in Gemini Apps and Vertex AI, but not inclusion in Google Search. AI Overviews and AI Mode remain a separate Search Console opt-out.
Microsoft splits training from grounding
Microsoft is taking a different route. It will use the shared token for training but keep AI grounding as a separate page-level control, which can also be set by meta tag or HTTP header. DisallowExtended allows content to inform Copilot and Bing AI Mode, but blocks grounding through Microsoft-provided services used by other businesses, known as Web IQ.
Microsoft is replacing NoArchive and NoCache for training and grounding. It will begin honoring the new options on February 1, 2027, and stop recognizing the older tags after March 1, 2027. Sites still using those tags and wanting to stay out of grounding can record a temporary preference in Bing Webmaster Tools starting December 15. Pages without a preference by March 1 may default to allowing grounding until Microsoft recrawls them.
What to do now
One shared line could eventually replace a stack of crawler rules, but the implementations are not identical. Google bundles training and grounding under Google-Extended; Microsoft splits them. OpenAI says it aims to honor the signal within months, while Anthropic says full support across its services will take time.
- Audit your current robots.txt rules for Google-Extended, GPTBot, NoArchive and NoCache.
- Decide what needs to stay out of training, grounding, or both.
- If you use Microsoft’s legacy tags, set the Bing Webmaster Tools preference before the March 1, 2027 deadline.
- Don’t assume one opt-out will cover every AI product in the same way.
For publishers, this is less a single switch than a migration project with different deadlines per provider. The reward is a cleaner robots.txt and one signal that most major AI crawlers will eventually read.
Source: Search Engine Journal
