For three years, publishers who wanted to keep their work out of AI models had to track a new crawler every few months: Google-Extended, GPTBot, ClaudeBot, Applebot-Extended, CCBot. Each company shipped its own toggle, and none used the same rules.
On Friday, October 30, Google and Microsoft said they would honor a single control. Sites can disallow a shared user-agent token in robots.txt, __usage_ai-training, to signal participating AI providers not to use content for training. “This will help websites express their AI training preferences across multiple AI providers in a single step,” Google said on its Search Central blog.
One token, two schedules
Google is adopting the proposal voluntarily through the IETF’s AI Preferences working group and plans support in early 2027. If a site uses the new token without naming Google-Extended, Google treats Google-Extended as disallowed too. Existing crawler-specific controls keep working.
Bing goes further by separating AI training from AI grounding, the moment-to-moment retrieval that lets assistants cite a page when answering.
- December 15, 2026: temporary grounding controls appear in Bing Webmaster Tools.
- February 1, 2027: Microsoft begins honoring the new preferences.
- March 1, 2027: deadline for sites relying on old NoArchive or NoCache directives.
- Early 2027: Google aims to support the shared training token.
What opting out may cost
The same morning, Cyrus Shepard published a single-site study: he removed a site from AI Overviews and AI Mode via Google’s opt-out controls and compared before and after. He estimated click-through from AI surfaces at about 0.9%. AI impressions made up roughly 8% of the site’s total.
The more striking result: when AI visibility was switched off, ordinary organic clicks also dropped, then recovered when visibility returned. Shepard called it a “Search SERP Halo Effect.” The sample is one site, so treat it as a signal, not a benchmark.
Separate training from grounding
Bing now lets publishers appear in Copilot answers while blocking model training. That makes the decision two-fold: whether your content trains models, and whether it can be cited in AI answers. If you block citations to protect content, you may lose the halo that produces later branded searches and clicks.
Measurement is getting harder too. Gemini stopped attaching UTM parameters around September 28, and its iOS app can send links through an embedded browser with no referrer, so visits land as Direct in analytics. Server logs may be your only trace.
Gary Illyes of Google warned that AI has made bot impersonation worse. Verify Googlebot by IP address and reverse DNS, not user-agent. Robots.txt is a suggestion, not a security control.
Source: PPC Land
