Google-Extended sounds like a crawler. It isn’t. It’s a robots.txt product token: a control that tells Google whether pages fetched by Googlebot may be used to train Gemini models and ground answers in Gemini Apps and Vertex AI. No request ever arrives from Google-Extended, so you won’t see it in server logs.
That design matters for SEO teams. Google crawls the web once and feeds the same copy into Search and AI systems. The token gives publishers a way to refuse the AI use without disappearing from organic search.
What the token covers
As of September 2026, the documentation defines two uses:
- Training future Gemini models that power Gemini Apps and the Vertex AI API.
- Grounding: supplying content from the Google Search index at prompt time to improve factuality and citations in Gemini Apps and Vertex AI.
In practice, you opt out with a robots.txt group that names Google-Extended and adds disallow: /. Rules can be scoped to paths, so a publisher could withhold current reporting while allowing an archive. Google says the token is not a ranking signal. Changes generally take effect on the next robots.txt refresh, which can be cached for up to 24 hours.
The search-sized hole
Google-Extended does not cover the generative features inside Search. AI Overviews and AI Mode are governed by Googlebot rules and snippet controls such as nosnippet, data-nosnippet, max-snippet or noindex. The opt-out is also not retroactive: Google told the UK Competition and Markets Authority that removing content already added to its search AI models was not feasible.
That split is the controversy. Publishers can block Gemini training while their content continues to feed free AI answers that reach far larger audiences. Eli Collins of Google DeepMind acknowledged in testimony that the search organisation could train on opted-out content for use in search.
Think of Google-Extended as a training control, not an AI visibility switch. If your goal is to disappear from AI Overviews, this is the wrong tool; the Search Console generative AI control is the Search-side counterpart.
Adoption is lower than you’d expect
Publisher uptake trails rival bots by a wide margin. Reuters Institute data showed 24% of top news websites across ten countries blocked Google’s AI crawler by the end of 2023, against 48% for OpenAI. Originality.ai counted 13.6% of the top 1,000 websites blocking the token in 2024, compared with 35.7% for GPTBot. BuzzStream found 46% of 100 leading UK and US news sites blocking it in December 2025.
There is a commercial reason for the hesitation. A later BuzzStream study found that 92.3% of top news sites blocking Google-Extended still appeared in AI citations across ChatGPT, Gemini, AI Overviews and AI Mode. A Rutgers and Wharton analysis found publishers blocking LLM crawlers lost about 7% of weekly visits within six weeks.
What to do with it
Before editing robots.txt, separate three decisions: search visibility, Gemini training, and AI search citations. Google-Extended only controls the second bucket. If you do block it, test a scoped path first, monitor organic visibility, and pair the move with snippet controls if AI Overviews exposure matters to you.
Source: PPC Land
