Robots.txt is one of the oldest tools in technical SEO, yet plenty of teams still assume it can do a job it was never designed for. In a Search Central video published Wednesday, Google’s Gary Illyes made the boundary explicit: the file controls which URLs a crawler may request—not what happens to the content after it has been fetched.
The clarification came in the second episode of The Non-Canonical Cabinet, where Illyes and Cherry Prommawin of Google Search Relations answer edge-case community questions. The question on the table: can robots.txt allow AI crawling of a website while keeping that same information out of generative model training?
The access-versus-usage split
Illyes pointed to RFC 9309, the internet standard for robots.txt. “As defined by RFC 9309, robots.txt is an access preference protocol,” he said. It “doesn’t dictate what a system does with the data after it has already crawled it.”
That means marketers cannot write one clever line in the file and expect every AI system to treat it as a training opt-out. If you want to block a specific AI crawler, you have to target that crawler’s user-agent token with a Disallow rule. A fictional on-screen directive such as “No-AI-Chicken-Training” would simply be ignored by a parser.
Google’s own robots.txt specification supports four elements: User-Agent, Allow, Disallow and Sitemap. Anything outside those lines is not a reliable instruction.
Google-Extended is a usage control, not a crawl control
Where Google does offer a usage control, it comes through Google-Extended. This token has no user agent of its own; it works with Google’s existing crawlers in a “control capacity.”
Google-Extended doesn’t change whether a page appears in Google Search and isn’t a ranking signal. What it does is govern whether content Google already crawls can be used to train future Gemini models or for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
That is the crucial distinction for performance and SEO teams: visibility in Search and suitability for AI training are two separate levers.
A standard for usage preferences is still a draft
Illyes is also co-author, with Mozilla’s Martin Thomson, of an IETF Internet-Draft that would update RFC 9309 with a Content-Usage rule for robots.txt and a matching HTTP header. In the draft, Allow and Disallow would still govern crawling, while Content-Usage would state how crawled content can be used. A sample line reads “train-ai=n” alongside permission to crawl.
But this is not a standard yet. The draft is a Working Group Document and has not been submitted to the IETF’s Steering Group for review. A separate vocabulary draft carrying the “train-ai” label states that it reflects no working group consensus. At present, there is no standard way to express a training preference independently of which bots may crawl a page.
What to do now
- Block a specific AI crawler: target its user-agent token with a Disallow rule.
- Manage Gemini: add Google-Extended if you want to keep Google Search visibility while limiting AI training and grounding.
- Percent-encode non-English paths: Google reads raw UTF-8 and encoded paths the same, but older crawlers may not.
- Don’t invent directives: unsupported lines will be ignored, not enforced.
Cherry Prommawin’s advice for paths with non-English characters or emoji was blunt: “you should use the encoded version in your robots.txt rules.” Google’s parser may handle raw characters, but other or older crawlers “might get confused.”
Why this changes your checklist
For digital marketers, the takeaway is not that robots.txt is powerless. It is that access controls and usage controls are different jobs. If your goal is to keep a particular AI crawler away, robots.txt can do that. If your goal is to stay in Google Search while limiting Gemini training, Google-Extended is the documented tool. If you want a clean, standardised “train-ai” instruction across the industry, that future is still in draft.
The next episode will cover the 404 status code, but the bigger signal for technical SEO is the IETF work. The Content-Usage draft is set to expire on Feb. 20, 2027 unless a new revision replaces it sooner.
Source: Search Engine Journal
