Google’s Search Relations team has put publishers on notice: robots.txt is a gate, not a licence. In the second episode of The Non-Canonical Cabinet, Gary Illyes and Cherry Prommawin walked through what the plain-text file actually controls—and the answer leaves a big gap for anyone trying to keep AI training crawlers away from their content.
Four valid instructions, everything else disappears
Published on October 21, 2026, the episode describes robots.txt as “the gatekeeper of your website.” Google’s parser recognises four kinds of lines: user-agent, allow, disallow and sitemap. The first three come from RFC 9309; sitemap is a widely supported extension.
The hosts were blunt about what happens to anything fancier. A line written in prose, a custom directive or a malformed token is silently discarded. There is no error message. If you write “don’t crawl secret page” instead of a valid disallow rule, the crawler can ignore the line and crawl the page anyway.
The AI distinction publishers needed to hear
The clarifying moment arrived when the hosts tackled generative AI. As Google presents it, robots.txt is an “access preference protocol”—it tells bots which rooms they may enter, not what they may do with the content after a crawl.
“It doesn’t dictate what a system does with the data after it has already crawled it.”
In practice, that means a disallow rule is a request to stay out. It is not a training opt-out and it is not a data-use term. To keep a specific AI crawler away, site owners must name that bot’s user-agent token explicitly. For Google’s Gemini training and grounding, the operator-specific Google-Extended token is the relevant control—not a generic “no AI” line.
What marketers should fix now
The practical takeaways are straightforward:
- Audit every line. Strip out prose, emoji in user-agent tokens and invented directives. Valid syntax is the only thing that counts.
- Percent-encode non-ASCII paths. Google’s parser handles raw UTF-8, but older or less forgiving crawlers may ignore a disallow line containing Chinese characters, umlauts or emoji.
- Separate access from use. Block crawling with user-agent rules, and negotiate data use through operator tokens, licences or contracts.
Why silence is the real risk
Because invalid lines fail with no warning, teams can believe a staging environment or internal search path is protected when it is not. Google’s documentation sets a 500 KiB limit on the file, so large comment blocks can also push real rules beyond what crawlers read. And blocking a page in robots.txt prevents crawling, not indexing: linked pages can still appear in results because Google may never fetch the page to see a noindex tag.
The wider signal is that the industry is already moving beyond access controls. Cloudflare has added a separate no-training preference, and testing shows a share of AI agents still fetch disallowed URLs. For publishers, the job is to make robots.txt boring, valid and explicit—then treat data use as a separate negotiation.
Source: PPC Land
