Robots.txt looks like infrastructure, but in 2026 it is a negotiation. The plain text file at the root of a website tells crawlers what they may fetch, and it has moved from a technical courtesy to the default opt-out mechanism for AI training, ad quality checks and even courtroom arguments.
Its formal name is the Robots Exclusion Protocol. A server publishes rules at /robots.txt, and crawlers such as Googlebot or GPTBot are expected to read them before fetching. Yet the IETF standard that governs the file, RFC 9309, is explicit: this is a request, not an authorisation system.
How the rules actually work
The file must sit in the top-level directory of each host and subdomain, which means example.com and shop.example.com need separate files. Inside, rules are grouped by user-agent token, followed by allow and disallow paths. The longest matching path wins, and a wildcard group catches crawlers that do not have their own named group.
Failure handling matters as much as syntax. A 404 response means the file is unavailable, so anything may be fetched. A 500-range error means the crawler should assume everything is disallowed until it can recover. Google enforces a parse limit of at least 500 kibibytes.
Crawling is not indexing
This is the part marketers keep getting wrong. Robots.txt stops a fetch; it does not directly take a page out of search results. Google’s John Mueller put it clearly in September 2024:
“The robots.txt isn’t a suggestion, it’s pretty much as absolute as possible.”
But a URL blocked in robots.txt can still be indexed on the strength of external links, because Googlebot never reaches the page to see a noindex tag.
Ad bots and AI bots read different rules
Google’s advertising crawlers are special cases. AdsBot-Google, which checks landing pages for Google Ads, and Mediapartners-Google, which reads pages for AdSense and Ad Manager, ignore the wildcard group entirely. A blanket disallow for all agents still lets them through; naming them explicitly can switch off ad quality checks that decide whether a destination is approved.
AI has raised the stakes. OpenAI launched GPTBot in August 2023 and Google added the Google-Extended token a few weeks later, giving publishers named targets for training opt-outs. By August 2024, 35.7% of the world’s 1,000 largest websites blocked GPTBot, compared with 5% at launch. News publishers that blocked large language model crawlers lost about 7% of weekly traffic within six weeks, according to Rutgers Business School and Wharton research.
What to do about it
- Audit /robots.txt on every subdomain, protocol and port your brand actually serves.
- Name advertising and AI crawlers explicitly instead of relying on the wildcard group.
- Use meta robots noindex or X-Robots-Tag when the actual goal is removal from search.
- Monitor what bots really fetch, because compliance is partial and undeclared crawlers exist.
Cloudflare’s managed tools already generate robots.txt directives for millions of domains, and the IETF has draft proposals for AI-specific content usage rules. But the underlying file remains the same: a voluntary signal with no enforcement force.
Bottom line
Robots.txt is now referenced by the EU General-Purpose AI Code of Practice and is being tested in US litigation over AI training. But its vocabulary remains coarse: a disallow rule stops a fetch, not “index this but do not train on it.” Treat it as a policy signal with real commercial consequences, and back it up with page-level controls.
Source: PPC Land
