Your robots.txt file is supposed to tell crawlers what you want. But many sites have a quiet mismatch: the text file says one thing, while edge rules or bot controls enforce something else. That gap can hand AI crawlers an argument for ignoring your preferences.
Cloudflare’s Bot Preference Sync is designed to close it. The feature takes your bot policy settings from the dashboard and writes matching robots.txt entries into your existing file. It preserves your current content but adds a managed block between Cloudflare markers.
The setup runs from the free tier. For new domains, Cloudflare says the sync will be on by default starting September 15, with Training and Agent blocked on ad-monetized pages and Search left allowed.
How the sync works
Bot Preference Sync is built around three categories: Search, Agent, and Training. Each can be set to block on all pages, block only on pages with ads, or allow. Cloudflare’s tracked bot list determines which crawlers fall into each bucket, and your three choices become robots.txt rules.
One important limit: you cannot exclude an individual crawler from the sync. If your policy is per-company rather than per-category, Cloudflare’s documented solution is to turn the sync off and maintain robots.txt yourself.
The tradeoff: category rules vs. business decisions
Many marketers don’t have a universal “AI training” policy. One common setup is allowing OpenAI’s GPTBot, Anthropic, and PerplexityBot while blocking Bytespider and meta-externalagent. Every one of those companies trains models, but the return to the site is different.
Bot Preference Sync cannot express that. Set Training to disallow and you tell OpenAI not to train on content you might be happy for it to use. Set Training to allow and you lose the distinction between Meta and ByteDance. That is a real gap for growth teams that treat AI crawler visibility as a partnership question.
Cloudflare has also attached four disclosure conditions for mixed-use crawlers that handle both search and training. A bot must:
- Respect a “no training” preference in robots.txt via any mechanism.
- Give site owners a way to opt out of AI summaries.
- Provide URL-level visibility into pages used for training plus search metrics.
- Publicly show that disallowing training does not hurt traditional search results.
Crawlers that miss these conditions are treated as opaque and blocked when Training is set to disallow. Microsoft’s NOARCHIVE handling is cited as meeting the AI summary opt-out part; Google’s documentation covers the ranking point but not a standalone AI Overviews opt-out.
A 10-minute audit before you let a vendor write policy
Robots.txt is documentation of intent. It matters in disputes and for well-behaved crawlers, but it does not stop bad actors. Before the sync reaches your site, check whether it would say what you actually want.
First, read your current robots.txt, including old entries. Then open Cloudflare’s Security Settings and compare your AI bot policies. If they disagree, you already have the mismatch. Finally, ask whether three categories can express your real crawler policy. If yes, the sync saves manual work. If not, switch it off and keep editing by hand.
The biggest risk is accepting a default you never read. A policy file you have not checked is a statement someone else is making on your behalf.
Source: Search Engine Journal
