Digital Marketing
Cloudflare Strips AI Opt-Outs From 118 Sites
HasData re-ran 10,894 domains after Cloudflare's September 15 update and found 118 sites lost their managed AI-training opt-outs in robots.txt.
robots.txt.
SEO
Google-Extended: What It Blocks and What It Leaves Open
Google's robots.txt token can block Gemini training without hurting rankings. But AI Overviews and AI Mode sit outside it. Here's what to do.
SEO
Robots.txt in 2026: Crawling Rules, AI Opt-Outs and Gaps
Robots.txt is voluntary, narrow and now central to AI training opt-outs. Here's what SEO and performance marketers need to know about RFC 9309, GPTBot blocks and crawler gaps.
SEO
Cloudflare’s AI Training Block Now Spares Googlebot
Cloudflare's new Disallow AI Training setting lets sites block AI training without losing Google, Bing, or Apple search visibility. Here's what to check before toggling.
SEO
Cloudflare’s Bot Policy Sync Automates AI Crawler Rules
Cloudflare's Bot Preference Sync writes robots.txt entries from your AI bot settings. Here's how it works, the SEO tradeoffs, and the quick audit to run.
SEO
Cloudflare’s New AI Training Setting Keeps Googlebot Crawling
Cloudflare’s Disallow AI Training setting blocks AI crawlers while preserving Googlebot, Applebot, and Bingbot search access. Here’s what to check.
SEO
llms.txt Is Not robots.txt: Common Crawl’s Reality Check
Common Crawl studied 500,000+ llms.txt files: two-thirds are plugin-generated, 22% have no links, and some try to block crawlers they cannot block.
