SEO
Faking Googlebot takes 30 seconds, warns Google’s Gary Illyes
Gary Illyes says spoofing Googlebot takes 30 seconds. Verify crawlers with reverse DNS, published IP ranges, and Google's Web Bot Auth.
Googlebot.
SEO
Google: ‘Couldn’t Fetch’ Sitemap Is Often a Quality Signal
Google's Splitt and Mueller explain why Search Console says couldn't fetch for valid sitemaps: host load, crawl demand, and site quality.
SEO
Robots.txt in 2026: Crawling Rules, AI Opt-Outs and Gaps
Robots.txt is voluntary, narrow and now central to AI training opt-outs. Here's what SEO and performance marketers need to know about RFC 9309, GPTBot blocks and crawler gaps.
SEO
Cloudflare’s AI Training Block Now Spares Googlebot
Cloudflare's new Disallow AI Training setting lets sites block AI training without losing Google, Bing, or Apple search visibility. Here's what to check before toggling.
SEO
Cloudflare’s Bot Policy Sync Automates AI Crawler Rules
Cloudflare's Bot Preference Sync writes robots.txt entries from your AI bot settings. Here's how it works, the SEO tradeoffs, and the quick audit to run.
SEO
Cloudflare’s New AI Training Setting Keeps Googlebot Crawling
Cloudflare’s Disallow AI Training setting blocks AI crawlers while preserving Googlebot, Applebot, and Bingbot search access. Here’s what to check.
