SEO
Google Clarifies 503s, Robots.txt Limits and Bracket URLs
Google clarifies 503s during PHP patches, robots.txt crawl limits, raw vs encoded bracket URLs, and why JPEG XL still isn't visible in Search.
AI crawlers.
SEO
Google: robots.txt Controls Crawling, Not AI Training
Google Search Central says robots.txt only governs crawl access, not AI training on collected content. What to audit now.
SEO
Google’s September Spam Update Ends After Nearly 14 Days
Google's September 2026 spam update is done after nearly 14 days. Here's how to read the data, plus new crawl timing ranges and author warnings.
Digital Marketing
Cloudflare Strips AI Opt-Outs From 118 Sites
HasData re-ran 10,894 domains after Cloudflare's September 15 update and found 118 sites lost their managed AI-training opt-outs in robots.txt.
SEO
John Mueller: Use Default Sitemap or RSS for AI Crawlers
Google's John Mueller says AI training crawlers rarely accept sitemap submissions. Use a default sitemap.xml or RSS feed to make your content discoverable.
SEO
Google says couldn’t fetch can be a quality verdict
Google Search Off the Record explains why a valid sitemap can show couldn't fetch: host load or low crawl demand tied to site quality.
AI in Marketing
Can You Change AI’s Opinion? A GEO Test Says Yes
AnswerShare's experiment shows that adding full brand context can flip AI models from warning buyers away to recommending a business.
SEO
llms.txt: What Marketers Need to Know Before Testing It
llms.txt promises a curated path for AI crawlers, but no major AI platform confirms using it. Here's what Semrush found and how to test it.
SEO
Robots.txt in 2026: Crawling Rules, AI Opt-Outs and Gaps
Robots.txt is voluntary, narrow and now central to AI training opt-outs. Here's what SEO and performance marketers need to know about RFC 9309, GPTBot blocks and crawler gaps.
SEO
Cloudflare’s AI Training Block Now Spares Googlebot
Cloudflare's new Disallow AI Training setting lets sites block AI training without losing Google, Bing, or Apple search visibility. Here's what to check before toggling.
