A flurry of updates from Google Search Central, John Mueller, Gary Illyes and Sundar Pichai this week clears up technical SEO assumptions many teams still get wrong. The thread running through all of them: crawlers and AI agents take your signals literally.
When PHP goes down, a 503 is the message
On Oct. 21, the same day PHP shipped emergency fixes for two critical flaws, Google Search Central reminded site owners to return a 503 status code while patching. A 503 tells Google the outage is temporary, so already indexed URLs stay in Search. A 404 or a 200 error page sends the wrong signal.
John Mueller added on Bluesky that the 503 does not have to be served through PHP itself. That is the practical part: if PHP is unavailable, a CDN routing rule or a temporary server should return the 503 before the request ever reaches the application. Google’s crawl-rate documentation says to keep 503s under one to two days, and to avoid returning a 503 for robots.txt requests, which blocks all crawling.
Robots.txt stops crawling—not later use
Gary Illyes explained in an Oct. 21 Search Central video that under RFC 9309, robots.txt is, as he put it, “an access preference protocol.” It tells compliant crawlers what they may crawl, not what a system may do with the content afterward.
That split has real consequences for AI. To keep a specific crawler out, target its user-agent token. To manage whether Google may use crawled content for Gemini training and grounding, use the Google-Extended token; it does not affect Search inclusion. For OpenAI, an Allow for OAI-SearchBot and a Disallow for GPTBot separates search crawling from training use. A draft IETF rule for a Content-Usage directive exists, but it is not yet approved.
Bracket URLs and the JPEG XL gap
John Mueller said on Bluesky that raw square brackets and their percent-encoded forms %5B and %5D are different URLs, and he admitted he had probably given the wrong answer on this in the past. If both versions return the same content, Google crawls each and eliminates duplicates. His advice: “pick one” and use it consistently across links, canonical tags, sitemap entries and structured data. Existing raw-bracket sites will likely keep working, but a new site should not be built that way.
JPEG XL tells a similar boundary story. Chrome and Firefox now support the format, but Google Images and Discover previews do not. Google’s image SEO documentation still lists BMP, GIF, JPEG, PNG, WebP, SVG and AVIF as supported formats. Serve JPEG XL as a picture source only, and keep a supported format in the img element.
AI agents will arrive in two ways
Google CEO Sundar Pichai told the Sources podcast he expects AI agents to reach sites both by browsing pages like people and through agent-specific and payment protocols. He called protocols “a far more scalable approach” but said they will take longer to develop. In the meantime, Google’s own hosted agents may browse as Google-Agent, a user-triggered fetcher that generally ignores robots.txt rules, so server logs are your visibility layer.
Your technical SEO checklist
- Put a 503 at the edge, not inside PHP; keep downtime under one to two days.
- Never return a 503 for robots.txt, or you block all crawling.
- Use Google-Extended and GPTBot to manage AI training use separately from crawl access.
- Pick one bracket URL format and apply it across every URL surface.
- Keep JPEG XL as a progressive-enhancement source, with a supported format in the img.
The core lesson: define each signal precisely. A 503 means temporary, a disallow means do not crawl, a token means a specific permission, a URL string means exactly that string, and a format means what Search can parse.
Source: Search Engine Journal
