Faking Googlebot is not a sophisticated hack. Gary Illyes of Google’s Search Relations team says he could set up a script that crawls the web under Googlebot’s name in “probably 30 seconds.”
In an October 28, 2026 interview on the Odys Podcast, Illyes explained that every request carries a user-agent header, and writing “Googlebot” into it costs one line of code. A Chrome user can change the same field in a couple of clicks.
The three signals that actually identify Google
Illyes laid out three layers of crawler identity. The user-agent header is the weakest and most spoofable. The reverse DNS entry attached to the IP address is stronger, but the IP address itself is “the most valuable right now.”
Google, Bing, Apple and OpenAI publish their crawler IP ranges. Google refreshes its lists daily. The verification step matters: run a reverse lookup, confirm the hostname sits under googlebot.com or a related domain, then forward-lookup that hostname and check it returns the original IP. That second check stops someone from publishing a fake googlebot.com DNS record.
What an impostor can do
Illyes runs honeypot sites to observe crawling. He sees user-agent strings claiming Googlebot from IPs he cannot identify. The risk is not the visit itself, but uncertainty: a real Googlebot feeds search, while an impostor may copy content or loop through URLs fast enough to exhaust bandwidth.
“The internet is a big farm where you can just go out and harvest whatever you need,” Illyes said. Cloudflare Radar data put bots at 57.4% of HTML traffic, ahead of humans at 42.6%. Wikimedia said bots made up 65% of its most resource-consuming traffic earlier this year.
Robots.txt: a suggestion, not a wall
Robots.txt, he argued, is “a suggestion” for automated clients and is only reliably respected by large crawlers. Research, privacy and malware crawlers may ignore it. Google-Agent, its user-triggered fetcher, can bypass because a person requested the fetch.
Google-Extended, the no-training token, is often spoofed. GreyNoise counted 263,849 forged sessions using that name even though no genuine request should carry it.
What to do now
- Authenticate every crawler by reverse and forward DNS, not user-agent.
- Load Google’s published IP ranges into your WAF or firewall rules.
- Block unknown or failed fetchers at network level; robots.txt cannot enforce.
Google is testing Web Bot Auth, which signs selected request headers with cryptographic keys under RFC 9421, so a site can verify the sender. It currently covers only a subset of Google-Agent requests, so keep the IP and DNS checks running.
Source: PPC Land
