Cloudflare’s September 15 update did more than change bot settings. It removed the machine-readable “no training” signal from 118 sites that had relied on Cloudflare to write their robots.txt rules, according to a new re-run by HasData.
The web scraping provider re-tested the same 10,894 domains on September 16 and compared the results with a July baseline. Among Cloudflare-hosted sites, the share disallowing OpenAI’s GPTBot fell from 17.1% to 9.9%. Outside Cloudflare, it barely moved.
What disappeared
HasData pulled archived robots.txt copies for 133 Cloudflare sites that stopped disallowing GPTBot. In 118 of them, the old file carried Cloudflare’s managed block: a Content-Signal line with ai-train=no, a rights reservation citing Article 4 of the EU copyright directive, and a GPTBot ban. By September 16, none of the 118 still carried it.
Named examples include the Smithsonian, Vatican News, National Review, Bellingcat and Charlie Hebdo. Patreon, which writes its own ai-train=no line scoped to two named crawlers, kept its signal. The replacement Bot Preference Sync did not appear in any of the 6,451 robots.txt files HasData collected.
Rules and reality already diverged
The 118-site finding sits inside a bigger gap. In July, 39.5% of sites that disallowed GPTBot still returned a live page to a request claiming to be GPTBot. “Robots.txt is a suggestion,” as HasData co-founder Roman Milyushkevich put it in the study.
Enforcement broke down like this for sites that did block GPTBot:
- 22.2% returned hard blocks (403, 401 or 451)
- 10.9% returned HTTP 402 Payment Required, Cloudflare’s pay-per-crawl signal
- 5.0% served Cloudflare-style challenges or JavaScript checks
- 2.4% rate-limited the request
- 2.3% showed CAPTCHAs
Publishers served a Chrome browser request 83.8% of the time and a GPTBot request only 54.2% of the time, a gap that widened in September. So enforcement is largely identity-based: the User-Agent header matters more than the IP.
What marketers should do
Do not treat a robots.txt line as proof of protection. Run a live test: send a request with the GPTBot user-agent, and send another with a normal Chrome user-agent as a control. If both return a 200, your public rule is not enforced at the door.
If you use Cloudflare, verify that your own custom rules survived the migration and check whether Bot Preference Sync is active. Also remember that blocking has a cost. One Rutgers and Wharton paper associated AI crawler blocks with roughly 7% weekly traffic loss within six weeks, while blocking OpenAI’s crawler does not remove a site from Google’s AI answers.
Source: PPC Land
