OpenAI’s ChatGPT search just became a much wider net. Cloudflare and OpenAI say an early research pilot surfaced hundreds of millions of web pages that ChatGPT can now discover and evaluate.
The pilot involved roughly 1% of Cloudflare’s network, and the companies shared initial results on Oct. 15. Cloudflare first announced the test on July 8.
A new path for pages that don’t earn links
Cloudflare supplies network signals to help OpenAI’s crawler find pages that may be useful even when almost nobody links to them. The examples are deliberately unglamorous: a washing machine manual, current passport fees, a restaurant’s menu and opening hours.
“A signal about a page doesn’t grant permission to access it.”
The update signals include page freshness, traffic quality and what actually changed on a page. The companies have not said which sites took part or whether the newly found pages sit on participating domains.
Smarter crawling, fewer wasted requests
The same signals tell OpenAI when not to crawl. Cloudflare describes bots hammering missing files, redirects that send every URL to the homepage, and duplicate content. Change signals help the crawler decide whether a revisit adds value. The companies are also testing flags for harmful pages, such as a planted login form on a legitimate site, aiming to remove one bad page instead of an entire domain.
The stated goal is to cut unnecessary load for Cloudflare customers, though no crawl-volume figures were published. The post also notes that discovery can initially increase crawling while new pages are evaluated.
What to do in robots.txt
- Allow OAI-SearchBot if you want ChatGPT Search to cite your pages.
- Block GPTBot separately if you don’t want your content used for training.
- Check Cloudflare’s AI Crawl Control to see which AI bots visit and set access.
OpenAI separates search and training crawls. If you block OAI-SearchBot, your site won’t be included in ChatGPT’s search results, although it could still show up as a navigational link. If you allow both bots, data from one crawl can be used for both search and training.
Visibility does not equal citations
Getting indexed is only step one. OpenAI says it independently evaluates and ranks pages, so being discovered through the pilot is no guarantee ChatGPT will mention you.
For tracking, ChatGPT referral visits depend on UTM tags or referrers. Earlier this month some ChatGPT links carried neither; a fix began rolling out around Oct. 15. Watch server logs and tag experiments carefully.
What’s next
OpenAI and Cloudflare plan to improve how AI selects pages, test faster content-change signals, and measure results when AI finds new content. Cloudflare expects to offer broad access to this capability later in 2026, though no date has been set.
The practical move today is small: separate your search and training permissions, then measure AI search traffic before broad rollout arrives.
Source: Search Engine Journal
