ChatGPT citations are becoming the new rankings, but a fresh preprint from researchers at the National University of Singapore and Xi’an Jiaotong University suggests many AI-visibility dashboards are only seeing part of the picture.
Posted on arXiv on October 11, the paper compared the pages ChatGPT’s web app actually cited with two common stand-ins: Google organic results pulled through SerpAPI and OpenAI’s Responses API with web search enabled. For GPT-5.5 Instant, the combined stand-ins found only 26.2% of cited pages. At the site level, the match was nearly double: 48.7% of cited sites.
That gap is the story for marketers. Page-level tracking is likely undercounting, while domain-level visibility is a more stable signal.
How the test was built
Researchers used 80 questions from Search Arena, a public database of real user interactions with search-enabled chatbots. The set split evenly between English and Chinese, and covered recommendation, fact lookup, information synthesis and analysis requests. GPT-5.3 Instant and GPT-5.5 Instant were each run 20 times per question; GPT-5.2 Instant and GPT-5.4 Thinking were phased out before collection could finish.
For the API comparison, the team sent 20 queries per question to OpenAI’s API with web search enabled. For the Google comparison, SerpAPI pulled up to five organic results per query, averaging about 43 results per question. A page counted as cited if ChatGPT selected it as a source in at least one of the 20 runs.
Google alone found 9.4% of GPT-5.3’s cited pages and 6.4% of GPT-5.5’s pages. The API’s source list was broader at 22.9% and 22.5%, but the pages the API actually cited in its answers were a closer match to ChatGPT. GPT-5.5 cited 23.3% of the pages the API cited, up from 16.4% for GPT-5.3.
Why the gap keeps showing up
The paper points to three reasons citations are hard to pin down:
- Prompt rewriting. ChatGPT almost never searches the user’s exact words. For Chinese questions, GPT-5.3 Instant and GPT-5.5 Instant used only English searches in 67.0% and 69.6% of visible runs.
- Run-to-run variance. 44.5% of distinct pages for a given question appeared in only one of 20 GPT-5.5 runs, and 40.8% did for GPT-5.3.
- Model drift. Two collection sessions switched models mid-run. Separately, Sistrix data around GPT-5.5’s May change showed day-to-day variation in German citations jump to 47%, versus a normal 1% to 2%.
Overlap was higher at the site level than the page level, which is why brand-domain visibility is more consistent than any individual URL.
What to do with this
Don’t abandon AI-visibility tracking, but stop treating it like a static ranking. Monitor page-level and site-level presence, run the same prompt multiple times, and check whether any comparison of webpage-level results uses API data or the ChatGPT app.
The paper did not test commercial tracking tools, and its figures describe models live from March 29 through July 29, 2026. GPT-5.6 Luna and GPT-6 are already rolling out, so monitoring strategies should be re-evaluated when the default model changes.
Source: Search Engine Journal
