What llms.txt actually is
llms.txt is a proposed standard designed to give large language model crawlers a curated map of your site’s most important content. Instead of letting a bot wander through JavaScript-heavy pages, navigation menus and years of old posts, you hand it a Markdown file that says: these are the pages worth reading.
It sits alongside robots.txt and XML sitemaps, but the intended audience is different. robots.txt is about what bots can access; an llms.txt file is about what AI systems should prioritize.
The hype is ahead of the evidence
As of now, no major AI platform — not ChatGPT, Google’s AI Overviews or Anthropic Claude — has confirmed that it reads or relies on llms.txt. Google’s John Mueller said as much on Bluesky. Anthropic has published an llms.txt file on its own domain, which signals interest, but not that its crawler consumes these files.
Google’s separate Open Knowledge Format is also worth watching, but it is not the same thing. OKF is built for an organization’s internal AI agents, while llms.txt faces outward toward crawlers.
Semrush tested an llms.txt file on its sister site Search Engine Land. The result? No meaningful lift in AI visibility and zero visits from Google-Extended, GPTBot, PerplexityBot or ClaudeBot between mid-August and late October 2025.
Adoption is real but still tiny
The tooling is maturing. Lighthouse now checks for llms.txt in its experimental Agentic Browsing audits, and adoption has grown fast:
- 951 domains had published an llms.txt in July 2025
- 16,670 domains had one roughly 14 months later
- That is still a sliver of the hundreds of millions of active sites
The window is open, but the signal is weak. You can publish one without being crowded out, but you should not expect it to move rankings or AI citations yet.
For performance marketers, the takeaway is not that one file will unlock AI answers. It is that AI visibility still depends on crawlability: clean HTML, JavaScript rendering, internal structure and content that is worth citing. llms.txt is one small, optional signal.
How to test it without overcommitting
If you want to run a low-cost experiment, treat llms.txt as a curation exercise that might become useful later. Create a plain Markdown file named llms.txt, place it at your root directory, and keep it focused.
Good candidates include product pages, pricing, a clear about page, contact information and up-to-date evergreen content. Skip gated pages, duplicate pages, personalized URLs and anything that will expire soon.
Once it is live, confirm the URL loads and check server logs periodically. If AI crawlers start requesting the file, that is your signal to pay closer attention.
Source: Semrush Blog
