If you added an llms.txt file to your site last year and quietly assumed it was doing something, this one is for you.
Common Crawl analysed more than 500,000 llms.txt files and published the findings via senior research engineer Malte Ostendorff. The headline: most of these files are auto-generated boilerplate, a chunk of them contain nothing useful, and a worrying slice of site owners appear to be confusing llms.txt with robots.txt.
What the numbers say
The shape of the llms.txt ecosystem, according to the crawl:
- 68% of files came from a plugin or template, not a human writing curated guidance.
- Wix alone accounted for 41% of the files examined.
- 22.56% contained no links at all.
- 49% matched the full structure tested for: an H1 title, a summary blockquote and sections of link bullets. The spec only requires the H1.
- 6% included AI usage guidance such as rate limits or copyright notices, which the llms.txt standard does not actually cover.
- Across 73,000 files, All in One SEO never writes a summary and has a median of 138 links, effectively treating llms.txt as a sitemap.
GoDaddy’s parked-domain files even passed the structural checks with no links, functioning as a sales pitch rather than site guidance.
The robots.txt confusion problem
Here is the part that should make SEO teams sit up. Common Crawl found 136,578 robots.txt files sitting at the llms.txt path, roughly 10% of the 1,287,207 responses that returned HTTP 200. People are literally uploading the wrong file to the wrong place.
Worse, 1,570 of the analysed files named a specific crawler, and 32 looked like they were trying to deny Common Crawl’s own CCBot. The team pulled robots.txt for all 32 and could assess 31. None of them blocked CCBot outright. Five explicitly allowed it, 11 restricted certain paths but permitted others, and 15 had no restrictions at all.
One site’s llms.txt claimed CCBot was blocked “as of June 2026” while its robots.txt allowed the crawler under a wildcard rule.
Common Crawl’s own framing is blunt: the file “grants nothing and blocks nothing, and no crawler is obliged to read it.” It describes llms.txt as a policy description rather than a policy, and says it has drifted out of sync with the robots.txt it claims to describe.
Prompt injection: real, but rare
Ten files matched the strictest test for instructions aimed at the model itself. Four turned out to be genuine, including one whose summary told the model to ignore prior instructions and fetch a second file. The other six were false positives, mostly documentation quoting control tokens to explain them.
The authors are clear that llms.txt is not overrun with injections, and that the real cases looked deliberate, made to prove a point. A milder tier of 3,793 files used soft guidance like asking the model to focus on certain pages.
What growth and SEO teams should do
Treat this as a hygiene checklist, not a panic moment:
1. Audit what your CMS is publishing. If two-thirds of files are plugin output, odds are yours is too. Open /llms.txt on your own domain and read it.
2. Move all access rules to robots.txt. Common Crawl says robots.txt is the file CCBot honours. A crawler restriction written into llms.txt does nothing by itself.
3. Keep the two files consistent. If your llms.txt mentions crawler policy, it should mirror what robots.txt actually enforces, or delete the claim.
4. Don’t confuse a sitemap with curation. Dumping 138 links is not guidance. If you keep the file, make it a short, genuinely useful map of your best pages.
Context and caveats
Earlier Ahrefs data, covered in June, found that 97% of llms.txt files in its dataset received no requests at all. That study looked at whether anyone fetched the files; Common Crawl looked at what is inside them. Neither confirms that any AI system reads them.
The sample is random only within sites CCBot could reach, so its 11% adoption figure is not directly comparable to Ahrefs or Web Almanac numbers. It is also a single crawl, so it cannot show trends over time, and the policy and injection counts rely on keyword matching, which the post concedes likely undercounts.
The llms.txt v2 update from August added link relations to help surface Markdown versions of pages, but it did not change what the file can enforce. With plugins and site builders producing about two-thirds of files, the format’s real-world behaviour now depends less on what marketers write and more on what their tools ship by default.
Source: Search Engine Journal
