Marketers are racing to make their brands discoverable inside AI answers. The temptation is obvious: publish a tidy llms.txt file, spin up agent-only pages, or serve special markdown feeds so chatbots pick the “right” story. But that shortcut may be rebuilding a problem search engines spent two decades trying to kill.
The distinction that matters isn’t between human-readable and machine-readable. It’s between making the same information easier for agents to consume and creating different information that only agents ever see.
Cloudflare’s Markdown for Agents, for example, presents the same page content in a cleaner format for machines. That’s plumbing, not manipulation. The danger starts when the machine-facing version says something the public site doesn’t, or exists only to shape what an AI says about a brand.
A shortcut with a familiar shape
SEO veterans have seen this pattern before. Early tactics like hidden text, doorway pages and cloaking were designed to influence search engines rather than serve users. Search engines didn’t crack down because they disliked SEOs; they cracked down because falsified relevance destroyed result quality.
Google’s own timeline makes the pattern explicit:
- 2011: Panda targeted low-quality content.
- 2012: Penguin tackled link spam.
- 2022: The Helpful Content system rewarded content made for people, not search traffic.
- 2024: Updated spam policies took on scaled content abuse, AI-generated or not.
The playbook is familiar: every time marketers find a new lair, search engines invest in judging what deserves trust. AI search providers have the same incentive. They need accurate, useful answers to keep users. They have little reason to legitimize a fragmented pile of agent-only sources they can’t rely on.
The trust problem no one has solved
The shift to AI search makes this harder, not easier. Google isn’t the only retrieval path anymore. Different systems, tools and pipelines can shape what an agent finds and uses. That doesn’t mean ChatGPT or another assistant will automatically trust everything, but it does mean there’s less of the accumulated anti-spam muscle that traditional engines built.
And when an agent encounters content created purely to influence its answer, a few uncomfortable questions follow: How does it know the AI-only version matches what real customers see? If the machine version contradicts the public page, which one wins? And why would an AI provider keep accepting information users can’t independently verify?
Those questions don’t have answers yet, which is exactly why betting the strategy on them is risky.
What to do instead
The answer is not to reject machine-readable formats. Structured data, XML sitemaps, APIs and llms.txt files are useful plumbing. The answer is to stop building a second content estate that requires its own maintenance, governance and tech stack.
If most teams struggle to keep one website accurate, adding a hidden second one isn’t a solution. It’s an old problem wearing a new shortcut.
The sensible middle ground: make genuinely useful information available from one consistent, authoritative source. Add the technical formats agents need, but keep the facts identical to what customers can read and verify. Useful only to AI creates a governance problem. Useful only to people creates a visibility problem. We don’t need a shadow web to fix it.
Source: Search Engine Journal
