- No confirmed consumption. As of August 2026, the crawler documentation from OpenAI, Anthropic, Google and Perplexity describes robots.txt controls — none of it documents fetching llms.txt for search, retrieval or training.3456
- The logs agree. Ahrefs analyzed 137,210 domains in June 2026: 97% of published llms.txt files received zero requests in the month measured, and only 1.1% of the bot requests that did arrive came from AI retrieval bots.1
- Google is on the record twice. John Mueller compared llms.txt to the keywords meta tag (April 2025),2 and Google's own docs say you don't need new "AI text files" to appear in AI features.3
- The verdict: cheap insurance, not a lever. Worth 10 minutes with the free generator — not worth a slot on anyone's quarterly roadmap.
llms.txt has split the GEO world into two camps that mostly argue past each other. One camp sells it as the robots.txt of the AI era — ship the file, get cited. The other calls it dead on arrival. Both camps tend to run on vibes.
There's no need for vibes anymore. Two years after the proposal, we have real evidence: engine documentation you can read, and server-log studies with six-figure sample sizes. This post is the companion to our free llms.txt generator — the tool builds the file in a minute, and this is the honest answer to the question you should ask before you bother: does it do anything?
What "working" would even mean
Be precise about the claim being tested. For llms.txt to "work," three things have to happen in order:
- Engines fetch it. A crawler or agent has to request the file at all.
- Engines use it. The content has to influence what gets read, retrieved or repeated about you.
- Visibility moves. The effect has to show up in measured mentions and citations, not in a vendor's testimonial.
We now have evidence on the first step, official documentation on the second, and — because the first step mostly doesn't happen — nothing credible on the third. Let's take them in order.
What the server logs show
The largest public study to date is Ahrefs' analysis of 137,210 domains with traffic in May 2026, published June 15, 2026.1 The team checked which domains served a valid llms.txt, then classified every request those files received by user agent. The headline numbers:
| Finding | Number | What it tells you |
|---|---|---|
| Domains publishing a valid llms.txt | 28% | Producing the file is mainstream — at least among the tech-forward sites in an analytics vendor's sample |
| Published files with zero requests in the month | 97% | The overwhelming majority of llms.txt files are never asked for |
| Share of llms.txt requests that came from bots | 96% | Almost nobody reads it by hand — this is a machine-to-machine file |
| Bot requests from AI agents / agentic tools | 10.5% | Coding and research agents are the file's most real audience |
| Bot requests from AI training crawlers | 5.3% | Occasional pickups, no documented use |
| Bot requests from AI retrieval bots | 1.1% | The bots that produce citations barely touch it |
The study's own caveat makes the picture stricter, not kinder: a fetch only proves a request, not that anything was read or acted on. The authors call their percentages a generous upper bound on real AI consumption.1
This matches what site owners see one server at a time. It's also exactly the check John Mueller pointed to a year earlier: "you can tell when you look at your server logs that they don't even check for it."2 Your own access log is the ground truth here — grep it for llms.txt before you believe anyone's claims about the file, including ours.
What the engines themselves say
Documentation is the other half of the evidence, because it's the only place an engine commits to behavior. Here is the state of the written record as of August 2026:
| Engine | Documents consuming your llms.txt? | What the docs actually cover |
|---|---|---|
| No — explicitly not needed | "You don't need to create new machine readable files, AI text files, or markup" to appear in AI Overviews or AI Mode3 | |
| OpenAI | No | GPTBot, OAI-SearchBot and ChatGPT-User, all controlled via robots.txt; llms.txt unmentioned4 |
| Anthropic | No | ClaudeBot and robots.txt directives; llms.txt unmentioned5 |
| Perplexity | No | PerplexityBot and Perplexity-User, robots.txt and WAF guidance; no llms.txt consumption documented6 |
One distinction keeps tripping people up: several AI companies publish llms.txt files for their own documentation sites — Anthropic and Perplexity among them.6 That makes them producers, betting the same hour on the convention that you would. It is not evidence that their assistants consume the file from your site, and none of them claims it is.
And Mueller's full comparison is worth quoting, because it names the structural problem rather than just the adoption gap: "To me, it's comparable to the keywords meta tag — this is what a site-owner claims their site is about."2 A self-declaration is cheap to fake, so engines that got burned by meta keywords two decades ago have little reason to trust a new one. Any future llms.txt support would have to solve that trust problem, not just parse the markdown.
So who does read it?
The file isn't entirely unread — and the 3% of files that do get requests sketch the real audience. In the Ahrefs data, the biggest identifiable AI consumers were agents and agentic infrastructure, ahead of training crawlers, with retrieval bots barely registering.1 That matches the anecdotal server-log reports: the requests come from coding assistants, research tools and browsing sessions where an agent (or its user) explicitly went looking for a shortcut.
In other words: llms.txt today is a file for the agentic web — the tools that navigate your site on someone's behalf — not for the consumer chatbots that decide whether your brand gets recommended. If your customers are developers whose IDE agents crawl your docs, the file has a small, real audience right now. If you're waiting for it to move your ChatGPT mention rate, the evidence says you'll be waiting.
Don't confuse the two root files. robots.txt is access control — it decides which AI crawlers may read your site at all, and every major engine documents honoring it. llms.txt is a courtesy map for whoever you've already let in. If you only maintain one of them carefully, make it robots.txt — and check your AI crawler access free to see how yours is set today.
The honest verdict: worth 10 minutes, not a strategy
Put the evidence together and the decision is unusually clean:
- Cost: one static markdown file and ten minutes of curation. No engineering dependencies.
- Documented harm: none. Search engines ignore it; Google says it's simply not needed.3 The only real failure mode is letting it go stale enough to contradict your live pages.
- Documented benefit: also none, today — no major engine consumes it, and no credible study links it to visibility movement.
- Option value: real but speculative. If agents standardize on the convention, you're early with your best self-description already in place.
That's a bet you take at a ten-minute price and refuse at a ten-hour price. What the evidence rules out is llms.txt as a strategy — the consultants selling llms.txt optimization sprints are selling work the consuming side of the market doesn't reward. The things that measurably move AI visibility remain unglamorous: crawlable server-rendered pages, structured data that makes your facts machine-legible, and content specific enough to quote. Sequence those first; curate the index after.
MentionBeat runs real buyer prompts across ChatGPT, Claude, Gemini and Perplexity on a fixed cadence — so when you change something, you see whether your mention rate moved against a baseline, not an anecdote.
Get a free visibility reportIf you ship one, spend the 10 minutes well
Given the price, we ship one ourselves and we build the tool for it. The free llms.txt generator produces a clean file from your name, one-line description and key pages — and stays honest with you about everything above while it does it. For the full craft — the format spec, a B2B template, and the do/don't list — read the companion piece: llms.txt: what it is, who supports it, and whether you should ship one.
Three rules cover most of the value:
- Curate, don't dump. A short list of canonical pages — product, pricing, specs, docs — beats a sitemap clone. Curation is the entire point of the format.
- Put the facts in the file. Category, buyer, differentiator, price anchor. An agent that reads nothing else should still describe you correctly.
- Review it when positioning or pricing changes. A stale self-description quietly feeding agents old facts is the one way this file can hurt you.
Frequently asked questions
No. As of August 2026, the crawler documentation from OpenAI, Anthropic, Google and Perplexity describes robots.txt controls and says nothing about fetching or using llms.txt from your site.3456 Several of those companies publish llms.txt files for their own docs sites — but publishing one and consuming yours are different things.
Neither. Google's guidance on AI features states you don't need to create new machine-readable files or "AI text files" to appear in AI Overviews or AI Mode,3 and John Mueller has said Google doesn't use llms.txt.2 Search engines ignore the file, so there's no documented ranking effect in either direction.
Watch two things. Your server logs: if engine crawlers start requesting /llms.txt, the user agents will show up there first — that's how the current evidence was gathered, and today most files get no requests at all.1 And your measured visibility: track mention and citation rates on a fixed cadence, so a change after shipping the file shows up against a baseline instead of a screenshot. That second habit is worth having regardless of what llms.txt ever becomes.
Sources & further reading
- Linehan, L. & Guan, X. / Ahrefs — "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read", June 15, 2026. Sample: 137,210 domains with traffic in May 2026.
- Search Engine Journal — "Google Says LLMs.Txt Comparable To Keywords Meta Tag", April 17, 2025 — reporting John Mueller's Reddit comments.
- Google Search Central — "AI features and your website" (last updated December 2025).
- OpenAI — "Overview of OpenAI crawlers" (GPTBot, OAI-SearchBot, ChatGPT-User).
- Anthropic — "Does Anthropic crawl data from the web, and how can site owners block the crawler?"
- Perplexity — "Perplexity crawlers" (PerplexityBot, Perplexity-User) — which also links Perplexity's own llms.txt for its docs site.
- Howard, J. / Answer.AI — "The /llms.txt file" proposal, September 2024.
Technical SEO Engineer at MentionBeat. Tomas lives in server logs and crawler documentation — he tracks how AI bots actually behave in the wild and builds the technical checklists behind MentionBeat's recommendations.
