llms.txt: You Are Probably Being Sold the Wrong Use Case

In plain terms

The file is a table of contents written for machines. A person browsing your site sees menus, images and adverts. A language model reading the same page has to wade through all of it. It offers a stripped-down list instead: here are the pages that matter, here is what each covers.

TL;DR

• Ahrefs studied 137,210 domains in May 2026 and found 97% of llms.txt files received zero traffic. AI retrieval bots barely fetch them, and never probe for one on sites that have not published it.
• In server logs covering 83 sites between April and July 2026, the same crawlers that fetched robots.txt thousands of times fetched llms.txt in single digits. PerplexityBot fetched it zero times.
• Google has said plainly that it does not use llms.txt. Gary Illyes confirmed in July 2025 that Google does not support it and has no plans to, and John Mueller compared it to the keywords meta tag.
• But the same Ahrefs dataset found Claude Code fetched llms.txt more than any AI retrieval bot did. Coding agents read it. Answer engines do not.
• So the question is not whether llms.txt works. It is what you are using it for. Publishing developer documentation: worth twenty minutes. Chasing ChatGPT citations: no evidence, and structural reasons to expect none.

The file sits inside a wider set of tactics sold for AI visibility, most of which are equally worth checking. Our guide to answer engine optimization covers which ones have evidence behind them.

What is llms.txt?

The file is a plain-text document placed at a website's root, at yoursite.com/llms.txt, listing in Markdown the pages an AI system should read and what each one covers. Jeremy Howard of Answer.AI proposed the format in September 2024. It is a community convention rather than an official standard.

Three terms worth knowing before the rest of this makes sense:

• A crawler or bot is software that visits web pages automatically. Search engines and AI companies each run several.
• Markdown is a simple text format using plain symbols for headings and links. It is much lighter than the HTML that browsers read.
• A context window is how much text a language model can hold in mind at once. Wasting it on navigation menus and advert code leaves less room for your actual content.

There is also llms-full.txt, a longer companion file that includes the full text of the listed pages rather than just links to them, so an AI system can take everything in one fetch.

Important: as of mid-2026 the format has no recognition from the W3C, the IETF or schema.org. There is no version number, no governing body and no conformance test. That is not a criticism of the idea. It is context for every claim made about it, including in this article.

In plain terms, what does it do?

It hands an AI system a clean list instead of making it read a messy website.

A normal web page is built for humans. It contains navigation, images, tracking code, cookie banners and adverts. A language model reading that page spends a large part of its limited attention on material that carries no meaning. Companies that serve plain Markdown instead of HTML have reported substantial reductions in the amount of text a model has to process for the same content.

The file is the index for that idea. It says: here are the twelve pages that matter on this site, here is a one-line description of each, go and read those rather than crawling everything.

That is the design. Whether anything acts on the file is a separate question, and it is the next section.

Does anything actually read llms.txt?

Search and answer engines: almost never. Coding agents: yes. That split is the whole story and most coverage reports only the first half.

The largest public dataset comes from Ahrefs, published 15 June 2026, covering 137,210 domains with server-side analytics and measuring May 2026 traffic. Its findings:

97%: the share of llms.txt files that received zero traffic (Ahrefs, 137,210 domains, June 2026)

• AI retrieval bots, meaning the crawlers that actually feed answers in AI search products, barely register in llms.txt request logs.
• AI bots never requested the file on domains where it did not exist. They are not probing for it. If these systems wanted the file, cheap probing is exactly what you would expect to see.
• Roughly 12% of the small traffic these files do receive comes from SEO and AEO audit tools checking whether the file exists. Part of the llms.txt economy is tools measuring compliance with a convention almost nobody consumes.

A second dataset compared robots.txt against llms.txt fetch counts on 83 sites between 27 April and 19 July 2026 (EZY Research):

Read the last two rows. Perplexity fetched the file zero times across three months. Meta's crawler is the single exception in the dataset, fetching it slightly more often than robots.txt.

On adoption, SE Ranking's analysis of roughly 300,000 domains found around 10% publishing the file, and found no correlation between having it and being cited by AI systems.

And now the finding that changes the conclusion. In the same Ahrefs dataset, Claude Code fetched the file more than any AI retrieval bot did. Coding agents, the tools developers point at documentation, are the one category genuinely using it. That fits the format's origin, which was a documentation problem rather than a marketing one.

What llms.txt is not

Five things the file gets confused with, and what actually does each job.

The recurring confusion is with robots.txt, and it matters because they are opposites in reliability. robots.txt is thirty years old and universally honoured. The newer file is two years old and, on the evidence above, largely unread by the systems it names.

So who is llms.txt actually for?

Developers pointing coding agents at documentation. That is the use case with evidence behind it, and it is the one the format was designed for.

Jeremy Howard's original proposal came out of a documentation problem: a developer asks an AI coding assistant to use a library, the assistant needs the library's docs, and HTML documentation is a poor way to deliver them. The format gives the assistant a clean map. Ahrefs' finding that Claude Code fetched these files more than any AI retrieval bot is that design working as intended.

So the honest decision rule is not whether the format works. It is:

Do you publish documentation that people point AI coding tools at?

• Yes. A software product with developer docs, an API, a library, a technical knowledge base. llms.txt has a real, evidenced use. Publish one, keep it current, and consider llms-full.txt as well.
• No. A marketing site, a services business, an ecommerce store, a blog. There is no evidence it will get you cited more, and there are structural reasons below to expect that not to change.

The file is not failing. It is being sold for a job it was never built to do. Almost every article recommending it for AI search visibility is recommending it on the strength of a name rather than a finding, and almost every article debunking it is measuring a use case its author never claimed.

What does Google say about llms.txt?

Google does not use it, and has said so more than once. Its guide to optimizing for generative AI features states that you do not need to create machine-readable files, AI text files, markup or Markdown to appear in Google Search including its AI features, because Google Search does not use them.

Two named statements go further:

• Gary Illyes, July 2025, confirmed Google does not support llms.txt and is not planning to.
• John Mueller, June 2026, observed that the file has existed for years and none of the AI systems use it, and separately compared it to the keywords meta tag.

That comparison is the sharpest critique available and deserves unpacking, because it explains why this is unlikely to change.

The keywords meta tag let site owners declare what a page was about in text that visitors never saw. Search engines abandoned it because self-declared invisible metadata is trivially manipulable: anyone can claim anything. The newer file has the same shape. An answer engine ranking or trusting sources on a file the site owner wrote about itself would be inviting exactly the manipulation that killed the keywords tag.

That is a structural reason to doubt adoption, not a temporary gap. It is not that the AI companies have not got round to it yet. It is that reading it would create a problem they would then have to solve.

Should you publish one?

The honest summary for most readers of this page: if you are here because the file appeared on an SEO proposal, ask what evidence the proposer has that it produces citations. There is currently none, and the burden of proof sits with them.

If you publish documentation, spend the twenty minutes. It is cheap, harmless and used by the one category of tool that genuinely reads it.

How do you create an llms.txt file?

Fifteen minutes by hand, and doing it by hand is better than a generator because the descriptions are the useful part.

1. Create a plain text file with that name.
2. Start with an H1, a single # followed by your site or product name.
3. Add a one-paragraph summary in a blockquote, using >, describing what the site is for.
4. Add H2 sections using ## to group your pages. Common groupings are Documentation, Guides, API Reference and About.
5. Under each heading, list links in Markdown as - [Page title](https://yoursite.com/page): one-line description of what it covers. The description is the part an agent uses to choose, so write it properly.
6. Add an ## Optional section for pages that can be skipped if the agent is short on space.
7. Upload it to your site root so it resolves correctly.
8. Check it loads as plain text in a browser and returns a 200 status.

If you also want llms-full.txt, that follows the same structure but includes the full Markdown content of each page rather than links. It is considerably larger and needs regenerating whenever the docs change, which is a maintenance cost worth planning rather than discovering.

Set a review reminder. A file listing pages that no longer exist is worse than none, and unlike a sitemap nothing will tell you it has gone stale.

What should you do instead?

If the goal is being cited by AI systems, four things have evidence behind them, and none of them is a file.

1. Be crawlable and snippet-eligible. Google requires a page to be indexed and eligible to appear with a snippet to appear in its AI features. Check for noindex and nosnippet on pages you want quoted, and confirm your CDN is not blocking legitimate crawlers.
2. Write extractable answers. A self-contained 40 to 70 word answer directly beneath a question heading, with the subject named rather than referred to as "it". This is covered in detail in our guide to ranking in AI Overviews.
3. Build third-party presence. The sources AI assistants retrieve from are frequently not your website. Review sites, directories, comparison pages and communities in your category carry more weight than anything you publish about yourself.
4. Measure your AI referral traffic so you can tell whether any of this is working. Guessing is how twenty-minute tactics end up on roadmaps in the first place.

Notice that all four are unglamorous and none of them is a file you can add on a Friday afternoon. That is the actual reason this format became popular.

Conclusion

You can now answer the question in the title with a rule rather than an opinion. If you publish developer documentation or an API, spend the twenty minutes: coding agents genuinely fetch these files and that is what the format was built for. If you are a marketing or services business hoping for more AI citations, skip it, and ask anyone recommending it what evidence they have. Then spend the effort on crawlability, extractable answers and third-party presence, which are duller and have data behind them. If you want the AI-visibility work done on evidence rather than on tactics that sound plausible, talk to Euracle about AEO and GEO.

Sources

1. Google Search Central, Optimizing your website for generative AI features on Google Search, last updated 10 July 2026. States that Google Search does not use machine-readable AI files. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
2. Ahrefs, study of 137,210 domains with server-side analytics, measuring May 2026 traffic, published 15 June 2026. The 97% zero-traffic finding, the absence of probing, the 12% audit-tool share, and the Claude Code finding.
3. EZY Research, server-log comparison across 83 sites, 27 April to 19 July 2026. robots.txt against llms.txt fetch counts by crawler.
4. SE Ranking, analysis of approximately 300,000 domains, adoption rate of around 10% and no correlation with AI citations.
5. Gary Illyes, Google, July 2025, confirming Google does not support llms.txt.
6. John Mueller, Google, June 2026, on the file existing for years without AI systems using it, and the keywords meta tag comparison, reported by Search Engine Journal.
7. Jeremy Howard, Answer.AI, original llms.txt proposal, September 2024.

Written by

Devanshu Takkar

AI-Native Product Manager + GTM Engineer