llms.txt is a plain-markdown file, published at a site's root, meant to give AI systems a curated map of the content worth reading. It is easy to create and genuinely useful for AI coding assistants pulling documentation. What it does not have, as of October 2026, is any confirmed evidence that major answer engines fetch it, or that publishing one changes how often a brand gets cited.
The file shows up constantly in answer engine optimization checklists, usually pitched as a quick win: spend twenty minutes writing a markdown file, improve your odds of being understood by ChatGPT, Claude, Gemini and Perplexity. The pitch is appealing because it is cheap and because it borrows the shape of two things SEO teams already trust - robots.txt and sitemap.xml. Whether the comparison holds up is a separate question from whether the file is worth publishing, and the two get conflated constantly.
What llms.txt actually is
The specification comes from Jeremy Howard (Answer.AI), first published 3 September 2024, with a version 2 update released 10 August 2026 after two years of real-world use. The file lives at `/llms.txt` in a site's root (or a subdomain, for documentation sites), written in plain markdown rather than a structured format like XML: an H1 with the site or project name, a one-line blockquote summary, optional context paragraphs, and then H2-delimited sections listing links with short descriptions. The intent is to hand an AI system a short, curated, ad-free map of what matters on the site, instead of making it parse full HTML pages built for human eyes.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Tells crawlers what they may or may not fetch | Lists every URL worth indexing | Curates the pages worth reading, with context |
| Format | Plain-text rules | XML | Markdown, human-readable |
| Enforceable? | Honored by compliant crawlers, not enforced | Advisory, not enforced | Advisory only - nothing requires any system to open it |
| Confirmed readership | Every major search and AI crawler respects it | Search engines read it routinely | Unconfirmed by any major answer engine as of Oct 2026 |
How many sites actually publish one
Adoption is real and growing - the disagreement is only about how big the number is, because different studies sample different populations. HTTP Archive/CrUX data (Casey Burridge, 20 June 2026) put llms.txt on 5.07% of the top 1 million sites, 5.61% of the top 10,000, and 6.28% of the top 1,000 - up from 1.04% in July 2025. SE Ranking's study of roughly 300,000 domains (7 November 2025) found 10.13% adoption, flat across traffic tiers (9.88% on sites under 100 monthly visits, 10.54% on mid-traffic sites, 8.27% on the largest). Ahrefs' narrower sample of 137,210 domains with measurable traffic (15 June 2026) found 28% adoption, skewed hard by platform: 78.1% of Shopify stores had a file versus 8.7% of WordPress sites, which tracks with Shopify auto-generating one by default.
| Source | Date | Sample | Adoption rate |
|---|---|---|---|
| HTTP Archive / CrUX | Jun 2026 | Top 1,000 sites | 6.28% |
| HTTP Archive / CrUX | Jun 2026 | Top 1M sites | 5.07% |
| SE Ranking | Nov 2025 | ~300,000 domains | 10.13% |
| Ahrefs | Jun 2026 | 137,210 domains with traffic | 28% overall (78.1% Shopify, 8.7% WordPress) |
Does anyone's AI system actually fetch it
This is the part the twenty-minute pitch usually skips. Ahrefs' server-log study (May 2026) found that 97% of llms.txt files received zero requests in the measured window. Of the requests that did land, 96% came from bots generally and 77% of those were non-AI bots (generic crawlers, uptime monitors, scrapers) rather than anything resembling an answer engine. Among named AI agents, GPTBot accounted for 4.51% of hits, ClaudeBot 0.80%, DeepSeek 0.02% - and even Claude Code, Anthropic's own coding tool, fetched the file more often than any retrieval-style bot did. A separate monitoring run (Limy, cited July 2026) tracked over 500 million AI bot visits across a 90-day window and found only 408 of them targeted `/llms.txt` directly.
The clearest on-record statement from a search engine came from Google's John Mueller at Google Search Central Live in Madrid (9 April 2025): he said he was not aware of any AI system actually using the file. One further data point, more amusing than rigorous: in August 2026, Mark Williams-Cook of Candour published a deliberately false llms.txt describing a set of office cats, and the file passed every one of the common proofs people cite as evidence llms.txt "works" - a reminder that most of those proofs (a file that validates, a file that gets crawled by something, a file search engines don't reject) do not actually demonstrate an answer engine reading or acting on the content.
Does it move your AI citation rate
The one outcome study on record, SE Ranking's correlation analysis across its ~300,000-domain sample, found no statistically significant relationship between having an llms.txt file and how often a domain got cited in AI answers. When the file's presence was removed as a variable from their predictive model, model accuracy improved rather than worsened - the variable was adding noise, not signal. That is a single vendor study with no independent replication, which is itself part of the problem: there is exactly one measured-outcome test in circulation, and it points the opposite direction from the twenty-minute-win pitch.
So is it worth publishing
Probably yes, as a low-cost addition - just not for the reason most checklists give. The strongest confirmed use case is AI coding assistants and developer tools: Cursor, GitHub Copilot and Claude fetching documentation in real time benefit from a curated, token-efficient map, and that is the corner of the ecosystem where llms.txt demonstrably gets read. If the goal is getting a brand named or cited inside ChatGPT, Perplexity, Gemini or Google's AI Overviews - the thing this site tracks - publishing the file is not shown to help, and spending real effort restructuring content around it instead of the fundamentals covered in what answer engine optimization actually is is effort misallocated for now.
The more useful move, either way, is not inferring visibility from a crawl file but checking it directly. Our GEO audit guide sets out a manual way to run that check yourself in an afternoon; the best AI SEO tools ranking covers the trackers that automate it on a schedule, sampling real questions across multiple assistants rather than hoping a markdown file got read. A rising reason to bother at all - fewer searches end in a click than they used to - is covered separately in zero-click search.