--- title: "llms.txt adoption statistics: why the studies disagree" description: "Published llms.txt adoption figures run from under 0.005% to 57%. What each study counted, why they disagree, and which figure to quote for which sites." url: "https://goodforbots.com/blog/llms-txt-adoption-statistics" image: "https://goodforbots.com/images/blog/llms-txt-adoption-statistics.webp" author: "Mateusz Pawlica" category: "Standards" date: "2026-09-29" lastFactChecked: "2026-09" readingMinutes: 12 --- # llms.txt adoption statistics: why the studies disagree Published llms.txt adoption figures run from under 0.005% to 57%. What each study counted, why they disagree, and which figure to quote for which sites. Published figures for `llms.txt` adoption run from "less than 0.005% of all websites" to 57%, and most of them are right about what they measured. They disagree because each one counts a different set of sites with a different test of what a file is. Crawls of ranked lists of large sites, made between May and September 2026 with a check that the response is a real file, found `/llms.txt` on 5.9% to 9.3% of them. The most common way to get a higher number is to count every HTTP 200. Many sites answer any path with their home page or an error page and a 200 status. SEOmator's scan of the top 100,000 domains found 20,171 answering `/llms.txt` with a 200, and 8,598 serving a file: "a 2.35× overcount if you stop at the status line." We read every public dataset we could find at its source on 29 September 2026. Below they are lined up by method, followed by the counts that do not measure adoption at all, and our own directory's figures, which update live. ## llms.txt adoption: every public study, side by side The llms.txt proposal is a convention, not a standard: Jeremy Howard published it on 3 September 2024 and revised it as version 2 in August 2026. It makes an H1 "the only required section" and lets the file live at the root "or at any subpath". Nobody governs how adoption is counted, so every study defines it for itself. | Study | Data from | Sites measured | What counted as a file | Result | | --------------------------------- | ------------------ | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | | Rankability | Sep 2026 | Tranco top 1,000 and top 10,000 | 200, Markdown structure, and a control request to a made-up path to catch sites that answer everything; unreachable sites stay in the denominator | 9.3% of the top 1,000; 8.3% of the top 10,000 | | SEOmator | Sep 2026 | Cloudflare Radar top 100,000 | 200, not HTML, JSON, a bot challenge or a `robots.txt`, at least 20 bytes | 8.6% of all; 11.4% of the 75,353 that answered | | Chris Humphrey | Jun 2026 | Majestic top 10,000 | 200, "genuinely Markdown rather than HTML" | 7.4% | | Thunderbit | May 2026 | Tranco top 10,000 | 200, final path still `/llms.txt`, not HTML, not empty | 5.86%; 7.50% of the top 1,000 | | ProGEO.ai | Mar 2026 | Fortune 500 | not defined; automated scan, then manual review | 7.4% (37 companies) | | Casey Burridge, from HTTP Archive | Jun 2026 | Chrome UX Report origins, by rank | 2xx and a `text/plain` header | 5.61% of the top 10,000 (421 of 7,504 crawled) | | HTTP Archive Web Almanac 2025 | Jul 2025 | about 12 million desktop and 15 million mobile home pages | 2xx and a `text/plain` header | 2.13% desktop, 2.1% mobile | | Common Crawl | Jul 2026 | a random sample of hosts Common Crawl fetches without problems | 200 with a `text/plain` or `text/markdown` body; redirects not followed | 11.72% | | Ahrefs | May 2026 | 137,210 domains using Ahrefs Web Analytics | 200, Markdown rather than HTML, no error text | 28%, which Ahrefs calls "an upper bound" | | SE Ranking | published Nov 2025 | "nearly 300,000 domains", not described | not described | 10.13% | | llmtxt.info | Sep 2026 | a fixed panel of 219 notable hosts, mostly developer-facing | 200, plain text, starting with a Markdown H1; unreachable hosts excluded | 57.4% (124 of 216) | | SerpPrism | Sep 2026 | 78 hand-picked large sites | 200, not HTML, final path still `/llms.txt` | 28.2% (22 of 78) | | ALLMO.ai | Jan 2026 | the 50 domains Ahrefs lists as most cited in Google's AI Mode | not described | 1 of 50 | The spread follows selection: the more a population leans towards technical, SEO-aware or developer-facing sites, the higher its rate. The publishers say so themselves: Ahrefs' customers "skew more technical and SEO-aware than the web at large", and llmtxt.info's panel "skews toward developer-facing companies". ## Why the counts disagree: five choices Every figure above comes from five decisions, and each of them moves the result. ### Which sites are counted A top-10,000 list, a vendor's customers and a panel of developer tools are three different populations. The 7.4% for the Fortune 500 and the 57.4% for llmtxt.info's panel are both accurate. Only the Common Crawl sample is random, and its authors describe the limit: "the sample is random only within the set of hosts Common Crawl fetches successfully, which is not the same thing as the web." They also topped the sample up with 27,394 hosts already known to serve the file. ### What counts as a file The tests range from a status code to a parsed document. The HTTP Archive's metric, behind the Almanac and Casey Burridge's figures, requests `/llms.txt` from the loaded page and records it as valid when the status is OK and the `Content-Type` includes `text/plain`. It does not look for Markdown, so a plain-text error page passes. It also rejects a file served as `text/markdown`, a type the proposal does not forbid: the proposal names no content type at all. The gap between a 200 and a file is large wherever a study reports both. Thunderbit saw 1,606 responses with a 200 in the Tranco top 10,000 and 586 valid files, and wrote that a crawler counting every 200 "would overestimate valid adoption by about 2.74 times." Chris Humphrey found that 313 of 1,050 responses with a 200 were "soft 404s, serving an HTML page rather than a real Markdown file." Common Crawl: 22.32% of attempted `/llms.txt` URLs returned a 200, and 11.72% returned a text body. Of these studies, only Rankability sends a control request to a path that cannot exist, which catches a site that answers every URL with the same Markdown. ### What goes in the denominator Blocked, timed-out and unresolvable sites either stay in the denominator or leave it. Rankability keeps them: "Read the denominator. These percentages include every sampled domain, even when a site blocked the scan or could not be reached." Its 830 adopters in the top 10,000 are 8.3% of the list, but 4,211 domains stayed unknown: blocked, unreachable or inconclusive, and 2,010 of them because the domain did not resolve. As a share of the 5,789 with a definite result, the same 830 are 14.3%, by our arithmetic. SEOmator reports both ways, 8.6% and 11.4%. ### Root, subdomains and redirects Every crawl in the table requests the root path of one host. The proposal also allows `/docs/llms.txt`, and documentation often lives on its own subdomain. Crystal Carter at Wix examined over 1,400 publicly listed files and found "28% of LLMs.txt files are on subdomains" and 10% in subfolders. A root-only crawl of `example.com` misses both. Redirects split the studies too: Rankability and SEOmator follow them, Thunderbit and SerpPrism require the final path to still be `/llms.txt`, and Common Crawl records the redirect without following it. ### When, and which platform switched it on The measurements span February 2025 to September 2026, and platforms changed the defaults in between. Wix "automatically generates and maintains your llms.txt file" for its sites; Common Crawl found Wix behind 41.34% of the files it analysed. Shopify's changelog of 28 May 2026 says every store includes a default `/agents.md`, and "the paths /llms.txt and /llms-full.txt also point to this content by default." SEOmator attributes 18.2% of the files it found to Shopify. The Web Almanac's SEO chapter found 39.6% of the files in its July 2025 data related to one WordPress plugin, All in One SEO, and concluded: "we cannot be sure this is always a conscious act". A rate measured after these switches counts platform defaults as well as decisions. The same effect breaks trend lines. Rankability's first reading, 0.3% of the top 1,000 in June 2025, used a different list ranked "by organic traffic and search visibility" and excluded redirects. Rankability now says: "No month-over-month growth claim is made." ## Counts that do not measure adoption Several widely quoted numbers have no population, no stated method or both: - **BuiltWith** showed 9,850,012 live sites under "LLMS Text" on 29 September 2026. Neither the index behind the count nor the detection rule is published. - **NerdyData**'s "18,001 websites using llms.txt" counts pages whose HTML or JavaScript contains the string "llms.txt". It does not request the file. Articles quoting "951 domains in July 2025" attribute that figure to NerdyData. - **Originality.ai** reported 110,607 `llms.txt` files in September 2026 among "3M+" websites, without saying which. Its headline "sites" totals equal the sum of its llms.txt, llms-full.txt and ai.txt counts, so a site with two of the files would count twice. - **SISTRIX** says "less than 0.005% of all websites worldwide are using the file", with no method or date for the measurement. - **Google `site:` counts**, which Wix used to estimate about 120,000 indexed files in May 2026, count indexed URLs and are estimates, not a census. - **Directories** such as directory.llmstxt.cloud and llmstxthub list sites that were submitted or collected. They count listings, not a share of anything. One early figure also has a slip in it: a February 2025 crawl of the Majestic Million reported "0.015% ... 15 sites", but 15 of a million is 0.0015%. ## llms-full.txt is not part of the proposal `llms-full.txt`, one file with a site's whole documentation, is common enough to be measured: 1.03% of the Tranco top 10,000 in May (Thunderbit), 1.2% in September (Rankability) and 2.5% of the top 100,000 (SEOmator). The name does not appear in the proposal: not in the current text at llmstxt.org, and not in version 1, which mentioned only two files the FastHTML project generated from its own llms.txt, `llms-ctx.txt` and `llms-ctx-full.txt`. Version 2 dropped that passage. Mintlify, which serves `llms-full.txt` for its customers, says the format was "subsequently included as part of the official llms.txt proposal"; the proposal's text does not support that. Count the file carefully. SEOmator found that half of the `llms-full.txt` files it read were within 5% of the size of the site's `llms.txt`, and Shopify's default points both paths at the same content. Our `llms-full-txt` check does not count a file that repeats `llms.txt` or redirects to the home page ([how we check it](https://goodforbots.com/standards#llms-full-txt)). ## Does anything read the file? Adoption is not use. Google says that "Google Search itself doesn't use them" and that an llms.txt "will neither harm nor help your site's visibility or rankings in Google Search". Chrome's Lighthouse has an llms.txt audit, but marks a missing file as not applicable, "as providing the file is optional at the moment." The crawler documentation of OpenAI, Anthropic and Perplexity says nothing about reading it. Ahrefs looked at requests: of the roughly 38,000 files in its population, "97% saw no requests for it whatsoever in May." Other uses need no crawler. Anthropic's engineering blog suggests giving Claude Code the documentation a tool depends on, noting that "LLM-friendly documentation can commonly be found in flat llms.txt files on official documentation sites". What to put in one is in [our llms.txt guide](https://goodforbots.com/guides/llms-txt). Publishing one does not guarantee that an assistant reads your site or cites it. ## Which llms.txt adoption figure to quote Quote a figure with its population and its date, never alone. For large sites, the most careful recent measurement is Rankability's: in September 2026, 8.3% of the Tranco top 10,000 domains served a confirmed `llms.txt`, counting every domain in the list. For the web as a whole there is no good figure yet. Common Crawl's 11.72% is the only random sample, but only of hosts it can fetch, and the Almanac's 2% is from July 2025, with a test that rejects `text/markdown`. Leave out any number that cannot say which sites it measured, and any count of files without a denominator. We re-read these sources every quarter and update this post; the date at the top says when it last changed. ## Our directory, counted the same way Our scanner requests `/llms.txt` at the root with GoodForBotsBot and reads the body. A web page, JSON, an empty body or text with neither an H1 nor a link is not an llms.txt, whatever the status. A bot challenge or a `robots.txt` rule against the path counts as neither having the file nor lacking it. The figure below counts the directory's listings from their latest scans and changes as sites are scanned. The directory is not a sample of the web. What the figure does show is how far the count moves when the definition changes, on the same sites. **Live from the Good for Bots directory: what `/llms.txt` returns on 117 listed sites.** Latest scans from Sep 30, 2026 to Oct 5, 2026. Refreshed hourly. | Measure | Listings | | -------------------------- | --------------- | | Publish an llms.txt | 75 of 117 (64%) | | No llms.txt | 42 of 117 (36%) | | Serve `llms-full.txt` | 30 of 117 (26%) | | Link `.md` copies of pages | 5 of 117 (4%) | **One directory, four ways to count** | Definition | Listings | | ------------------------------------------------------------------------------------------------------ | --------------- | | Answer `/llms.txt` with HTTP 200. What a survey reading status codes counts, catch-all pages included. | 81 of 117 (69%) | | Serve an llms.txt file. Markdown with an H1 title or a link: not a web page, JSON or an empty body. | 75 of 117 (64%) | | Pass our llms.txt check. An H1 title, link entries and enough text to be useful. | 63 of 117 (54%) | | Pass our descriptive llms.txt check. A summary, and descriptions on most of the links. | 37 of 117 (32%) | Counted from each listing's latest scan with the current checks (`llms-txt` v2, `llms-txt-quality` v2 and `llms-full-txt` v3). The directory is not a random sample of the web. It holds sites we chose to scan and sites people submitted, so these figures describe the directory, not the web. [Browse listings with llms.txt](https://goodforbots.com/directory?capability=llms-txt) · [How we check llms.txt](https://goodforbots.com/standards#llms-txt) We look only at the root, so a site that keeps its file under `/docs/` counts as having none here, as it does in every study above. The criteria for each result are on [our standards page](https://goodforbots.com/standards#llms-txt). See what your /llms.txt returns to a crawler [Check a site](https://goodforbots.com/#scan) ## Frequently asked questions ### What percentage of websites have an llms.txt file? It depends on which websites. Crawls of the largest sites between May and September 2026 found a real file at the root of 5.9% to 9.3% of them. Common Crawl found a text file at 11.72% of a random sample of the hosts it can fetch in July 2026. The HTTP Archive measured about 2% of the home pages in its July 2025 crawl, with a test that only checks the status and the Content-Type header. ### Is llms-full.txt part of the llms.txt standard? No. llms.txt is a proposal published at llmstxt.org, not a standard, and it does not mention llms-full.txt. Documentation platforms serve the file; Mintlify says it developed the format with Anthropic. ### Does Google use llms.txt? Not in Search. Google's guide to its generative AI features says Google Search "doesn't use them" and that an llms.txt "will neither harm nor help your site's visibility or rankings in Google Search". Chrome's Lighthouse audits the file but treats a missing one as not applicable. **Sources** 1. [The /llms.txt file, v2](https://llmstxt.org/) — Jeremy Howard, llmstxt.org 2. [LLMS.txt adoption research report](https://www.rankability.com/blog/llms-txt-adoption/) — Rankability 3. [llms.txt: We Scanned the Web's Top 100,000 Domains](https://seomator.com/blog/llms-txt-adoption) — SEOmator 4. [The State of llms.txt: What the Top 10,000 Sites Actually Do](https://chrishumphrey.ai/research/state-of-llms-txt-2026) — Chris Humphrey 5. [The Rise of llms.txt: How Websites Are Signaling to AI](https://thunderbit.com/blog/llms-txt-website-trend) — Thunderbit 6. [Signaling the Shift to Generative Engine Optimization (GEO)](https://www.progeo.ai/research/generative-engine-optimization-signals-q1-2026) — ProGEO.ai 7. [Does anyone actually have an llms.txt? I checked millions of websites](https://caseyrb.com/blog/state-of-llms-txt-adoption/) — Casey Burridge 8. [Web Almanac 2025: Generative AI](https://almanac.httparchive.org/en/2025/generative-ai) — HTTP Archive 9. [Web Almanac 2025: SEO](https://almanac.httparchive.org/en/2025/seo) — HTTP Archive 10. [llms\_txt\_validation custom metric](https://github.com/HTTPArchive/custom-metrics/blob/main/dist/llms_txt_validation.js) — HTTP Archive 11. [A Content Analysis of llms.txt Files from the July 2026 Crawl Archive](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive) — Common Crawl 12. [We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read](https://ahrefs.com/blog/llmstxt-study/) — Ahrefs 13. [Does LLMs.txt impact your AI visibility and citations?](https://seranking.com/blog/llms-txt/) — SE Ranking 14. [llms.txt Adoption: Who Uses It?](https://llmtxt.info/llms-txt-adoption/) — llmtxt.info 15. [llms.txt Adoption: 22 of 78 Big Sites](https://www.serpprism.com/guides/llms-txt-in-the-wild) — SerpPrism 16. [llms.txt for AI Search Report](https://allmo.ai/articles/llms-txt) — ALLMO.ai 17. [LLMS Text](https://builtwith.com/LLMS-Text) — BuiltWith 18. [Websites using llms.txt](https://www.nerdydata.com/reports/llms.txt/b012b7e8-c50d-45e3-8719-0d72f097c3db) — NerdyData 19. [LLMs.txt Tracking Study and Live Dashboard](https://originality.ai/blog/llms-txt-tracking-study) — Originality.ai 20. [llms.txt: how useful is the file for LLM optimisation?](https://www.sistrix.com/ask-sistrix/ai-basics/llms-txt-how-useful-is-the-file-for-llm-optimisation/) — SISTRIX 21. [Debunking LLMs.txt Myths](https://www.wix.com/studio/ai-search-lab/llms-txt-myths) — Wix Studio AI Search Lab 22. [Crawling a Million Websites in Search of LLMs.txt](https://www.chris-green.net/post/million-websites-in-search-of-llms-txt) — Chris Green 23. [Understanding your site's llms.txt file](https://support.wix.com/en/article/understanding-your-sites-llmstxt-file) — Wix 24. [Customize /llms.txt, /llms-full.txt and /agents.md](https://shopify.dev/changelog/customize-llmstxt-llms-fulltxt-and-agentsmd) — Shopify 25. [What is llms.txt?](https://www.mintlify.com/blog/what-is-llms-txt) — Mintlify 26. [Writing effective tools for agents — with agents](https://www.anthropic.com/engineering/writing-tools-for-agents) — Anthropic 27. [Google's guide to optimizing for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) — Google 28. [llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) — Chrome for Developers ## About the author **Mateusz Pawlica**, Founder of Good for Bots. Mateusz has been building digital products since 2014 — games and mobile apps first, then websites, directories and AI automations for businesses. He runs [Pawlica Web & AI](https://pawlicaweb.pl) in Tychy, Poland. Before Good for Bots he built two directories of his own, SaaS Cubes and Mapa Oświatowa, where llms.txt files, Markdown versions of pages and structured data are part of the everyday work rather than a checklist. [LinkedIn](https://pl.linkedin.com/in/mateusz-pawlica-65238816b) · [pawlicaweb.pl](https://pawlicaweb.pl) ## Structured data ```json { "@context": "https://schema.org", "@graph": [ { "@type": "WebPage", "@id": "https://goodforbots.com/blog/llms-txt-adoption-statistics#webpage", "url": "https://goodforbots.com/blog/llms-txt-adoption-statistics", "name": "llms.txt adoption statistics: why the studies disagree", "description": "Published llms.txt adoption figures run from under 0.005% to 57%. What each study counted, why they disagree, and which figure to quote for which sites.", "inLanguage": "en", "isPartOf": { "@id": "https://goodforbots.com/#website" }, "breadcrumb": { "@id": "https://goodforbots.com/blog/llms-txt-adoption-statistics#breadcrumb" }, "mainEntity": { "@id": "https://goodforbots.com/blog/llms-txt-adoption-statistics#article" }, "datePublished": "2026-09-29", "dateModified": "2026-09-29" }, { "@type": "BreadcrumbList", "@id": "https://goodforbots.com/blog/llms-txt-adoption-statistics#breadcrumb", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://goodforbots.com/" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://goodforbots.com/blog" }, { "@type": "ListItem", "position": 3, "name": "llms.txt adoption statistics: why the studies disagree", "item": "https://goodforbots.com/blog/llms-txt-adoption-statistics" } ] }, { "@type": "BlogPosting", "@id": "https://goodforbots.com/blog/llms-txt-adoption-statistics#article", "url": "https://goodforbots.com/blog/llms-txt-adoption-statistics", "mainEntityOfPage": { "@id": "https://goodforbots.com/blog/llms-txt-adoption-statistics#webpage" }, "headline": "llms.txt adoption statistics: why the studies disagree", "description": "Published llms.txt adoption figures run from under 0.005% to 57%. What each study counted, why they disagree, and which figure to quote for which sites.", "image": { "@type": "ImageObject", "url": "https://goodforbots.com/images/blog/llms-txt-adoption-statistics.webp", "width": { "@type": "QuantitativeValue", "value": 1280, "unitCode": "E37", "unitText": "px" }, "height": { "@type": "QuantitativeValue", "value": 720, "unitCode": "E37", "unitText": "px" }, "caption": "An ink drawing of three small robots on a table measuring one stack of index cards: one holds a ruler against it, a clay-coloured robot closes calipers on it, and the third lowers a plumb line beside it." }, "datePublished": "2026-09-29", "dateModified": "2026-09-29", "author": { "@id": "https://goodforbots.com/#person-mateusz-pawlica" }, "publisher": { "@id": "https://goodforbots.com/#organization" }, "isPartOf": { "@id": "https://goodforbots.com/blog#blog" }, "articleSection": "Standards", "inLanguage": "en", "wordCount": 2490, "timeRequired": "PT12M" }, { "@type": "Person", "@id": "https://goodforbots.com/#person-mateusz-pawlica", "name": "Mateusz Pawlica", "givenName": "Mateusz", "familyName": "Pawlica", "jobTitle": "Founder of Good for Bots", "description": "Web developer who builds directories and AI-powered products, and the founder of Good for Bots.", "url": "https://goodforbots.com/about#founder", "sameAs": [ "https://pl.linkedin.com/in/mateusz-pawlica-65238816b", "https://pawlicaweb.pl/o-mnie" ], "knowsAbout": [ "llms.txt", "robots.txt", "AI crawlers", "Markdown content negotiation", "Structured data", "Technical SEO", "Web directories", "Next.js", "TypeScript", "Generative AI" ], "worksFor": { "@id": "https://goodforbots.com/#organization" } }, { "@type": "Blog", "@id": "https://goodforbots.com/blog#blog", "url": "https://goodforbots.com/blog", "name": "AI visibility & web standards blog", "description": "Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.", "inLanguage": "en", "publisher": { "@id": "https://goodforbots.com/#organization" }, "isPartOf": { "@id": "https://goodforbots.com/#website" } }, { "@type": "Organization", "@id": "https://goodforbots.com/#organization", "name": "Good for Bots", "url": "https://goodforbots.com", "logo": "https://goodforbots.com/icon-512.png", "founder": { "@id": "https://goodforbots.com/#person-mateusz-pawlica" }, "contactPoint": { "@type": "ContactPoint", "contactType": "customer support", "email": "support@goodforbots.com", "url": "https://goodforbots.com/contact" } }, { "@type": "WebSite", "@id": "https://goodforbots.com/#website", "name": "Good for Bots", "url": "https://goodforbots.com", "description": "Check if AI crawlers can access and read your website, free. Get a score out of 100 with every finding and fix prompts, then browse reports in the directory.", "inLanguage": "en", "publisher": { "@id": "https://goodforbots.com/#organization" }, "potentialAction": { "@type": "SearchAction", "target": { "@type": "EntryPoint", "urlTemplate": "https://goodforbots.com/directory?q={q}" }, "query-input": "required maxlength=120 name=q" } } ] } ```