We crawled 4,000 llms.txt files. Four in five pass Chrome’s new test, and two in five were never files at all.
In August we published a reading of Chrome’s new Agentic Browsing category and noted that nobody had measured how many existing llms.txt files would satisfy its definition. We said we intended to. This is that measurement.
What we found
We took 4,000 domains our audit engine had recorded as having an llms.txt, and fetched each one again. Not the stored result, the file itself, judged today against the two conditions Lighthouse applies: a Markdown document with at least one top-level heading and at least one Markdown link.
Of the 4,000, only 1,775 still served a genuine file. Among those, 1,409 passed, which is 79.4% with a 95% confidence interval of 77.4 to 81.2. So the answer to the question we set is that most existing files already satisfy Chrome’s definition, and the compliance gap is narrower than the coverage of the release implied.
Where files fail, they fail in one particular way. 20.3% contain no Markdown link at all, listing their references as bare URLs. 7.9% have no top-level heading, and 7.6% miss both. A file of the shape - Pricing followed by - URL: https://example.com/pricing reads perfectly well to a person and satisfies neither condition.
The larger finding is about counting, not compliance
The compliance number was the question. It is not the interesting part.
41.8% of the domains returned HTTP 200 at /llms.txt and served an HTML page. Not a file, a web page: a catch-all route, a soft 404, a single-page application answering every path with its shell. A further 13.7% no longer return 200 at all.
That matters because of how adoption gets measured. A checker that treats a 200 response as a file, which is the obvious way to build one and the way ours worked until this month, counts every one of those sites as having published an llms.txt. They have not. The path simply never returns a 404.
So published adoption figures for llms.txt, including our own, are inflated by sites that never had a file. If the rate in this sample holds more broadly, a presence figure of 27.3% describes something closer to 16% in reality. We are stating that as an implication rather than a new headline, because it compares a crawl run today against a measurement taken earlier, and the two are not the same instrument.
The practical version for anyone measuring this: a 200 is not a file. Check that the body is not HTML before counting it.
What this does not show
Nothing here is evidence that any AI engine or agent reads an llms.txt. That was true when we published the pre-registered study, it was true when Chrome shipped the audit, and it is true now. This measures whether files satisfy a published definition, which is a much smaller claim than whether the file does anything.
The sampling frame is also worth stating plainly. These are domains our engine had already recorded as having a file, so this is a study of files that existed, not an estimate of how common llms.txt is across the web. It cannot tell you the denominator. It can tell you what is inside the numerator.
What to do about it
If you publish one, write every reference as a Markdown link and give the file a top-level heading. That is the whole of the test, and four in five files already manage it.
If you measure llms.txt for anyone else, stop trusting the status code. Two in five of the sites you are counting are answering with a web page.
And if you are deciding whether to publish one at all, our position has not moved: the file is worth getting right if you have one, and it is not the thing standing between you and being recommended. Crawlable HTML, working links and structured content still do more.
Method
Sampling frame. 4,000 domains whose most recent audit before scorer version geo-153 recorded an llms.txt as present. That epoch matters: before geo-153 the check meant “a file exists and is over 50 bytes”, which is the population this question is about. Sampling later rows would have over-selected compliant files, because from geo-153 the same field means compliant.
Collection. Each domain fetched once at https://domain/llms.txt on 20 August 2026, with a desktop browser user agent, following redirects, 20 second timeout, response capped at 2MB. 3,322 fetches completed in the run; results were written incrementally.
Classification. A response counts as a file if it returned 200, exceeded 50 bytes, and did not begin with an HTML doctype or html element. Compliance is a reimplementation of the two Lighthouse conditions, written from the specification rather than imported from our scorer, so a defect in one could not silently reproduce in the other. Intervals are Wilson 95%.
Known limits. One fetch per domain, so a transient failure is recorded as a failure. Sites that block automated requests are counted in the 13.7% that did not return 200, and we have not separated blocking from removal. The frame is our corpus, which skews towards small and mid-sized business sites in the sectors our customers operate in.
This measurement was promised in our reading of the Lighthouse Agentic Browsing category and follows the pre-registered study Does llms.txt work?
Questions and answers
What share of llms.txt files pass the Lighthouse audit?
In a crawl of 4,000 domains recorded as having an llms.txt, 1,775 still served a genuine file and 1,409 of those satisfied Lighthouse's definition, which is 79.4% with a 95% confidence interval of 77.4 to 81.2. The dominant failure is a file whose references are written as bare URLs rather than Markdown links.
Why do so many sites appear to have an llms.txt when they do not?
Because a request for /llms.txt on many sites returns HTTP 200 with an HTML page, usually a catch-all route or a soft 404 rather than a missing-file response. In this crawl 41.8% of domains did that. Any measurement that treats a 200 response as a file will count those sites as adopters.
What does Lighthouse check in an llms.txt file?
The contents rather than the presence. It expects a Markdown document containing at least one top-level heading and at least one Markdown link. A reference written as a bare URL does not count, and an absent file is marked Not Applicable because publishing one is optional.
Does passing the audit mean AI engines read the file?
No. Nothing in this crawl or in the audit is evidence that any AI engine or agent reads an llms.txt. It measures whether files satisfy a published definition, which is a smaller claim.
How large is a typical llms.txt?
Among files that genuinely exist the median is about 4KB with a median of five Markdown links. The distribution has a long tail: the largest file in this crawl was just under 2MB and the highest link count was 7,100.