What is llms.txt?
llms.txt is a Markdown file, placed at your site root, that lists your most important pages for AI tools to read. The proposal, published September 3, 2024 by Jeremy Howard at llmstxt.org, states the whole idea in one line: “We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content.” It is a community proposal, not a standard. Google has said on the record that it does not use the file, and no major AI engine has committed to it. It is optional, harmless, and close to irrelevant today.
What the proposal says
The idea is a table of contents for machines. Context windows are finite, and full HTML pages carry navigation, ads, and scripts that waste them. So the proposal asks sites to publish one clean Markdown file at /llms.txt: an H1 with the site name, a short blockquote summary, then sections of links to your key pages with one-line notes. The spec's own framing is that the file will mainly be useful for inference, meaning the moment a user is asking an AI tool for help, not for training crawls.
As a piece of design, it is tidy and well-intentioned. Nothing in this entry is a knock on the idea. The problem is what happened after the idea: mostly nothing.
Who reads it today
Here is the honest inventory. A handful of AI companies engaged with the proposal, and some developer-tool docs sites publish the file. Anthropic indicated support in late 2024, and Perplexity moved in 2025. That is real but small.
Now the other side of the ledger. Google confirmed in 2025 that it does not use llms.txt and is not planning to. Google's John Mueller compared it to the keywords meta tag, the tag search engines learned to ignore because anyone could stuff it with anything. No major AI engine has committed to reading the file as a matter of policy, and the AI crawlers that dominate server logs are not requesting it in meaningful volume. So the practical answer to "who reads llms.txt today" is: approximately nobody that matters for your traffic.
That is why we do not treat a missing llms.txt as a failure in our audits. A check that fails you for skipping a file no major engine reads would be folklore, and folklore is what we exist to cut.
Not a standard, and not a requirement
Two words get blurred in most coverage of this file, so we will keep them separate. A standard is something implementers agreed to honor, the way robots.txt is honored across every serious crawler. A proposal is a document hoping to become that. llms.txt is the second thing. It sits at llmstxt.org waiting for adoption that, so far, the big engines have declined to give.
That distinction settles the anxiety we hear from owners: no, you are not invisible to ChatGPT because you lack an llms.txt, and no, building one will not move you into Google's AI Overviews. Nothing about your eligibility for AI citations runs through this file. The engines decide what to cite from what they can crawl and read on your actual pages. If you enjoy shipping a clean index and it takes ten minutes, fine. There is no downside. There is also no meaningful upside to chase. Our full breakdown, including the file format and the adoption record, is in What is llms.txt, and do you need it?
What to do instead
The time people budget for llms.txt is better spent on the 3 things that decide whether an AI engine can cite you at all:
- Crawler access. Make sure your robots.txt is not blocking the search and citation bots while trying to fence out training bots. This mistake is common and expensive. Start with how to fix robots.txt blocking AI crawlers.
- Real HTML. If your answer only exists after JavaScript runs, most AI crawlers see an empty shell. Your content has to be in the raw HTML the server sends.
- Quotable passages. Lead with a self-contained answer, include a real number or named entity, and cite your sources. That is what retrieval systems lift.
None of those involve a special file. They are the same fundamentals Google points to for its AI features: crawlable, readable, worth quoting.
See also
llms.txt is a proposed convention. robots.txt, by contrast, is the control file the AI bots actually honor, and its tokens are where your real control lives. See What is GPTBot? for OpenAI's crawlers and What is Google-Extended? for Google's training control token, which draws exactly the standard-versus-folklore line this file blurs.