llms.txt explained: a proposed convention, and the honest case for shipping one

What llms.txt is, plainly

llms.txt is a single Markdown file placed at the root of a domain that offers AI tools a short, curated map of your most important pages. It was proposed in September 2024 by Jeremy Howard at Answer.AI, and the spec lives at llmstxt.org, where it is described in plain terms as a proposal to standardise on the file. The word proposed matters. This is a community idea that site owners can choose to follow, not an adopted or official standard. We want to lead with that because most of what gets written about llms.txt skips straight past it and treats the file as a settled best practice. It is not one yet.

Why the idea exists

The reasoning behind the proposal is sound, which is part of why it spread. Language models work inside a limited context window, so they cannot read a whole site at once, and a normal HTML page is full of navigation, ads, and scripts that are awkward for a model to parse cleanly. The proposal's argument is that the site author knows best which pages matter, so the author should curate a short list in a format a model can read without fighting the markup. There is also a companion idea, an expanded file that carries full content rather than just links, for sites that want to ship more than a map. As a piece of reasoning about how models actually consume the web, this holds up well.

The honest adoption picture

Here is where the popular framing falls apart, and where we have to be candid. No major model provider has committed to honoring llms.txt as a production signal. OpenAI's GPTBot respects robots.txt but does not officially use llms.txt. Google manages AI crawling through robots.txt and the Google-Extended token, with no mention of llms.txt support. Anthropic publishes its own llms.txt for its developer docs but points only to robots.txt in its guidance on crawler control. So the common comparison to robots.txt and sitemap.xml is misleading. Those are widely honored protocols. llms.txt is a proposal that the people who run the crawlers have, so far, declined to adopt.

Google has been the most direct. John Mueller of Google's Search team compared the file to the old keywords meta tag, the thing a site owner claims their site is about, and said that as far as he knows none of the AI services have stated they use llms.txt. He added that you can look at your own server logs and see that they do not even check for it. That is about as plain a signal as you get from a search engine on an emerging convention it has chosen not to support.

What the evidence actually shows

The case that publishing llms.txt lifts your AI citations does not survive contact with the data. Ahrefs looked at roughly 137,000 domains with traffic, found that about 28 percent published a valid llms.txt file, and then found that around 97 percent of those files received zero requests in the month they measured. They also found that AI systems never went looking for the file when it did not exist, so the tools fetch it only when something explicitly points them at it, not through any active discovery on their own. Separately, SE Ranking studied roughly 300,000 domains, put adoption at about ten percent, and found no measurable link between having the file and how often a domain gets cited in AI answers. Their model actually performed slightly better when the llms.txt signal was removed. Ahrefs' own recommendation on whether to implement it, as of 2026, was no, not yet.

What a good one actually contains

If you do ship one, follow the real spec rather than the loose descriptions floating around. The only required element is a single H1 with the site or project name. Everything else is optional. You can add a one-line summary as a blockquote, some free-form detail, and then zero or more H2 sections, each holding a list of Markdown links with an optional one-line note after each. One H2 has a defined meaning: a section literally named Optional marks links that a model can skip when it needs a shorter context. It is a Markdown file, not loose plain text, and that formatting is the point, because the structure is what makes it cleanly machine-readable.

Two failure modes are worth naming. The first is dumping every URL from your XML sitemap into the file. The value is in curation, so a list of eight hundred blog posts defeats the purpose. Point at the thirty or so pages a knowledgeable person would. The second is writing the one-line notes in marketing voice. Short, factual descriptions read far better to a model than leading-provider prose.

The honest case for shipping one anyway

None of the above means you should not write one. It means you should write one for the right reasons and with clear eyes about what it does. The file is cheap to produce and harmless to serve. The act of curating your key pages and describing each in one plain sentence is good hygiene regardless of who reads it, and it tends to surface stale or duplicate pages you had forgotten about. There is also a concrete use today that has nothing to do with search citations: developer and agent tooling such as IDE assistants and coding agents will fetch the file when you point them at it, so for documentation-heavy or product sites it can already earn its keep. And if adoption does grow, you have a low-cost hedge already in place.

What we will not tell you is that publishing one will lift your citations in ChatGPT, Perplexity, or Google's AI answers. The current evidence does not support that, and a before-and-after citation test today is unlikely to show any llms.txt-driven effect you could trust. Treat it as cheap, sensible hygiene and a small bet on an emerging convention. That is the honest size of the thing, and it is still worth doing on those terms.

Sources

External sources are provided for verification. NavoTech is not affiliated with and does not endorse the organizations cited.

Written by the team at NavoTech Digital Solutions. Have a project or counter-example? Get in touch.