llms.txt, shipped honestly: cheap hygiene, not a promise.
llms.txt is an emerging proposal for a curated Markdown map of your key pages, plus a deliberate AI-crawler policy in your robots.txt. Adoption is thin and unconfirmed, and Google has said plainly it does not use the file. We still ship one, because it costs almost nothing, tidies your site, and gets read by the agents people point at you. We will not tell you it earns citations. It does not, yet.
> A proposal, thinly adopted.
> Cheap to keep. No promises.
A discovery file, told straight instead of oversold.
llms.txt is a single Markdown file you place at the root of your domain. Its job is to hand an AI tool a short, curated map of your most important pages, so a model working inside a limited context window does not have to reconstruct your whole site from raw HTML full of navigation, ads, and scripts. The reasoning behind the proposal is sound, which is part of why it spread: the site author knows best which pages matter, so the author curates the short list. That much holds up well.
What does not hold up is the way the file usually gets sold. You will read that llms.txt is the AI-era robots.txt, a settled requirement you cannot afford to skip. It is not. It is a proposal, made in September 2024, that the people who run the major crawlers have so far declined to adopt as a production signal. Google has been the most direct about it, and the measured evidence for citation gains is thin to absent. We would rather lead with that than sell you a comparison that does not survive contact with the data.
So this is the small, honest page of the hub. We write you a good llms.txt by hand from your service taxonomy, and we set an AI-crawler policy in your robots.txt that reflects a decision you actually made. Both are cheap, both are tidy, and neither is a promise. If you want the work that genuinely changes how AI tools read and cite you, that is the structural and schema work on the pages themselves, and it lives on the neighboring pages here.
What this covers
Two small, honest pieces of discovery hygiene, done by hand and explained plainly.
- A hand-authored llms.txt from your real taxonomy
- An optional llms-full.txt for content-heavy sites
- A deliberate AI-crawler policy in robots.txt
- Per-bot allow and disallow, decided with you
- Curation that surfaces stale and duplicate pages
- A straight read on what it does and does not do
What the crawlers do, and what the data shows.
We think you deserve the unflattering version before the pitch, so here it is: the major AI providers have not committed to the file, and the studies do not show a citation lift. Read this first, then decide.
// a proposal the crawler operators have, so far, declined to adopt
Not committed
No major model provider honors llms.txt as a production signal. OpenAI's GPTBot respects robots.txt but does not officially use llms.txt. Google manages AI crawling through robots.txt and the Google-Extended token, with no llms.txt support, and said in June 2026 the file has no effect on Search or AI Overviews. Anthropic publishes its own llms.txt for developer docs but points to robots.txt in its crawler guidance.
Google's John Mueller compared the file to the discredited keywords meta tag and pointed out you can check your own server logs and watch the crawlers never request it. That is about as plain a signal as an emerging convention gets.
Thin and unproven
Ahrefs looked at roughly 137,000 domains, found about 28 percent published a valid llms.txt, and then found around 97 percent of those files got zero requests in the month they measured. AI systems did not go looking when the file was absent, so it is fetched only when something explicitly points at it.
SE Ranking studied roughly 300,000 domains, put adoption near ten percent, and found no measurable link between having the file and citation frequency. Their model performed slightly better with the signal removed. Ahrefs' own answer to whether you should implement it, as of 2026, was no, not yet.
What is included when we ship one.
A good llms.txt follows the real spec, not the loose descriptions that circulate. The only required element is a single H1 with your site name. Everything else is optional structure: a one-line summary, some detail, and H2 sections of curated Markdown links with short factual notes. A section literally named Optional marks links a model can skip for a shorter context.
We write it by hand from your service taxonomy so the pages that matter are named and ordered, and we avoid the two common failure modes: dumping your whole sitemap into it, and writing the notes in marketing voice. For content-heavy sites we can add a longer llms-full.txt carrying the actual content of priority pages in Markdown.
What's included
- A hand-authored llms.txt at /llms.txt
- Curation from your real service taxonomy
- Spec-correct H1, summary, and H2 link sections
- An Optional section for skippable links
- Short, factual notes, not marketing voice
- Optional llms-full.txt for content-heavy sites
- An AI-aware robots.txt with explicit crawler rules
- Per-bot allow and disallow decisions, made with you
- Linked-page reachability and Markdown validation
- Documentation your team can keep current
The AI-crawler policy is the real decision here.
It is worth separating the two files, because they do different things and carry different weight. llms.txt is an unproven proposal about content discovery. Your robots.txt is a widely honored file about access, and the AI-crawler tokens in it are a genuine, deliberate policy choice, not a ranking trick.
The tokens split into three kinds. Training crawlers, like GPTBot, ClaudeBot, and CCBot, fetch content to train models, so disallowing them is how you opt out of future training. Retrieval bots, like OAI-SearchBot, Claude-SearchBot, and PerplexityBot, power search and answers and are the ones that can actually cite you. Training-control tokens, like Google-Extended and Applebot-Extended, make no requests of their own and only govern existing crawlers.
A common stance in 2026 is to allow the retrieval bots that can cite you while opting out of training. That is a reasonable default, not a rule. We will walk you through the tradeoff for your business and set robots.txt to match the position you actually hold, rather than paste in someone else's template.
Two files, two jobs
| llms.txt | robots.txt (AI tokens) | |
|---|---|---|
| Status | Emerging proposal | Widely honored standard |
| Job | Curated content map | Access and crawl policy |
| Honored by crawlers | Not yet, in practice | Yes, broadly |
| The real value | Hygiene and a hedge | A deliberate policy choice |
The approach. We treat the crawler policy as the decision that matters and the llms.txt as low-cost hygiene alongside it. You get a robots.txt that reflects a position you chose, an llms.txt written by hand, and a plain explanation of why each bot is allowed or blocked, so your team can adjust it as your stance changes.
Curate honestly, serve it, keep it cheap.
This is a small engagement on purpose, usually a single pass rather than an ongoing retainer. We curate the file by hand, set a crawler policy you actually decided, validate that the links resolve, and hand it over documented so your team can keep it current without us.
// small, honest, and cheap to maintain, exactly as it should be
- Curate the pages that matterWe work from your real service taxonomy, not your XML sitemap, and pick the thirty or so pages a knowledgeable person would point an AI tool at. The act of curating usually surfaces stale or duplicate pages worth cleaning up regardless of who reads the file.
- Write it to the real specA single H1 with your site name, an optional one-line summary, and H2 sections of Markdown links with short, factual notes. We add an Optional section for skippable links, and for content-heavy sites we can produce a longer llms-full.txt carrying the actual page content in Markdown.
- Decide the crawler policy with youWe lay out the training, retrieval, and training-control tokens and what allowing or blocking each one means for your business, then set robots.txt to match the position you hold. This is the part that is a real decision, so we treat it as one.
- Validate and serveWe check the Markdown is well-formed and every linked page actually resolves, then serve the file at /llms.txt. No broken links, no pages that redirect or 404, nothing that would make the file useless to the agents that do fetch it.
- Document and hand overYou get the files, a plain-language note on why each crawler is allowed or blocked, and instructions for keeping the curation current as your site grows. Because it is cheap to maintain, it stays yours to run rather than a standing invoice.
We wrote the honest, sourced case in full.
If you want the whole argument, with the studies cited and the failure modes named, we published it. It covers what llms.txt actually is, why the idea exists, the adoption picture, what a good one contains, and the clear-eyed case for shipping one anyway. It is the reading we would hand a skeptical client, because we are skeptical too.
Frequently asked questions
Related services
llms.txt is the smallest tablet in the record. The pages that actually move how AI tools read and cite you sit right beside it.
Want an llms.txt and a crawler policy you can trust?
Send us your URL. We will curate a spec-correct llms.txt by hand, lay out the AI-crawler tradeoff so you can set a policy you actually chose, and give you a straight read on what it will and will not do. No promises about citations, because there are none to make.