About the llms.txt Generator
llms.txt is a plain-text file at the root of a site that gives language models and AI agents a curated map of what is there, written in Markdown rather than HTML. It exists because a crawler working from a sitemap and rendered pages spends most of its budget on navigation, markup, and boilerplate rather than content.
The clearest current benefit is not search ranking but agent efficiency. Coding assistants that read documentation directly use these files to locate the right page quickly, and Markdown costs a fraction of the tokens the equivalent rendered HTML would.
It is a proposal rather than a ratified standard, and adoption varies. It costs very little to publish, and the discipline of writing one — deciding which pages actually matter and describing each in a sentence — is worth something regardless of who reads it.
Frequently asked questions
What should go in llms.txt?
A single H1 with the site or product name, a short blockquote summary, then Markdown link lists grouped under H2 headings, each link followed by a one-line description of what the page covers. Keep it curated. A file that lists every page is a sitemap in different clothing and provides none of the prioritisation that makes the format useful.
How is llms.txt different from robots.txt?
robots.txt is about permission — which paths a crawler may fetch. llms.txt is about guidance — which content is worth reading and what each part covers. They are complementary and neither substitutes for the other. Blocking a path in robots.txt while listing it in llms.txt is a contradiction worth avoiding.
Does publishing llms.txt improve search rankings?
There is no evidence that it affects traditional ranking, and it should not be sold as an SEO tactic. The demonstrable benefits are to agents and assistants that read it directly, and to the chance of being cited accurately when a model summarises your area. Treat it as a distribution improvement rather than a ranking one.
Should the file be generated or hand-written?
Generated from the same registries that drive your navigation, so it cannot drift out of date. A hand-maintained list is accurate on the day it is written and quietly wrong within a month. Generation also forces the descriptions to live alongside the content metadata rather than in a separate file nobody updates.
Is there an llms-full.txt as well?
Some sites publish one — a single file containing the full text of the documentation rather than links to it. It suits a product with a compact, stable documentation set that an agent can load in one request. For a large site it produces an unusable file, and the linked form is the better choice.