Build an llms.txt from the sitemap you already have
It finds your pages, groups them into sections and lays them out in the conventional structure. The one-line descriptions are left blank, because a description invented from a URL is a confident guess about a page nobody read.
Reads the sitemap you already publish. It does not crawl your pages, and nothing is stored.
Before you spend an afternoon on this
llms.txt is a proposal, not a standard. No major assistant has publicly committed to reading it, and anyone telling you it is how you get cited by ChatGPT is selling something. We would rather say that here than have you find out later.
What makes it worth doing anyway is the price. It is one small file, it cannot break anything, a number of developer-facing tools do consume it, and if the convention does settle you are already there. That is a reasonable bet at twenty minutes and a bad one at two days.
There is also a side effect worth more than the file. Deciding which forty of your pages actually answer something is an exercise most sites have never done, and the answer tends to be uncomfortable. If writing this turns into a cull of thin pages, the cull was the valuable part.
The things that demonstrably decide whether an assistant can quote you are duller and older: can it reach the page, does the page answer the question near the top, and is there anything on it worth repeating. That is what the writing here is about.
Questions
- What is llms.txt?
- A markdown file at the root of a domain that tells an AI assistant what the site is and which pages are worth reading. It was proposed by Jeremy Howard in 2024. The idea is that an assistant working from your pages should not have to reconstruct your site from navigation menus and cookie banners, so you hand it a table of contents instead.
- Do ChatGPT, Claude or Perplexity actually read llms.txt?
- None of them has committed to it publicly, and you should not build a strategy on it. What makes it worth twenty minutes anyway is the cost: it is one small file, it cannot hurt anything, and several developer-facing tools do consume it. Treat it as cheap optionality, not as a ranking factor. The things that demonstrably decide whether you get cited are access, structure and being worth quoting.
- Where does llms.txt go?
- At the root of the domain, served as plain text: https://example.com/llms.txt. The convention is that a longer llms-full.txt may sit beside it with the whole content inlined, for a reader that wants everything in one fetch rather than following links.
- Why are the descriptions blank?
- Because a description generated from a URL is a confident guess about a page nobody read, and a file full of those is worse than no file: it tells an assistant something specific and wrong. The structure is the part a machine can do honestly. One clause per page on what it answers is the part that has to be yours, and it is the part that makes the file worth having.
- Is llms.txt the same as robots.txt?
- No, and they are not alternatives. robots.txt is permission: which crawlers may read you at all. llms.txt is orientation: what an assistant should look at once it is allowed in. A site with a beautiful llms.txt and a robots.txt that disallows OAI-SearchBot has written directions to a locked door.
- How does this build the file without crawling my site?
- It reads the sitemap you already publish, found either from the Sitemap: line in your robots.txt or at /sitemap.xml, and follows one level of sitemap index. That is a few requests to files that exist to be read by machines. It never fetches your pages, which is why the link titles come from your URLs rather than from your headings.
Directions are no use if the door is locked
Check whether your robots.txt actually lets the assistants in before you write them a table of contents.