What is llms.txt, and does your site need one?

Who created llms.txt, and what is it for?

Jeremy Howard published the llms.txt proposal in September 2024. It describes a file that provides brief background information, guidance and links to detailed Markdown files, in a format that both people and language models can read. The file can sit at the site root or at any path, covering the pages beneath it.

The idea behind it is practical. A model working on a user's request has limited context, and a typical web page is mostly navigation, scripts and layout. A short, curated index tells an agent what the site is and where the substance lives.

What should an llms.txt file contain?

The format is ordinary Markdown with a fixed order. Only the first item is required.

• An H1 with the name of the project or site. This is the only required part.
• A blockquote with a short summary containing the key information.
• Optional paragraphs or lists with more detail, but no further headings at this level.
• H2 sections, each containing a list of links with a short note on what each page covers.
• A section titled Optional, by convention, for links an agent can skip when it needs a shorter context.

What is the difference between llms.txt, robots.txt and sitemap.xml?

The three files answer different questions. robots.txt tells automated tools what access is acceptable. A sitemap lists the URLs you want indexed, usually all of them. llms.txt is selective and descriptive: a handful of the pages that best explain the site, with a sentence about each, meant to be read on demand when an agent needs information.

The proposal also suggests offering a clean Markdown version of each important page at the same URL with .md appended. That is more work than the index file, and it only pays off for documentation-heavy sites whose users already work through coding agents.

Does llms.txt improve AI search visibility?

Not as far as anyone has shown. Google's documentation says there are no additional requirements to appear in AI Overviews or AI Mode and that you do not need to create new machine-readable files or AI text files. The crawler documentation from OpenAI, Perplexity and Anthropic linked below describes robots.txt controls and does not list llms.txt as a requirement. We know of no published evidence that adding the file changes whether a page is cited.

So the honest case for llms.txt is narrow. It costs a few minutes, it cannot hurt, and agents and tools that do look for it get an accurate summary in your own words instead of inferring one. dotmap.ai publishes one for that reason. If you have limited time, spend it first on pages that are crawlable, indexed and written to answer a question directly.