Glossary
llms.txt
llms.txt is a proposed Markdown file, placed at /llms.txt on a website, that gives AI language models and agents a short, curated guide to the site's most useful content. It is a proposal, not a formal standard, and it does not replace robots.txt or a sitemap.
Jeremy Howard published the proposal in September 2024 and it is documented at llmstxt.org. It is a proposal that is open for community input, not a formal web standard.
The file follows a set structure:
- a heading with the name of the site or project, which is the only required part;
- a short summary;
- optional notes;
- lists of links grouped under headings, each with an optional description;
- an "Optional" section for links an agent can skip when it needs less context.
The proposal also suggests offering clean Markdown versions of the pages an agent is likely to need.
llms.txt does not replace robots.txt or a sitemap. robots.txt states which automated access is acceptable. A sitemap lists every indexable page. llms.txt is a short overview meant to be read at the moment a model is answering a question.
Why it matters to a business website
For most businesses it is a small, optional extra. Google's documentation on AI features states that site owners do not need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode.
The file costs little to add and keep up to date. It is not a substitute for pages that are built correctly, indexed and clearly written.
Related
Stryx looks at the evidence in Does llms.txt matter? and covers the wider work under AI search optimisation. See also generative engine optimisation.