What is llms.txt?
llms.txt is a proposed convention for a text file on a website that gives AI models a short, curated guide to the site: what it is about and links to its most useful pages, written in Markdown. It is not yet a formal standard.
Last updated
In more detail
Web pages are built for people, with menus, scripts and styling that make them harder for a language model to read. The llms.txt proposal, published in September 2024, suggests a simple fix: a Markdown file at the path /llms.txt that names the site, summarises it in a few lines and lists links to its key pages. It is meant to help AI tools use a site when answering a question, and it sits alongside robots.txt and the sitemap, which do different jobs. Creating one takes little effort, and some website platforms generate it automatically. The common mistake is expecting too much. It is a proposal, not an adopted standard, and AI companies are not obliged to read it. Treat it as a low-cost extra, and keep doing the basics: clear pages that crawlers can fetch. Another mistake is confusing it with robots.txt. An llms.txt file offers content; it does not grant or refuse access.
Source
The facts that can change were checked against llmstxt.org (opens in a new tab) in November 2026.
Related terms
- Robots.txtRobots.txt is a plain text file at the root of a website that tells search engine crawlers and other bots which parts of the site they may or may not request.→
- AI crawlerAn AI crawler is a piece of automated software run by an AI company that visits websites and reads their pages, either to gather training material for models or to fetch content for answers in AI search and chat.→
- XML sitemapAn XML sitemap is a file on a website that lists the pages its owner wants search engines to find.→
- Generative engine optimisation (GEO)Generative engine optimisation (GEO) is the practice of making a brand's content easy for AI assistants and AI search features to find, understand and cite when they generate answers.→
More in AI search
Do it with Wonib
Put the words into practice
Wonib learns your brand once, then makes the posts, pictures, copy, decks and documents in it, checked against the brand before you see them. Start free with 1,000 credits, no card needed.
Last updated