llms.txt is a proposed web standard: a markdown file at your site's root that gives large language models a curated summary of your most important content. Proposed by Jeremy Howard of Answer.AI in September 2024, it costs almost nothing to add — but no major AI provider has confirmed actually using it yet.
llms.txt is a proposed convention for making websites legible to large language models. The idea: publish a single markdown file at your domain root — /llms.txt — that states what your site is, summarizes it in a sentence or two, and links to your most important pages with brief descriptions. An LLM (or an AI agent acting for a user) can fetch that one file and get an accurate, token-efficient map of your site instead of parsing navigation menus, scripts, and boilerplate across dozens of HTML pages.
The proposal came from Jeremy Howard of Answer.AI in September 2024, with the specification maintained at llmstxt.org. Howard's core argument was practical: context windows are finite, HTML is noisy, and a curated markdown summary is the cheapest way to help a model use your site correctly. It slots into the broader answer engine optimization conversation as one of several attempts to make sites machine-readable by design.
The format is deliberately simple markdown with a loose structure:
Some sites also publish llms-full.txt, a companion file containing the complete text of key documentation in one markdown document, so an agent can ingest everything in a single request. Nothing enforces any of this — it works only if a consuming system chooses to fetch and honor the file, which is exactly where the debate begins.
The three root files are often confused because they live in the same place and all talk to machines. Their jobs are different:
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Tells crawlers what they may not access | Lists URLs that exist for indexing | Explains what your site is and which pages matter |
| Audience | Crawlers (Googlebot, GPTBot, PerplexityBot) | Search engine indexers | LLMs and AI agents at inference time |
| Format | Plain text directives | XML | Markdown prose and links |
| Carries meaning? | No — permissions only | No — inventory only | Yes — curated summary and context |
| Status | Universally honored convention | Universally supported standard | Proposal with unconfirmed consumption |
Put simply: robots.txt is permissions, sitemap.xml is inventory, llms.txt is explanation. They complement rather than replace one another.
Here is the honest picture. Adoption has two very different sides:
Publishing is real. A meaningful number of sites — especially developer-tool and documentation-heavy companies — now serve llms.txt files, and popular docs platforms can generate them automatically. Publishing momentum is genuine and growing.
Consumption is unproven. No major AI provider — OpenAI, Anthropic, Google, or Perplexity — has publicly committed to fetching llms.txt in production. Google's John Mueller has said that no major AI system currently uses the file, comparing it to the old keywords meta tag — a self-declared signal engines learned to ignore. Independent log-file analyses have generally found little AI-bot traffic hitting these files.
So the fair summary is: llms.txt is a low-cost bet on an emerging convention, not a working visibility channel today. That distinction matters when you decide where your team's hours go.
If you decide the hour is worth it, creation is simple. Write it by hand — curation is the point. A minimal example:
# Acme Analytics
> Acme Analytics is a product analytics platform for B2B SaaS
> teams, with self-serve dashboards and a warehouse-native model.
## Docs
- [Quickstart](https://acme.com/docs/quickstart): Install and send
your first event in under ten minutes
- [API Reference](https://acme.com/docs/api): REST and SDK docs
## Guides
- [Pricing](https://acme.com/pricing): Plans, limits, and FAQs
- [Acme vs Amplitude](https://acme.com/compare/amplitude):
Feature and pricing comparison
## Optional
- [Blog](https://acme.com/blog): Product updates and essaysSave it as UTF-8 plain text, serve it at /llms.txt, and revisit it quarterly so links and descriptions stay accurate. A stale llms.txt that misdescribes your product is worse than none.
Our position after watching this play out across client sites: add it, but rank it honestly. In the AEO work we do, llms.txt sits far below the things with demonstrated citation impact — answer-first content structure, crawlability for AI bots, entity consistency, and schema markup for AI search, which (unlike llms.txt) plugs into infrastructure search engines already consume.
The case for adding it anyway: the cost is one hour, the risk is zero, agent traffic that fetches pages on demand is growing, and if a major provider ever announces support, sites with accurate files are positioned on day one. The case against over-investing: unconfirmed consumption means every hour spent polishing llms.txt is an hour not spent on the content restructuring that actually earns citations — the work covered in our AEO content strategy guide. Do it once, keep it current, move on.
No. llms.txt is a community proposal, not a ratified standard. It was introduced by Jeremy Howard of Answer.AI in September 2024 and is maintained as an open specification at llmstxt.org. No standards body has adopted it, and no major AI provider has committed to honoring it. That said, many useful web conventions — including robots.txt originally — started exactly this way.
llms.txt is a curated index: a short markdown file listing your most important pages with one-line descriptions and links. llms-full.txt is the expanded companion some sites also publish, containing the full text of key documentation in a single markdown file so an LLM can ingest everything in one fetch. Most sites only need llms.txt; llms-full.txt is mainly useful for documentation-heavy products.
No. Google's John Mueller has said that no major AI system currently uses llms.txt and has compared it to the old keywords meta tag — a signal sites publish about themselves that engines do not consume. Google's AI features rely on standard crawling and indexing instead. If your goal is Google AI visibility, crawlability, content structure, and schema markup matter; llms.txt does not.
No. llms.txt is a plain text file that search engines neither index against you nor treat as a signal. The realistic downside is not harm but waste: time spent maintaining a file few systems read, and a false sense of progress if it substitutes for real optimization work. Keep it accurate, spend an hour on it, and invest your real effort in content and crawlability.
Start with an H1 containing your site or company name, then a short blockquote summary of what you do. Follow with H2 sections — such as Docs, Guides, and Products — each containing a markdown list of your most important URLs with a one-line description per link. Keep it curated and current: a concise file pointing to your twenty best pages beats an automated dump of your whole sitemap.
The structured data types that actually matter for AEO, with a working JSON-LD example.
How to structure content so AI answer engines can extract, quote, and cite it.
The complete guide to AEO — why it matters and how it differs from SEO.
Our AI Visibility Audit separates the levers that earn citations from the busywork — with a prioritized fix list for your site.