What Is llms.txt? A Practical Guide to the Anthropic-Proposed Standard for AI Search

There is a new file that has been quietly showing up at the root of tech-forward websites for the last twelve months. It sits next to robots.txt and sitemap.xml, it is written in Markdown, and if you check the sites of Anthropic, Vercel, Cloudflare, and a growing pile of SaaS products, you will find it at /llms.txt.

The proposal is Anthropic's. The format is simple. The question every marketer, engineer, and site owner is asking is the same: does this actually do anything, or is it another spec that will die in draft?

Here is what llms.txt is, what it looks like, whether anyone reads it today, and how to decide if your business should ship one.

What is llms.txt?

llms.txt is a curated Markdown file served at the root of a website (https://example.com/llms.txt) that describes what the site is about and lists the pages an AI model should read first. It is a proposed standard, modeled after robots.txt and sitemap.xml, but designed for large language models instead of search-engine crawlers.

Where robots.txt tells crawlers what they cannot read, and sitemap.xml gives crawlers a machine-readable list of every URL on the site, llms.txt does something different: it hands a language model a curated, prioritized Markdown map of the pages that best represent the site. It is a table of contents written for a machine that is going to synthesize an answer, not for a bot that is going to index a database.

The idea is that a modern LLM does not need every URL — it needs the right ones, described in a format it can parse cleanly. Markdown is the format nearly every LLM is trained on. A file that says "here are the twenty pages that matter, here is what each one covers" is dramatically more useful to a model that is answering a buyer's question than a sitemap with 4,000 entries.

Who proposed llms.txt and why?

llms.txt was proposed by Jeremy Howard in 2024, popularized by Anthropic's public adoption of the format on anthropic.com, and has since been shipped by Vercel, Cloudflare, and hundreds of SaaS documentation sites. The proposal exists at llmstxt.org, and while it is not a W3C standard, it is picking up mass adoption the same way robots.txt did in the mid-1990s — through voluntary, self-organizing publisher agreement rather than committee ratification.

The motivation is straightforward. LLMs pull context from web pages, but they hit two problems: rendering (a lot of modern sites are JavaScript-heavy and hard for a text-focused model to parse cleanly), and prioritization (there is no way for a site owner to say "start here, this page is the canonical description of what we do"). llms.txt solves both. Markdown renders identically for every model. And the site owner explicitly declares the priority order.

Anthropic's version, at anthropic.com/llms.txt, is the reference implementation. Read it if you want to see the pattern applied by the people who wrote the spec.

What does an llms.txt file look like?

An llms.txt file has three required elements — an H1 with the site name, a blockquote with a one-paragraph description, and Markdown sections with linked bullets — followed by whatever additional sections you want to prioritize. Here is a real example, taken directly from the file I ship at corristonconsulting.com/llms.txt:

# Corriston Consulting

> Independent marketing consulting for growth teams, agencies, and
> operators. Twenty-plus years running paid media, SEO, marketing
> operations, and demand-gen infrastructure at private companies and
> agencies I owned.

## About and services

- [About Gary Corriston](https://www.corristonconsulting.com/about): 20+ years of demand generation, paid media, SEO, and marketing operations experience — the operator behind the practice.
- [Services](https://www.corristonconsulting.com/services): Full inventory of consulting engagements.
- [Contact](https://www.corristonconsulting.com/contact): Start a conversation or book an intro call.

## Products and tools

- [SEO / GEO / AEO Checker](https://www.corristonconsulting.com/products/seo-geo-aeo-checker): Free tool that scores any page across three search stacks.

## Blog

- [Blog](https://www.corristonconsulting.com/blog): Full index of published articles.

The pattern to notice: the H1 is the business name (no marketing copy), the blockquote is a compact one-paragraph description that a model can quote verbatim, and each linked bullet ends with a colon-separated one-line summary of what the reader will find at that URL. That summary is what a model uses to decide whether to fetch the page for a specific query.

Does anyone actually read llms.txt today?

The honest answer is some models, sometimes. There is no public transparency into whether OpenAI, Google, or Meta consume /llms.txt at training or inference time. There is observable signal that Anthropic's own crawlers fetch the file, that Claude's documentation-lookup mode uses llms.txt when it exists to prioritize which pages to read, and that a handful of AI-first developer tools (Cursor, Cody, and others) fetch it explicitly when a user references a documentation site.

Here is what we can say with confidence in mid-2026:

Consumer	Reads llms.txt?	Evidence
Anthropic crawlers	Yes	ClaudeBot server logs on adopting sites, plus Anthropic's own documentation lookup patterns
Perplexity	Unknown, likely partial	No public statement; some observed prioritization of listed URLs
OpenAI (ChatGPT / GPTBot)	Unknown	No public statement; no clear observed usage
Google (Google-Extended, AI Overviews)	Unknown	No public statement; Google has its own crawler infrastructure
Developer tool crawlers (Cursor, Cody, Continue)	Yes, explicitly	Documentation lookup features fetch /llms.txt when present

The correct read: this is an emerging standard, adoption is compounding, and the cost of shipping one is under an hour. The downside is a Markdown file at the root of your site. The upside is being ahead of the standard when a major model starts weighting it explicitly.

Do I need an llms.txt file?

If your business depends on AI-generated recommendations, technical documentation, or getting cited when a buyer asks ChatGPT or Claude a question in your category — yes. If you are running a purely brochureware site with no product depth and no editorial content, the ROI is closer to nothing.

The businesses that should ship llms.txt now, in rough order of ROI:

• Documentation sites and developer tools. The strongest current signal. Developer LLM tools already fetch llms.txt, and shipping one measurably improves how your docs get pulled into AI answers.
• B2B SaaS and consulting practices. Buyers ask ChatGPT for recommendations. A curated map that declares what you do and which pages are canonical shortens the model's path to naming you.
• Media, editorial, and long-form content sites. llms.txt lets you flag your best editorial for citation instead of letting a model guess which of your 4,000 posts to read.
• Local service businesses. Lower priority, but shipping one costs nothing and gives you an entity-clarity signal that generalized crawlers can use.

You can skip it if your site is a single-page landing site with no editorial content, or if you have decided to opt out of AI training entirely (in which case you should be using robots.txt and noai meta tags, not llms.txt, which is an invite to be read).

How do I write my own llms.txt file?

The fastest path is fifteen minutes with a text editor. Open a new file, save it as llms.txt, and follow the four-step pattern below. Then ship it to the root of your web server, publicly accessible at yourdomain.com/llms.txt.

1. Start with an H1 that names your business. No marketing copy. Just # Your Business Name. This is the anchor that models use to identify the entity the file describes.
2. Add a blockquote with your one-paragraph description. Use the > Markdown syntax. Write it the way you would want an LLM to quote you when someone asks "what does your company do?" — clear, specific, one to three sentences.
3. Add Markdown sections with linked bullets. Section headings become the topic clusters (About, Services, Products, Blog, etc.). Each bullet is a link to a canonical page, followed by a colon and a one-line summary of what the page covers.
4. Prioritize ruthlessly. Twenty to forty links is the sweet spot. Do not dump your sitemap in. Include only the pages that best represent your business and would answer the questions your buyers ask.

Once you ship it, test it with the SEO / GEO / AEO Checker. The tool fetches /llms.txt on the domain you enter and validates it against the format spec — H1 present, blockquote present, at least three linked bullets in valid Markdown. If any of those are missing, the tool tells you exactly what to fix.

What is the difference between llms.txt and sitemap.xml?

They serve completely different consumers with completely different intents. Sitemap.xml is a machine-readable list of every URL on your site, meant for search-engine crawlers that will index each page. llms.txt is a curated, prioritized Markdown map of your best pages, meant for language models that will synthesize an answer.

Here is the practical breakdown:

Attribute	sitemap.xml	llms.txt
Format	XML	Markdown
Audience	Search engine crawlers	Large language models
Content	Every URL on the site	20–40 curated links
Purpose	Complete index for ranking	Prioritized map for citation
Descriptions	None (URL + lastmod only)	One-line summary per bullet
Standard body	W3C / Sitemaps.org	Community proposal (llmstxt.org)
Adoption	Universal	Growing (Anthropic, Vercel, etc.)

They are complementary, not alternatives. Ship both. A serious site in 2026 has both a sitemap for search engines and an llms.txt for language models.

What are the most common mistakes when writing llms.txt?

The four most common mistakes I see when auditing sites that have shipped llms.txt: missing the required blockquote, using noun-phrase section headings instead of clear topic clusters, dumping every URL instead of curating, and forgetting the one-line summary on each bullet.

The blockquote is the single most-missed element. The spec requires an H1 followed by a > blockquote with a compact description. Many implementations skip it, which means a model has no clean one-paragraph description to quote. That is a citation opportunity thrown away.

The other three mistakes are all versions of the same problem: writing llms.txt as if it were a sitemap. It is not. It is an editorial file. Curate ruthlessly, describe every link, and organize by topic clusters that map to how your buyers actually think about your business.

Test your llms.txt with the free checker

I built a free tool that fetches any URL, parses the HTML, and scores three pillars — SEO, GEO, and AEO — plus emerging signals including llms.txt validation. Paste your homepage. The tool will fetch /llms.txt on your domain and tell you whether it exists, whether it has the required H1 and blockquote, and where the format is failing spec.

Run the free SEO / GEO / AEO checker →

It takes ten seconds per page. No login, no email. If your llms.txt is missing or malformed, the tool tells you exactly what to fix. If you need help writing one for a complex site with lots of editorial or product pages, reach out — I write llms.txt files as part of most GEO engagements.

The bottom line on llms.txt in 2026

llms.txt is a low-cost, high-optionality bet. The file takes under an hour to write. It is served for free from your existing web server. The downside if no model ever reads it is negligible. The upside if models start weighting it explicitly — which is the direction adoption is moving — is entity clarity that competitors without one will not have.

If you have a site that depends on being found by AI models, ship one. If you have documentation or long-form editorial content, ship one urgently. If you are a small local business, ship one when you next touch the site. In every case, the trend line points toward this being a table-stakes signal within two years, and the cost of being early is negligible.