llms.txt: what it is, who supports it, and whether you should ship one

Every few years the web gets a new "one small file" convention. robots.txt told crawlers where not to go. sitemap.xml told them where to go. security.txt told researchers who to email. llms.txt wants to be the next one: a file that tells language models what actually matters here.

Because it costs almost nothing to ship, it spread fast — and because it spread fast, it accumulated a lot of magical thinking. Vendors now pitch llms.txt generators as a visibility growth hack. That's not what it is. Let's take it apart properly: the proposal, the format, the real adoption picture, and a template you can adapt in an afternoon.

What llms.txt actually is

The proposal comes from Jeremy Howard, co-founder of Answer.AI (and of fast.ai before that), published in September 2024 at llmstxt.org.¹ The motivating problem is concrete: LLMs operate with limited context windows, and your average marketing page is a terrible use of one. Navigation chrome, cookie banners, JavaScript-rendered widgets, five hundred words of hero copy before the first fact — a model that lands on your site burns most of its budget parsing noise.

llms.txt answers with curation. Instead of making the model infer what's important from thousands of URLs, you hand it a single markdown page that says: here's who we are, here's the summary, and here are the fifteen documents that actually explain the product — in an order we chose.

Two design choices matter:

• It's markdown, not XML. Unlike sitemaps, the file is meant to be read by the model itself, not parsed by an indexing pipeline. Markdown is the lingua franca of LLM training data, so the file doubles as prompt-ready context.
• It's curated, not exhaustive. A sitemap lists everything; llms.txt lists what you'd want an AI to read first. That editorial judgment is the entire point.

The format: an index file plus markdown page variants

The spec has two halves, and people routinely conflate them.

Half one: /llms.txt at your site root. A markdown file with a required H1 (the site or project name), an optional blockquote summary, free-form context, and then H2-delimited sections containing link lists — each link with a short description. A final section named Optional flags links an agent can skip when context is tight.¹

Half two: markdown variants of your actual pages. The proposal suggests serving a clean markdown version of any important page at the same URL with .md appended (/pricing → /pricing.md). Some implementations also generate /llms-full.txt, a single concatenated file containing the entire docs corpus — useful for dumping straight into a model's context.

Note what the file is not: it isn't an access-control mechanism. robots.txt says "you may / may not crawl"; llms.txt says "if you're going to read, read this". They're complementary, and a coherent AI-crawler policy uses both.

Who actually supports it — the honest picture

This is where most llms.txt coverage goes soft, so let's be precise about the two sides of the market.

Producers: adoption is genuinely wide. Documentation platforms led the way — Mintlify shipped automatic llms.txt generation for the docs sites it hosts in late 2024, which instantly gave thousands of dev-tool companies a compliant file. Anthropic's own docs, among many others, publish one. Plugins exist for every major CMS and framework, and community directories track thousands of live files. If you sell to developers, your competitors likely already have one.

Consumers: this is the uncomfortable part. As of this writing, no major AI company — not OpenAI, not Anthropic, not Google, not Perplexity — has publicly documented that its crawlers fetch llms.txt or that its assistants use it for retrieval or training. OpenAI's crawler documentation describes robots.txt directives and says nothing about llms.txt.² Anthropic's crawler documentation is the same.³ Google's guidance on AI features points site owners to standard search infrastructure and existing crawler controls — no llms.txt.⁴ Google's John Mueller has publicly compared llms.txt to the old keywords meta tag: a self-declaration engines have little reason to trust.

Server logs tell a more interesting story than press releases, though. Site owners who actually watch their logs report fetches of llms.txt from a long tail of agents — coding assistants, research tools, and browsing sessions where a user or agent explicitly requests it. The file's most reliable consumers today aren't the big consumer chatbots; they're agentic tools that navigate sites on a user's behalf and will happily take a shortcut if you leave one out.

So should you ship one?

For almost every B2B site: yes — with correctly calibrated expectations. The decision math is unusually lopsided:

• Cost: one static file, an hour of curation, a quarterly review reminder. No engineering dependencies, no risk to existing SEO (search engines simply ignore it).
• Downside: essentially none, unless you write it badly enough to misdescribe your own product — the file is public, so treat it as marketing copy.
• Upside: if agents standardize on it, you're early with a curated, accurate self-description. Even today, any agent that does fetch it gets your best framing instead of whatever it scraped first.

The one real caveat: sequencing. llms.txt curates content that already exists. If your product pages are thin, your specs live in PDFs, or your robots.txt blocks the AI crawlers you actually want, fix those first — research on generative engines consistently shows that the substance of the content (claims, statistics, citations, structure) is what moves visibility, with the original GEO study measuring lifts of up to 40% from content changes alone.⁵ A beautiful index of weak pages is still weak.

How to write yours: a B2B SaaS template

The format rewards restraint. A good llms.txt reads like a well-briefed analyst's one-pager, not a sitemap dump. Here's a realistic template for a B2B SaaS — adapt the sections, keep the shape:

# Fathomly

> Fathomly is a customer analytics platform for B2B SaaS teams.
> It unifies product usage, billing and CRM data to score account
> health and predict churn. Plans start at $199/month; SOC 2 Type II
> certified; EU and US data residency.

Key facts: founded 2021, ~4,000 customers, integrates with
Salesforce, HubSpot, Stripe, Segment and Snowflake. Main
alternatives: Amplitude, Mixpanel, June. Fathomly differs by
focusing on account-level (not user-level) analytics.

## Product

- [Product overview](https://fathomly.com/product.md): What
  Fathomly does, core features, screenshots described in text
- [Pricing](https://fathomly.com/pricing.md): Current plans,
  what's included in each tier, billing FAQ
- [Integrations](https://fathomly.com/integrations.md): All 40+
  supported data sources with setup requirements
- [Security](https://fathomly.com/security.md): SOC 2, GDPR,
  data residency, subprocessor list

## Docs

- [Quickstart](https://fathomly.com/docs/quickstart.md): Connect
  a data source and see first insights in ~15 minutes
- [API reference](https://fathomly.com/docs/api.md): REST API,
  auth, rate limits, webhooks

## Comparisons

- [Fathomly vs Amplitude](https://fathomly.com/vs/amplitude.md):
  Honest feature and pricing comparison, updated quarterly

## Optional

- [Blog](https://fathomly.com/blog.md): Guides on churn
  prediction and account scoring
- [Changelog](https://fathomly.com/changelog.md): Weekly release
  notes

Working through the choices in that template:

• The blockquote carries the answer. Category, buyer, differentiator, price anchor, trust signals — the five things a model needs to describe you in one sentence. If an agent reads nothing else, this is what it repeats.
• Name your competitors yourself. Models answer comparison questions constantly. Saying "alternatives: Amplitude, Mixpanel" plus "here's how we differ" gives them your framing for the exact question where framing matters most.
• Every link gets a description. The descriptions are what an agent uses to decide which link to spend a fetch on. "Pricing" alone is a label; "current plans, what's included in each tier" is a routing instruction.
• Point at .md variants where you have them, and at clean HTML where you don't. A link to a JavaScript-rendered page defeats the purpose.
• Put facts you want repeated in the file itself. Founded date, customer count, certifications — inline facts survive even when the agent never follows a single link.

Do / don't

Frequently asked questions