Skip to main content
All guidesScan my site

Add an llms.txt file so AI crawlers know what to read

llms.txt tells AI crawlers which pages on your site are most important — in 30 seconds of reading.

Worth up to 10 points

Why it matters

The llms.txt spec (proposed by Jeremy Howard) gives LLMs a hand-curated map of your site in Markdown. Unlike sitemap.xml (which lists every URL), llms.txt lists the URLs you want AI models to prioritize, with short descriptions. Anthropic, Mistral, and several agent frameworks already read it.

How to fix it

  1. 1. Generate one with Crawlable

    Run a scan, then click "Generate llms.txt" on the report. We pull your sitemap, write a one-line summary per URL with AI, and emit spec-compliant Markdown.

  2. 2. Or write it by hand

    Create `/llms.txt` at your site root. The spec calls for: H1 (site name), blockquote (one-line description), then H2 sections grouping `[Title](url): description` bullets.

    # Acme

> Acme builds invoicing software for freelancers.

## Docs

- [Getting started](https://acme.com/docs/start): Install Acme and send your first invoice in 60 seconds.
- [API reference](https://acme.com/docs/api): REST endpoints with curl examples.

## Product

- [Pricing](https://acme.com/pricing): Free for 5 invoices/month, $19/mo unlimited.

  3. 3. Deploy it at the root

    It must be reachable at `https://yourdomain.com/llms.txt` and return `Content-Type: text/plain` or `text/markdown` with HTTP 200.

FAQ

Is llms.txt an official standard?
It's a community proposal, not a W3C spec — but it's been adopted by Anthropic's docs, Mistral, Vercel, and many SaaS companies. Worth shipping; takes 10 minutes.
Do I also need llms-full.txt?
Optional. llms-full.txt is the full text of every page inlined as Markdown. It's a heavier lift; ship llms.txt first.

See your own score

Run a free Crawlable scan to find every check that needs fixing on your site — not just this one.

Check my site

More guides