Fix every check that's costing you AI search traffic
Crawlable scores your site against 9 weighted checks. Each guide below shows exactly what to change, with copy-paste-ready code.
- 25AI crawler access
If GPTBot or ClaudeBot is blocked in robots.txt, your site is invisible to ChatGPT and Claude.
- 25Server-side content
Most AI crawlers don't run JavaScript. If your site renders client-side only, they see a blank page.
- 10llms.txt
llms.txt tells AI crawlers which pages on your site are most important — in 30 seconds of reading.
- 10Sitemap
A valid sitemap.xml at /sitemap.xml lists every page worth crawling, with last-modified dates.
- 8Structured data (JSON-LD)
JSON-LD tells crawlers what type of content the page is (article, product, FAQ) in machine-readable form.
- 7Heading structure
One H1 per page, then H2s for sections, then H3s under them. No skipped levels.
- 8Title & Open Graph
Every route gets its own title, description, og:title, og:description, og:url. Never reuse the homepage's tags.
- 4Canonical URL
A canonical link tells crawlers which URL is the authoritative one — preventing duplicate-content penalties.
- 3llms.txt format
llms.txt must start with H1 (site name), then a blockquote summary, then H2 sections of bullet links.
Don't know which one to fix first?
Run a free Crawlable scan — we'll rank your fixes by impact.
Scan my site free