Crawler access
How to write robots.txt rules for AI crawlers
Write robots.txt rules that treat AI training, search and user-triggered crawlers separately, with tested examples, the current token list and common fixes.
Reviewed
Practical guides for AI crawlers and agents
Step-by-step guides to making your website readable for AI crawlers and agents, with examples tested against our scanner.
Each guide covers one topic our scanner checks: a complete example that passes those checks, common mistakes with their fixes, and the primary sources it was reviewed against. The current rules for every check are on Standards and scoring; dated news and measurements go to the blog.
Page content
How to serve content AI crawlers can read without JavaScript
Which AI crawlers run JavaScript, how to fix empty app shells in Next.js, Nuxt, SvelteKit or Vite, and how to test the raw HTML response yourself.
Reviewed
Page content
How to serve Markdown to AI agents with content negotiation
Return Markdown from the same URL when an AI agent sends Accept: text/markdown, with tested responses, Vary and CDN caching, and nginx and Next.js setups.
Reviewed
Discovery files
How to write a useful llms.txt
Write an llms.txt that tells AI agents what your site covers: a complete tested example, the format explained, publishing checks and fixes for common mistakes.
Reviewed