Page content
How to serve content AI crawlers can read without JavaScript
Which AI crawlers run JavaScript, how to fix empty app shells in Next.js, Nuxt, SvelteKit or Vite, and how to test the raw HTML response yourself.
Reviewed
Practical guides for AI crawlers and agents
Step-by-step guides to making your website readable for AI crawlers and agents, with examples tested against our scanner.
Each guide covers one topic our scanner checks: a complete example that passes those checks, common mistakes with their fixes, and the primary sources it was reviewed against. The current rules for every check are on Standards and scoring; dated news and measurements go to the blog.
Make each page readable without a browser: server-rendered HTML, Markdown versions and structured data.
Page content
How to serve Markdown to AI agents with content negotiation
Return Markdown from the same URL when an AI agent sends Accept: text/markdown, with tested responses, Vary and CDN caching, and nginx and Next.js setups.
Reviewed