Crawler access
How to write robots.txt rules for AI crawlers
Write robots.txt rules that treat AI training, search and user-triggered crawlers separately, with tested examples, the current token list and common fixes.
Reviewed
Practical guides for AI crawlers and agents
Step-by-step guides to making your website readable for AI crawlers and agents, with examples tested against our scanner.
Each guide covers one topic our scanner checks: a complete example that passes those checks, common mistakes with their fixes, and the primary sources it was reviewed against. The current rules for every check are on Standards and scoring; dated news and measurements go to the blog.
Decide which crawlers may read your site: robots.txt rules, usage preferences and the firewall settings that can override them.