AI Crawlers
Which AI crawlers to block in robots.txt, and which to keep
Blocking GPTBot or ClaudeBot refuses model training and leaves AI search alone. Blocking OAI-SearchBot or Claude-User keeps you out of their answers.
· 10 min read
AI visibility & web standards blog
Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.
The bots themselves: who runs them and how they behave.