Standards
llms.txt adoption statistics: why the studies disagree
Published llms.txt adoption figures run from under 0.005% to 57%. What each study counted, why they disagree, and which figure to quote for which sites.
· 12 min read
AI visibility & web standards blog
Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.
AI Crawlers
Which AI crawlers to block in robots.txt, and which to keep
Blocking GPTBot or ClaudeBot refuses model training and leaves AI search alone. Blocking OAI-SearchBot or Claude-User keeps you out of their answers.
· 10 min read