
Methodology
Why a scanner can't show you the page GPTBot gets
A copied User-Agent does not get you GPTBot's page. How firewalls, hosts, robots.txt, location and JavaScript make a scanner's view differ, and how to check.
· 11 min read
Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.

Methodology
A copied User-Agent does not get you GPTBot's page. How firewalls, hosts, robots.txt, location and JavaScript make a scanner's view differ, and how to check.
· 11 min read

Standards
Published llms.txt adoption figures run from under 0.005% to 57%. What each study counted, why they disagree, and which figure to quote for which sites.
· 13 min read

AI Crawlers
Blocking GPTBot or ClaudeBot refuses model training and leaves AI search alone. Blocking OAI-SearchBot or Claude-User keeps you out of their answers.
· 11 min read