
Methodology
Why a scanner can't show you the page GPTBot gets
A copied User-Agent does not get you GPTBot's page. How firewalls, hosts, robots.txt, location and JavaScript make a scanner's view differ, and how to check.
· 11 min read
Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.
How the score is built, and every change to it.

Methodology
A copied User-Agent does not get you GPTBot's page. How firewalls, hosts, robots.txt, location and JavaScript make a scanner's view differ, and how to check.
· 11 min read