---
title: "Why a scanner can't show you the page GPTBot gets"
description: "A copied User-Agent does not get you GPTBot's page. How firewalls, hosts, robots.txt, location and JavaScript make a scanner's view differ, and how to check."
url: "https://goodforbots.com/blog/what-gptbot-sees"
image: "https://goodforbots.com/images/blog/what-gptbot-sees.webp"
author: "Mateusz Pawlica"
category: "Methodology"
date: "2026-10-27"
lastFactChecked: "2026-10"
readingMinutes: 11
---

# Why a scanner can't show you the page GPTBot gets

A copied User-Agent does not get you GPTBot's page. How firewalls, hosts, robots.txt, location and JavaScript make a scanner's view differ, and how to check.

A scanner cannot reliably show you the page GPTBot receives, and ours does not try to pass for GPTBot. Firewalls, CDNs and hosts recognise GPTBot by the IP addresses OpenAI publishes, and many give each named crawler its own rules, so a scanner receives the answer meant for the scanner. On the same site, GPTBot can get the same page, a fuller one, or nothing.

The pages split in five places: firewall rules set per bot, hosts that render pages only for crawlers they verify, `robots.txt` groups, the country a request comes from, and JavaScript, which one site owner's logs suggest OpenAI's crawlers began running on some sites in September 2026. A scan sees only its own side of each.

GPTBot is also only one of OpenAI's three crawlers, and it collects training data. `OAI-SearchBot` decides what can appear in ChatGPT search, and `ChatGPT-User` fetches pages a person asks about. Everything below applies to all three, and the last two matter more if you want to be read in answers ([which AI crawlers to block, and which to keep](https://goodforbots.com/blog/which-ai-crawlers-to-block)).

## Why copying GPTBot's User-Agent does not get you its page

OpenAI publishes a User-Agent for each crawler and a list of the IP addresses it crawls from: `gptbot.json`, `searchbot.json` and `chatgpt-user.json`. On 27 October 2026 `gptbot.json` held 18 IPv4 prefixes, with a `creationTime` of 22 September 2026. OpenAI's crawler page describes no other way to verify a request, neither a reverse DNS name nor a signature. The only OpenAI agent documented as signing its requests with Web Bot Auth is ChatGPT's Cloud browser, which sends a `Signature-Agent` header set to `"https://chatgpt.com"`.

Firewalls check that origin. Cloudflare's Verified Bots policy asks a bot to declare who it is "deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS", and Cloudflare Radar lists GPTBot as verified. AWS WAF's Bot Control "uses the IP address from the web request origin to help determine whether a bot is verified". Of self-identifying bots it cannot verify, it says "these types of bot signatures can be falsified, and so are treated as unverified". They often are. In a two-week analysis published in September 2025, HUMAN Security found that "5.7% of all observed traffic presenting an AI crawler or scraper user agent was spoofed". For `ChatGPT-User` the rate was about 16.7%.

A scanner sending GPTBot's User-Agent from its own address is therefore an unverified bot using GPTBot's name. What it receives depends on how the site checks:

- A firewall that verifies origin does not treat it as GPTBot.
- A tool that reads only the name does. On Cloudflare's free plan, AI Crawl Control "identifies AI crawlers based on their user agent strings", and Prerender.io "uses User-Agent matching to decide which requests to pre-render".

The same request can be refused on one site and served GPTBot's page on the next, and nothing in the response says which kind of site answered. Our crawler, [GoodForBotsBot](https://goodforbots.com/bot), sends one User-Agent on every request and never presents itself as another bot.

Google offers a check of this kind for its own crawler. URL Inspection in Search Console runs "a live test: the tool fetches and examines the URL in real time", with "a screenshot of the rendered page as the Google-InspectionTool sees it". Even so, Google warns that "the information shown in the live test can differ from the indexed URL", and the request comes from its testing agent, not from Googlebot. We found no equivalent tool from OpenAI.

## Firewalls and CDNs give each bot its own rules

Cloudflare sorts verified bots by behaviour, and "Search, Agent, and Training are also available as managed presets you can act on across all plans." Its Radar directory, read on 27 October 2026, files OpenAI's crawlers and ours under different behaviours:

| Bot              | Cloudflare category | Access       |
| ---------------- | ------------------- | ------------ |
| `GPTBot`         | Training            | Direct       |
| `OAI-SearchBot`  | Search              | Direct       |
| `ChatGPT-User`   | Agent               | Intermediary |
| `GoodForBotsBot` | Agent               | Intermediary |

A site that blocks Training refuses GPTBot and lets us in. A site that blocks Agent does the opposite. Since 15 September 2026, Cloudflare has offered new domains that show ads a preset with Training on "Disallow AI Training" and Agent on "Block on pages with ads". Domains that had the old "Block AI Bots" toggle on, and had not set the newer per-behaviour controls, were moved to the same two values. Such a site refuses GPTBot throughout: under "Disallow AI Training", "every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI". On pages with ads it also refuses `GoodForBotsBot` and `ChatGPT-User`, while `OAI-SearchBot` gets through on every page.

Other vendors split traffic the same way. AWS WAF's `CategoryAI` rule blocks AI bots "regardless of whether the bots are verified or unverified", while its other categories block only unverified bots. Vercel's bot protection "automatically excludes verified bots", and its AI bots ruleset, which can deny AI crawlers, is "inactive by default".

A scan sees only the answer given to its own crawler. On 27 October 2026 we scanned theviatlas.com. Its `robots.txt` allowed all the search and user-fetch crawlers we track, and its `llms.txt` came back normally, but the homepage answered our signed request with HTTP 403 and Cloudflare's `cf-mitigated: challenge` header ([report](https://goodforbots.com/r/theviatlas.com#check-waf-challenge)). The report charges the [bot challenge](https://goodforbots.com/standards#waf-challenge), because our crawler received it. Whether GPTBot got the same 403 depends on rules a scan cannot see.

## Some hosts render the page only for crawlers they verify

The difference can also run the other way, with GPTBot getting more than a scanner. Lovable apps created before 13 May 2026 (22 June on Enterprise plans) are React and Vite single-page apps: the HTML is an empty shell that JavaScript fills in. Lovable's hosting makes an exception for crawlers it trusts: "When a verified crawler arrives, Lovable renders the page on the fly and returns the resulting HTML." Its verified list includes "AI search crawlers such as those of ChatGPT, Perplexity, Claude, and Gemini", while "unverified bots and link checkers see the regular single-page app, unless they request the page as Markdown". The hosting documentation states the consequence: "Third-party SEO scanner tools are not verified crawlers, so they may report your pages as empty". Lovable does not say how it verifies a crawler. Newer apps render on the server, and there "people, crawlers, and scanners all get the same rendered HTML".

Pre-rendering services do the same by User-Agent. Prerender.io matches the User-Agent against a list, and its documentation warns that "GPTBot, ClaudeBot, PerplexityBot, and others may not appear in older integration versions". A site on an old integration may serve GPTBot the empty shell after all. Google calls this pattern dynamic rendering and describes it as "a workaround and not a recommended solution".

When a scan finds Lovable's empty shell on Lovable's hosting, our report adds an unscored note saying that the host renders pages for verified crawlers only, and the score grades what all other readers get. We never ask for the version made for verified crawlers. [How to serve content AI crawlers can read without JavaScript](https://goodforbots.com/guides/server-rendered-html) covers the fix that gives everyone the same page.

## robots.txt can name each crawler separately

A `robots.txt` file can let one crawler in and keep another out. RFC 9309 says "crawlers set their own name, which is called a product token, to find relevant groups", and a crawler falls back to `User-agent: *` only "if no matching group exists". On 18 October 2026, ezdistro.space let `GoodForBotsBot` in through its `*` group and refused GPTBot in a group of its own, `User-agent: gptbot` followed by `disallow: /` ([report](https://goodforbots.com/r/ezdistro.space#check-ai-crawler-access)). Our scan read the homepage. GPTBot was told to read nothing. [How to write robots.txt rules for AI crawlers](https://goodforbots.com/guides/robots-txt-ai-crawlers) explains how groups are matched.

That refusal costs nothing in the score, because refusing training is the owner's choice. The same file also refused `ChatGPT-User` and six other user-triggered fetchers, which [does cost points](https://goodforbots.com/standards#ai-crawler-access). We measure our own crawler's access and read the others' from the file, and each report says so: "Observed access describes GoodForBotsBot only. Other crawler entries describe declared rules, not tests of those operators' access."

## Where the request comes from

Our scans start from servers in the EU. OpenAI does not say where its crawlers run. We matched its published prefixes against Microsoft's Azure IP ranges and service tags of 19 October 2026: all 18 GPTBot prefixes and all 39 `OAI-SearchBot` prefixes are in US regions.

A site that varies its answer by country therefore serves us and GPTBot different pages. Cloudflare "adds the `CF-IPCountry` header to all requests to your origin server", and consent platforms such as OneTrust choose the banner from "the general location of the user". A homepage that refuses our region with HTTP 451 gets no score from us, because we cannot see what it serves elsewhere. A geo-redirect or a cookie wall served with HTTP 200 looks like an ordinary page, and we grade it as one.

## JavaScript: OpenAI does not say whether its crawlers run it

OpenAI's crawler page does not mention JavaScript. The largest public measurement, Vercel's from December 2024, found that "none of the major AI crawlers currently render JavaScript", OpenAI's three included. ChatGPT's crawlers fetched JavaScript files, 11.50% of their requests, but "they don't execute them".

That may have changed for some sites. On 30 September 2026, a site owner reported on Indie Hackers that from 25 September `OAI-SearchBot` and GPTBot began rendering pages on their Next.js sites. The author checked each request against OpenAI's published IP lists. The evidence was Next.js prefetch requests ending in `?_rsc=`, which, the author notes, appear only when a page's JavaScript runs: on 28 September they made up 97% of GPTBot's requests. The author found no announcement from OpenAI and wrote that "it looks like rendering is only switched on for some sites". It is one operator's logs, and we have not seen it confirmed elsewhere.

The two kinds of scanner err in opposite directions. One that renders the page in a browser shows content that a crawler without JavaScript never receives. One that reads the raw HTML, as ours does, may show less than GPTBot gets wherever OpenAI renders. We grade the raw HTML because every crawler receives it first, and it is the only part all of them can read ([how we score it](https://goodforbots.com/standards#javascript-required)).

## What our score describes

- **One identity.** Every request carries `GoodForBotsBot/1.0 (+https://goodforbots.com/bot)` and a Web Bot Auth signature. Cloudflare lists the bot as verified, in its Agent category.
- **Your rules first.** If `robots.txt` refuses GoodForBotsBot the homepage, the report says "opted out" and shows no score.
- **The bytes the server sent.** When a refused request is retried through a browser, from the US, we keep the HTTP response and never the rendered page. The challenge still counts, because a crawler without a browser met it.
- **Measured and declared, kept apart.** We measure our own access. Other crawlers' access, GPTBot's included, is read from your files.

A score describes what a well-behaved crawler outside OpenAI's network receives, and what your files declare. It does not show what OpenAI stored, and a high score does not guarantee a citation.

## How to check what GPTBot received

Your server and CDN logs are the only record of what OpenAI's crawlers were served.

1. **Keep only the requests that come from OpenAI**

   Filter your access log by the `GPTBot` User-Agent and keep only the addresses listed in `gptbot.json`. Do the same for `OAI-SearchBot` with `searchbot.json`, and for `ChatGPT-User` with `chatgpt-user.json`. A request from outside the list is someone else using the name.
2. **Read the status and the size**

   A 403 or 429, or a response far smaller than the page your visitors get, points at a firewall rule or an empty shell.
3. **Look at the edge as well**

   A request your CDN or firewall refused never reaches your server's log. Look for the same User-Agent in the firewall or security events of the service in front of your site.
4. **Read the rules each crawler is given**

   Check the `robots.txt` group each OpenAI token falls under with the [robots.txt checker](https://goodforbots.com/tools/robots-txt-checker), and ask whether your host or a pre-rendering service sends crawlers a different page from the one browsers get.

This script checks addresses from your log against OpenAI's current list:

`is_gptbot.py`

```python
import ipaddress, json, sys, urllib.request

with urllib.request.urlopen("https://openai.com/gptbot.json") as response:
    ranges = json.load(response)
networks = [ipaddress.ip_network(p["ipv4Prefix"]) for p in ranges["prefixes"]]

for ip in sys.argv[1:]:
    verified = any(ipaddress.ip_address(ip) in network for network in networks)
    print(ip, "GPTBot" if verified else "not GPTBot")
```

`python3 is_gptbot.py 132.196.86.10 203.0.113.7` printed `GPTBot` for the first address and `not GPTBot` for the second on 27 October 2026. The list changes, so the script downloads it each time it runs. Swap the file name for `searchbot.json` or `chatgpt-user.json` to check the other two crawlers.

See what our crawler receives from your site [Check a site](https://goodforbots.com/#scan)

## Frequently asked questions

### Can I test my site as GPTBot by changing my User-Agent?

Not reliably. Firewalls such as Cloudflare and AWS WAF verify GPTBot by OpenAI's published IP addresses, so a copied User-Agent sent from any other address is not treated as GPTBot. Tools that match only the name, such as Prerender.io, do treat it as GPTBot. The response does not tell you which kind of site answered.

### Does GPTBot run JavaScript?

OpenAI does not say. Vercel found in December 2024 that none of OpenAI's crawlers rendered JavaScript. A site owner reported on 30 September 2026 that OAI-SearchBot and GPTBot had begun rendering pages on some sites from 25 September, with no announcement from OpenAI. Content in the first HTML response is read either way.

### My report shows a Cloudflare challenge, but ChatGPT cites my site. How?

Cloudflare files GPTBot under Training, OAI-SearchBot under Search, and ChatGPT-User and GoodForBotsBot under Agent, and a site can allow or block each behaviour separately. The report shows what our crawler received. Your logs show what OpenAI's crawlers received.

**Sources**

1. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI
2. [GPTBot IP addresses (gptbot.json)](https://openai.com/gptbot.json) — OpenAI
3. [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting) — OpenAI
4. [Verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) — Cloudflare
5. [Bots Directory: GPTBot](https://radar.cloudflare.com/bots/directory/gptbot) — Cloudflare Radar
6. [Bots Directory: GoodForBotsBot](https://radar.cloudflare.com/bots/directory/goodforbotsbot) — Cloudflare Radar
7. [Manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) — Cloudflare
8. [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) — Cloudflare
9. [IP geolocation](https://developers.cloudflare.com/network/ip-geolocation/) — Cloudflare
10. [AWS WAF Bot Control rule group](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html) — Amazon Web Services
11. [Bot Management](https://vercel.com/docs/bot-management) — Vercel
12. [AI crawler spoofing: attackers impersonate ChatGPT and Perplexity](https://www.humansecurity.com/learn/blog/ai-crawler-spoofing-chatgpt-mistral-perplexity/) — HUMAN Security
13. [URL Inspection tool](https://support.google.com/webmasters/answer/9012289) — Google
14. [Dynamic rendering as a workaround](https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering) — Google
15. [SEO and AI search](https://docs.lovable.dev/features/seo-aeo) — Lovable
16. [How Lovable hosts your app](https://docs.lovable.dev/features/hosting) — Lovable
17. [How do I add AI crawlers and bots to my integration?](https://docs.prerender.io/docs/how-to-add-additional-bots) — Prerender.io
18. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — IETF
19. [Azure IP Ranges and Service Tags: Public Cloud](https://www.microsoft.com/en-us/download/details.aspx?id=56519) — Microsoft
20. [Custom Geolocation Integration](https://developer.onetrust.com/onetrust/docs/custom-geolocation-integration) — OneTrust
21. [The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler) — Vercel
22. [Traffic from ChatGPT jumped! Is it because its crawlers now run JavaScript?](https://www.indiehackers.com/post/traffic-from-chatgpt-jumped-is-it-because-its-crawlers-now-run-javascript-5dea2e48d0) — Indie Hackers

## About the author

**Mateusz Pawlica**, Founder of Good for Bots. Mateusz has been building digital products since 2014 — games and mobile apps first, then websites, directories and AI automations for businesses. He runs [Pawlica Web & AI](https://pawlicaweb.pl) in Tychy, Poland. Before Good for Bots he built two directories of his own, SaaS Cubes and Mapa Oświatowa, where llms.txt files, Markdown versions of pages and structured data are part of the everyday work rather than a checklist. [LinkedIn](https://pl.linkedin.com/in/mateusz-pawlica-65238816b) · [pawlicaweb.pl](https://pawlicaweb.pl)

## Structured data

```json
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "WebPage",
      "@id": "https://goodforbots.com/blog/what-gptbot-sees#webpage",
      "url": "https://goodforbots.com/blog/what-gptbot-sees",
      "name": "Why a scanner can't show you the page GPTBot gets",
      "description": "A copied User-Agent does not get you GPTBot's page. How firewalls, hosts, robots.txt, location and JavaScript make a scanner's view differ, and how to check.",
      "inLanguage": "en",
      "isPartOf": {
        "@id": "https://goodforbots.com/#website"
      },
      "breadcrumb": {
        "@id": "https://goodforbots.com/blog/what-gptbot-sees#breadcrumb"
      },
      "mainEntity": {
        "@id": "https://goodforbots.com/blog/what-gptbot-sees#article"
      },
      "datePublished": "2026-10-27",
      "dateModified": "2026-10-27"
    },
    {
      "@type": "BreadcrumbList",
      "@id": "https://goodforbots.com/blog/what-gptbot-sees#breadcrumb",
      "itemListElement": [
        {
          "@type": "ListItem",
          "position": 1,
          "name": "Home",
          "item": "https://goodforbots.com/"
        },
        {
          "@type": "ListItem",
          "position": 2,
          "name": "Blog",
          "item": "https://goodforbots.com/blog"
        },
        {
          "@type": "ListItem",
          "position": 3,
          "name": "Why a scanner can't show you the page GPTBot gets",
          "item": "https://goodforbots.com/blog/what-gptbot-sees"
        }
      ]
    },
    {
      "@type": "BlogPosting",
      "@id": "https://goodforbots.com/blog/what-gptbot-sees#article",
      "url": "https://goodforbots.com/blog/what-gptbot-sees",
      "mainEntityOfPage": {
        "@id": "https://goodforbots.com/blog/what-gptbot-sees#webpage"
      },
      "headline": "Why a scanner can't show you the page GPTBot gets",
      "description": "A copied User-Agent does not get you GPTBot's page. How firewalls, hosts, robots.txt, location and JavaScript make a scanner's view differ, and how to check.",
      "image": {
        "@type": "ImageObject",
        "url": "https://goodforbots.com/images/blog/what-gptbot-sees.webp",
        "width": {
          "@type": "QuantitativeValue",
          "value": 1280,
          "unitCode": "E37",
          "unitText": "px"
        },
        "height": {
          "@type": "QuantitativeValue",
          "value": 720,
          "unitCode": "E37",
          "unitText": "px"
        },
        "caption": "An ink drawing of a dark doorway with two hands reaching out of it: one gives a small robot a thick clay-coloured bundle of papers, the other hands a second robot a single blank card."
      },
      "datePublished": "2026-10-27",
      "dateModified": "2026-10-27",
      "author": {
        "@id": "https://goodforbots.com/#person-mateusz-pawlica"
      },
      "publisher": {
        "@id": "https://goodforbots.com/#organization"
      },
      "isPartOf": {
        "@id": "https://goodforbots.com/blog#blog"
      },
      "articleSection": "Methodology",
      "inLanguage": "en",
      "wordCount": 2474,
      "timeRequired": "PT11M"
    },
    {
      "@type": "Person",
      "@id": "https://goodforbots.com/#person-mateusz-pawlica",
      "name": "Mateusz Pawlica",
      "givenName": "Mateusz",
      "familyName": "Pawlica",
      "jobTitle": "Founder of Good for Bots",
      "description": "Web developer who builds directories and AI-powered products, and the founder of Good for Bots.",
      "url": "https://goodforbots.com/about#founder",
      "sameAs": [
        "https://pl.linkedin.com/in/mateusz-pawlica-65238816b",
        "https://pawlicaweb.pl/o-mnie"
      ],
      "knowsAbout": [
        "llms.txt",
        "robots.txt",
        "AI crawlers",
        "Markdown content negotiation",
        "Structured data",
        "Technical SEO",
        "Web directories",
        "Next.js",
        "TypeScript",
        "Generative AI"
      ],
      "worksFor": {
        "@id": "https://goodforbots.com/#organization"
      }
    },
    {
      "@type": "Blog",
      "@id": "https://goodforbots.com/blog#blog",
      "url": "https://goodforbots.com/blog",
      "name": "AI visibility & web standards blog",
      "description": "Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.",
      "inLanguage": "en",
      "publisher": {
        "@id": "https://goodforbots.com/#organization"
      },
      "isPartOf": {
        "@id": "https://goodforbots.com/#website"
      }
    },
    {
      "@type": "Organization",
      "@id": "https://goodforbots.com/#organization",
      "name": "Good for Bots",
      "url": "https://goodforbots.com",
      "logo": "https://goodforbots.com/icon-512.png",
      "founder": {
        "@id": "https://goodforbots.com/#person-mateusz-pawlica"
      },
      "contactPoint": {
        "@type": "ContactPoint",
        "contactType": "customer support",
        "email": "support@goodforbots.com",
        "url": "https://goodforbots.com/contact"
      }
    },
    {
      "@type": "WebSite",
      "@id": "https://goodforbots.com/#website",
      "name": "Good for Bots",
      "url": "https://goodforbots.com",
      "description": "Check if AI crawlers can access and read your website, free. Get a score out of 100 with every finding and fix prompts, then browse reports in the directory.",
      "inLanguage": "en",
      "publisher": {
        "@id": "https://goodforbots.com/#organization"
      },
      "potentialAction": {
        "@type": "SearchAction",
        "target": {
          "@type": "EntryPoint",
          "urlTemplate": "https://goodforbots.com/directory?q={q}"
        },
        "query-input": "required maxlength=120 name=q"
      }
    }
  ]
}
```
