Serve our crawler without a challenge
A Cloudflare challenge identified by the cf-mitigated header. A model fetching this page gets the challenge, not the content. We read the page only by escalating to a browser, which a model cannot do.
Read prompt
My website gmapsscraper.io scored 32/100 on Good for Bots (Poor). Good for Bots measures one thing: whether a language model can read a site and cite it. I need you to: let the Good for Bots crawler through the WAF. The check it fails is "Serves our crawler without a challenge", a penalty taking 15 points off the score. What the scanner saw: A Cloudflare challenge identified by the cf-mitigated header. A model fetching this page gets the challenge, not the content. We read the page only by escalating to a browser, which a model cannot do. The challenge is Cloudflare's. Verifying this crawler's Web Bot Auth signature does not exempt it from every Cloudflare protection, so start from the challenged request in Security → Analytics → Events, whose Service field names the protection that issued the challenge. Bot Fight Mode, on the Free plan, runs outside the rules engine: no custom rule can skip, bypass or allow a single crawler through it, and it can only be switched off for the whole zone. Super Bot Fight Mode (Pro plan and above), custom rules, managed rules and rate limiting all accept an exception: a custom rule matching `(cf.client.bot and http.user_agent eq "GoodForBotsBot/1.0 (+https://goodforbots.com/bot)")` with the *Skip* action, set to skip what the event named (**All Super Bot Fight Mode rules**, for example, or Security Level and Browser Integrity Check among the products) and placed above any custom rule that issues the challenge. Requiring `cf.client.bot`, *Known Bots* in the expression builder, keeps out a client that only copies the user agent. When something in front of a site answers a crawler with an interstitial, a CAPTCHA or a flat refusal, none of the work that site has done for machines is reachable. The same thing happens to model crawlers, which is the part that matters: a challenge nobody can solve is indistinguishable, from the other side, from having no content at all. The crawler sends exactly one user agent on every request: ``` GoodForBotsBot/1.0 (+https://goodforbots.com/bot) ``` It makes roughly twenty requests per scan, sequentially, and it obeys `robots.txt`. Add an allowlist rule for that user agent in whatever sits in front of this site: - **Cloudflare**: a WAF custom rule with the *Skip* action matching this user agent together with `cf.client.bot`, the field Cloudflare sets for a verified bot. Bot Fight Mode, the Free plan's bot protection, accepts no exceptions. - **AWS WAF**: a rule with `Allow` above the bot-control rule group, matching the `User-Agent` header. - **Akamai**: a Bot Manager exception for the user agent; a rate policy alone is not enough. - **Imperva / Incapsula**: an allowlist entry for the user agent under bot access control. - **Vercel**: exclude the user agent from the Attack Challenge Mode or firewall rule. - **Fastly, nginx, or a hand-rolled rule**: whatever matches on `User-Agent` before the challenge is issued. If the intention is to block us, do that in `robots.txt` instead. A `Disallow` for this crawler is honoured immediately and without argument, it costs no points, and it is the honest way to say no. Only `robots.txt` counts as a refusal; a challenge is read as a site a model cannot use. Do not widen the rule beyond this one user agent. Where the protection accepts an exception, one entry is the whole change. Where it does not, as with Cloudflare's Bot Fight Mode, leave the protection on and tell the site owner: switching it off changes how the site treats every bot, and that decision is theirs. The whole report, as markdown, is at https://goodforbots.com/r/gmapsscraper.io.md?scan=cmvw4tgqf43qs01n0fdohwmxw. Read it first: this prompt covers one finding, and the report has everything the scan saw. When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.