Frequently asked questions

Good for Bots is a directory of websites scored on one question: can a language model or an AI agent read the site? Every listed site gets a Good for Bots Score from 0 to 100, a public report and a live badge; the report for stripe.com is a real example. The directory compares sites, and Standards and scoring publishes every check behind the number.

Yes. Checking a site, its score, its public report and the live badge cost nothing and need no account. Pro is an optional one-time upgrade of $9 per listing, and Featured is a place on the home page for $20 a month. No plan changes the score; the pricing page compares them.

Good for Bots was founded by Mateusz Pawlica, a web developer who built the directories SaaS Cubes and Mapa Oświatowa before it. It is run by PawlicaWeb Mateusz Pawlica, a sole trader registered in Poland. Who we are gives the full business details, and About us explains why the project exists.

It has to: our own site must pass the same checks as every site we score. Public pages are rendered on the server, the main ones also answer in Markdown (this page is at /faq.md), /llms.txt maps them for language models, and every page carries schema.org structured data. The standards catalogue lists those checks.

The Good for Bots Score measures how easily language models and AI agents can reach, read and understand a website, from 0 to 100. It looks at what a site offers machines, such as robots.txt, llms.txt, sitemaps, Markdown versions and structured data, and at barriers such as bot challenges. It does not rate the quality of the content, the business behind it or its search ranking. Standards and scoring explains every check.

Achievements add points and penalties take them away. The achievements come in two parts. The readable base, scaled to 50 points, asks whether a model can read and follow the HTML our crawler receives: headings and landmarks, working links, structured data, a sitemap and robots.txt. The agent layer, scaled to 50 points, covers what a site offers language models on purpose, such as llms.txt, llms-full.txt, Markdown versions and rules for AI crawlers. A check that passes earns full credit, a warning half and a failure nothing, and the score never falls below zero. How scoring works shows how the points add up, and What can lower the score lists the deductions.

Every score falls into a band: Excellent (90–100), Good (75–89), Fair (45–74), Poor (0–44). The band is a shorthand for the number, and the badge and the report show both. A complete readable base on its own reaches Fair; Good needs what the site does for agents on top of it. How scoring works shows where each band starts.

No. The score is never for sale: paying does not add points, improve a band, hide a score or remove a listing. Free and paid listings are scanned and scored by exactly the same rules. It is the first of our commitments.

A high score does not guarantee that ChatGPT, Claude or Perplexity will cite your site. Our crawler checks access and machine-readable content; an assistant chooses sources for the question it receives from the material it retrieves. Check your site to find technical issues you can address.

No. Blocking crawlers that collect training data, such as GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended, costs nothing. What costs points is blocking the bots that search and fetch pages to answer people's questions, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot: each one blocked is a deduction. Declaring your training preference with Content Signals even earns points.

Barriers that stop a bot from reading the site: blocking AI search or user-triggered bots in robots.txt, a noindex or nosnippet directive on the homepage, a firewall challenge in front of our bot, a homepage that is not HTTPS or does not answer 200, a missing title or meta description, a long redirect chain and a slow response. When one refused response explains several problems, only the most specific penalty counts. When the homepage gives a crawler nothing to read, because of a challenge, an answer other than 200 or a page that stays empty without JavaScript, the score also stays below Fair. What can lower the score gives each deduction.

An experimental check runs on every scan and appears on the report, but at zero weight: it adds and subtracts nothing. Every new check starts that way and counts only after we have tested it on real sites; the new rule then applies to every listing at once. The standards catalogue marks which checks are experimental.

It counts capabilities: signs that a site offers agents something beyond reading, such as an API catalog, an MCP server card or an agent card. Capabilities never add or cost points; they show as +N on the badge and on the report. Capabilities for agents lists the ones we detect.

Either the site changed or our rules did. Each scan reads the site afresh, so a new robots.txt rule, a missing file or a firewall challenge shows up in the next score. We also add and adjust checks; new scoring rules apply to every listing at once, paid or not, as Changes to the service describes. The report dates its scan and lists every check with its result.

Enter its address in the home page form. The first scan of a site needs no account; it runs in the background, and the progress page opens the report when it is ready. If the site is already listed, the form takes you straight to its report.

The homepage and up to three other pages chosen from your sitemaps and homepage links, each also requested as Markdown, and the public files meant for machines: robots.txt, llms.txt, a sample of llms-full.txt, sitemaps and discovery files under /.well-known/. It also fetches the logo and preview image for the listing. It never signs in, submits forms, makes purchases or crawls every page in your sitemap; What we read has the full list.

No. Four checks (structured data, page structure, Markdown negotiation and typed links) look at the homepage and up to three other pages from your sitemaps and homepage links, one page at a time, and the median page sets the result: with four pages, one weak page does not fail the check, but two do. The other checks read the homepage and site-wide files such as robots.txt, llms.txt and sitemaps. A problem on a page outside that sample does not show in the score, and a badge counts only on a page the scan reads, so put it on your homepage. What we read lists every request.

No. The scan reads the HTML your server sends, which is what a crawler that does not run JavaScript receives. When a firewall refuses the plain request, we may fetch the same address through a browser, but only to get past the refusal, never to render the page. The JavaScript check penalises empty script-bearing responses when the captured HTML gives no evidence of text, fallback content or meaningful elements. Short or ambiguous pages receive no penalty; partial hydration can go undetected. How to serve content AI crawlers can read without JavaScript shows how to put your content in the response.

Claim it first: only a listing's verified owner can ask for another scan, from the report or the dashboard. On Free, that is 5 manual rescans per 30 days, at least 24 hours apart; on Pro, 30 manual rescans per 30 days, at least 1 hour apart. A scan that fails for technical reasons does not use up the allowance, and a finished manual scan resets the automatic schedule.

Claimed Free listings are rescanned every month, and claimed Pro listings every week. Unclaimed listings are rescanned every 90 days, with or without a badge. A finished manual scan resets the schedule, and the pricing page compares the plans.

Opted out means your robots.txt refuses GoodForBotsBot, so we stopped and published no score; allow our bot and the next scan will read the site. Blocked means a firewall or bot challenge stopped every attempt to read the homepage, so there was nothing to score; Cloudflare and firewalls explains what to check. Neither state is shown as a number.

GoodForBotsBot is the crawler behind Good for Bots. Every request it makes carries one User-Agent, GoodForBotsBot/1.0 (+https://goodforbots.com/bot), and in robots.txt its name is simply GoodForBotsBot. It reads sites to score them and to keep their reports current; Meet GoodForBotsBot documents it in full.

Always. It reads robots.txt before anything else and stops if the file refuses it, whatever status the file is served with. It honours Crawl-delay, stops collecting on HTTP 429 and waits for Retry-After before trying again. Crawl limits and frequency covers the rest.

Add a group for it to robots.txt: User-agent: GoodForBotsBot followed by Disallow: /. The next scan stops there, and your site becomes Opted out: no score and no directory listing. To take the report down as well, request removal, which is free. Allow, slow down or stop the bot has examples to copy.

Find the blocked request in your firewall's security events and make an exception for that traffic, covering only the protection that blocked it. GoodForBotsBot is not yet a Cloudflare Verified Bot and we do not publish a list of our IP addresses, so contact us with a log excerpt if you want help confirming a request is ours. Cloudflare and firewalls explains the options.

A small, declared number. Each scoring attempt is limited to 32 file or page fetches and five minutes; listing images and ownership checks have smaller limits of their own, and requests are spaced out. Crawl limits and frequency gives every limit and pause.

No. GoodForBotsBot collects content to produce reports and the AI-written parts of a listing, not a training dataset. Google's Gemini model writes the listing description and the report's “Through a bot's eyes” section from what the scan collected, and we keep the raw responses for 30 days to check and explain results. If your website is in the directory has the details.

The score and its band, every check with its result and the evidence behind it, and the fixes ranked by the points each would add, with prompts for your coding agent. A report also carries an AI-written description of the site, the date of its scan and the code for its badge. The report for stripe.com is a real example.

Someone scans it. Anyone can submit a public website on the home page, and a scan that ends with a score puts it in the directory. We do not fill the directory with scans of our own, and a site that refuses our bot in robots.txt is not listed.

A fix prompt is a ready-written instruction for a coding agent such as Claude Code, Cursor or Codex: it names the standard and what the scan found, and your agent fits the change to your stack. Free shows prompts for the 3 fixes worth the most. Pro shows one for every fix. The score is the same either way. See them on a real report.

An AI model, Google's Gemini, writes it from the content the scan collected, and the report labels it as AI-written. It can be wrong; if it is, tell us and include the website address. Scores, reports and AI-written text explains which parts of a report a model writes.

A language model picks one category for the type of site, such as Documentation or E-commerce Stores, and up to three industry tags, all from fixed lists; a site that fits nothing goes to Other. The category decides where a site appears in the directory, never its score: every category is scored by the same rules.

A report is open to search engines only once its latest scan is complete: the homepage answered with HTTP 200, and robots.txt, llms.txt, the sitemap, the page structure and schema.org all have a result. Until then the report carries noindex. Reports of opted-out and blocked sites stay noindex, and older scans are never indexed separately: the current report is the one address.

Open the claim page or the claim link on your report, then prove you control the site in one of four ways: a meta tag on the homepage, a file at /.well-known/goodforbots.txt, a DNS TXT record, or a confirmation email to webmaster@ or hostmaster@ on the domain. Claiming is free, needs an account and never changes the score. Claiming a website sets out the rules.

Claiming verifies that you control the website and adds its listing to your dashboard. You can request rescans within your plan's limits and upgrade to Pro. Claimed Free listings are automatically rescanned every month; unclaimed listings every 90 days. Claiming does not change the score or publish your name.

Not to scan a site, read reports or remove a listing. You need one to claim a site, request rescans or buy a plan. There is no password: we send a single-use sign-in link to your email address. You must be at least 16; Your account and claims explains what we store.

Send a request on the removal page. Removal is free and unconditional: no account, no reason and no payment, on any plan. Once we have checked that the request comes from someone with authority over the site, we hide its directory entry, report, history and badge, stop scanning it and keep it from being listed again, then email you to confirm.

Opting out is a robots.txt rule: our bot stops, the site loses its score and leaves the directory, but its last report can stay reachable by its link, marked noindex. Removal is a request to us: the entry, report, history and badge disappear, and the domain is not listed again. Removing a listing and opting out compares the two.

The listing keeps its current owner until support steps in, because we never replace an owner automatically. If you have taken over a domain, contact us and we will review the claim. Pro belongs to the listing, not to the person who paid, so it stays with the site.

A small image for your site that shows its live score and band and links to its report. It is an SVG inside an ordinary link, so it needs no JavaScript or plugin, and it changes after every published scan. It is free on every plan; the badge page shows what it looks like.

Find your site on the badge page or open your report, copy the HTML snippet and paste it into your homepage, ideally the footer. Keep the hosted image address so the score stays current. There is a Markdown snippet for GitHub READMEs too.

Every listing's report links to its website: that link is the listing's backlink. On Free, it turns dofollow once a scan finds a link to goodforbots.com on your homepage, which the badge provides; the find counts for 60 days, and a scan that reads the whole homepage without it takes the dofollow away. Pro listings are dofollow with or without the badge. A badge in a GitHub README earns nothing, because it is not on your site. The badge page explains the rest.

Pro changes what a listing's owner gets, never the score. It keeps the listing's backlink dofollow without the badge; shows a prompt for every fix; drafts a ready-made llms.txt and robots.txt from your scan; rescans the listing every week; allows 30 manual rescans per 30 days, at least 1 hour apart; emails you when an automatic scan detects a score drop or a change in bot access; publishes the score's history as a chart; and adds gold corners to the listing, which change its look, never its position. What Pro adds shows each one on a report.

No. Pro is one payment of $9 per listing. It belongs to the listing, never renews and lasts for as long as Good for Bots runs. Only the verified owner can buy it, so claim the site first.

Featured is a place on the home page: one of 3 slots showing a site's logo, description and score, with a dofollow backlink, for $20 a month. You can feature any public listing that has a score, yours or someone else's. It buys no points and none of Pro's tools; the Featured page shows the current placements.

Yes. You can ask for a full refund within 14 days of a Pro payment or of the first payment of a Featured subscription, without giving a reason; the purchase then ends. Monthly renewals are not refunded, because you can cancel before any renewal. Refunds and your right to withdraw has the details.

Stripe runs checkout and collects the billing details needed for your payment. Receipts and invoices come from Link. Payment history and billing documents are also under Orders in your account.

With the readable base, worth 50 of the 100 points: a homepage whose HTML carries its text without JavaScript, with headings, a main element, working links, structured data and a sitemap. Then add what agents use, starting with llms.txt, llms-full.txt and Markdown versions of your pages. Your report ranks the remaining fixes by the points each would add, and each comes with a prompt for your coding agent. Check your site to get the list.

llms.txt is a Markdown file at the root of a site that tells language models what the site is and links its most useful pages, as proposed at llmstxt.org. It counts in the agent layer, the half of the score for what a site offers language models, together with llms-full.txt and Markdown versions. Our llms.txt check shows what we look for, and How to write llms.txt walks through writing one. The llms.txt checker looks at your file on its own.

That is your choice, and the score respects it. Blocking training crawlers costs nothing. Blocking the bots that search and fetch pages for people's questions keeps your site out of those answers, and each one blocked lowers the score. Explicit rules for AI crawlers earn points whichever way they go: AI crawler rules shows how we read them, and Content Signals lets you state your terms for AI use. How to write robots.txt rules for AI crawlers has a tested example and the crawler list, and the robots.txt checker shows how your own file treats each crawler.

Your server answers a request carrying Accept: text/markdown with a Markdown version of the page, and sends Vary: Accept so caches keep the two apart. Agents get the content without the navigation and scripts. This page does it: request it with that header, or open /faq.md. Our Markdown check explains the details.

Valid JSON-LD on the homepage and the other pages the scan reads, using the schema.org vocabulary, that names what each page describes, such as an organisation, a product or an article, with the properties that make it useful. A block that does not parse, an entity without a name or data only in microdata or RDFa makes that page a warning. The median page sets the result, and a warning is worth half the points. Our schema.org check lists what it expects.

The published report and its history stay online while the site is listed; the raw responses behind them are kept for 30 days to check and explain results. Whoever requests a scan is kept only in temporary counters, as keyed hashes, for up to 24 hours, and public scan progress never shows who asked. Scans and free tools and How long we keep data have the details.

One essential cookie keeps you signed in. Google Analytics loads only if you accept it, never on sign-in or claim pages, and Cookie settings in the footer changes your choice at any time. Badge views are counted without cookies. Cookies and analytics lists everything we store.

No. There is no newsletter and no marketing: we send only emails that belong to the service, such as sign-in links, the monitoring alerts your plan includes and payment receipts. Emails we send lists them all.