--- title: "GoodForBotsBot: crawler & robots.txt" description: "Meet the Good for Bots crawler: its purpose, User-Agent, crawl limits, robots.txt controls and how to identify or report its traffic." url: "https://goodforbots.com/bot" --- # Meet GoodForBotsBot: crawler & robots.txt. Our crawler checks how easily language models and agents can read a website. Its findings power the Good for Bots directory, public reports and 0–100 scores. ## Why we visit If you see GoodForBotsBot in your logs, we are checking your site for the Good for Bots directory or updating an existing report. We look at whether AI tools can access your content, understand it and find their way around. - [Explore the checks and scoring methodology](https://goodforbots.com/standards) The result is a public report with a score and practical suggestions. Payment never changes that score, and a high score does not guarantee a citation from an AI assistant. GoodForBotsBot is operated by Good for Bots, founded by Mateusz Pawlica. We use collected content to produce reports and AI-assisted directory descriptions, not to build a model-training dataset. - [About Good for Bots](https://goodforbots.com/about) ## Recognise our requests HTTP User-Agent ```text GoodForBotsBot/1.0 (+https://goodforbots.com/bot) ``` Every request identifies itself with this User-Agent. In robots.txt, our name is simply GoodForBotsBot. Our scanner and free tools sign HTTPS GET and HEAD requests with Ed25519 Web Bot Auth, including reads through our browser provider. Verify the signature against our public key directory, check its expiry and confirm the signed destination. A copied User-Agent alone does not prove a request came from us. Ownership checks and listing-image requests are not signed. - [Public signing-key directory](https://goodforbots.com/.well-known/http-message-signatures-directory) - [Cloudflare: Web Bot Auth verification](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/) ## What we read We check your homepage, up to three other pages and the public files that help bots understand your site. We do not crawl every page in your sitemap, sign in, submit forms or make purchases. - Your homepage, including its Markdown version when available through [Markdown negotiation](https://goodforbots.com/standards#markdown-negotiation). - Up to three other pages chosen from your sitemaps and homepage links, each also requested as Markdown. The next scan reads the same pages while they stay available. These requests never go through a browser, and we skip them when the scan is close to its limits. - robots.txt, llms.txt, a sample of llms-full.txt and your sitemaps. - Public discovery files under /.well-known/, auth.md and related metadata your site links to. - If your /.well-known/x402 file lists paid resources, one of them, requested once with GET to read its payment request. We never pay and never send payment credentials. - Your logo and preview image, to illustrate your directory listing. - When an account requests ownership verification: robots.txt and either your homepage’s verification meta tag or /.well-known/goodforbots.txt. DNS verification reads a TXT record instead. - When someone runs one of our [free tools](https://goodforbots.com/tools) on your site: robots.txt, then /llms.txt, the start of /llms-full.txt and your homepage as Markdown for the llms.txt checker, or your homepage for the robots.txt checker. ## Crawl limits & frequency - Small, bounded scans. Each scoring attempt is limited to 32 file or page fetches and five minutes. Redirects and browser-assisted requests can add traffic; listing images have a separate limit of 12 requests and 30 seconds. - Ownership checks have a separate limit of eight HTTP requests, including robots.txt and redirects, and 30 seconds. Requests pause at least two seconds apart and respect Crawl-delay. These checks do not run a scoring scan. - A free tool run is limited to 8 file or page fetches and 45 seconds, never uses a browser and never overlaps a scan of your site. It follows the same robots.txt rules, pauses and Retry-After as a scan, and publishes nothing. - Time between requests. Automatic rechecks pause 2–5 seconds between requests. Other queued scans use a half-second pause by default. You can ask us to go slower with Crawl-delay. - Your rules come first. We never request a path your robots.txt disallows for us, and a scan stops when it refuses your homepage. If your server responds with HTTP 429 (Too Many Requests), we stop collecting and respect Retry-After before retrying. - Where we scan from. Our scans run from servers in the EU. A homepage that answers HTTP 451 (Unavailable For Legal Reasons) gets no score, because we cannot see what it serves elsewhere. How often we return depends on your listing’s update schedule. Requested scans and retries can also cause a visit. Contact us if our traffic is causing a problem. ## Allow, slow down or stop the bot Add the appropriate example to your site’s robots.txt. Keep your existing rules for other bots. Rules addressed to GoodForBotsBot take precedence over your wildcard (*) group, so include any private paths you still want to disallow. For rules aimed at other AI crawlers, read [how to write robots.txt rules for AI crawlers](https://goodforbots.com/guides/robots-txt-ai-crawlers). Allow access ```text User-agent: GoodForBotsBot Allow: / ``` Allow access with at least 10 seconds between requests ```text User-agent: GoodForBotsBot Allow: / Crawl-delay: 10 ``` Stop scanning ```text User-agent: GoodForBotsBot Disallow: / ``` We read your rules again on the next scan. If you block us from your homepage, your site becomes Opted out: no score and no directory listing. An existing report may still be reachable by its link, but is marked not to be indexed by search engines. If you block only some paths, we leave them unread and the report says which files your rules keep from crawlers. To remove your listing, submit a removal request with the domain. Removal is free and unconditional; you do not need a paid plan. - [Request removal](https://goodforbots.com/remove) - [Report a problem](https://goodforbots.com/contact) ## Cloudflare & firewalls GoodForBotsBot is not currently a Cloudflare Verified Bot. Our application is awaiting review. We will update this page when our status changes. Your firewall can block a scan even when robots.txt allows it. Our [bot challenge check](https://goodforbots.com/standards#waf-challenge) explains how we recognise these barriers. If that happens, find the request in your security events and contact us. We can help confirm the traffic and identify what needs to be allowed. Do not disable your firewall or trust a request just because it uses our name. Any exception should cover confirmed traffic and only the protection causing the block. Cloudflare’s settings differ by product; its basic Bot Fight Mode cannot be bypassed with a custom Skip rule. - [Cloudflare: Verified Bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) - [Cloudflare: Skip rule options](https://developers.cloudflare.com/waf/custom-rules/skip/options/) ## Something looks wrong? Email support@goodforbots.com about unexpected traffic, access issues or listing removal. Include your domain and what happened. For a traffic question, the time, source IP and a short log excerpt help us investigate. Please leave out passwords, cookies and other private information. - [support@goodforbots.com](mailto:support@goodforbots.com) - [Contact form](https://goodforbots.com/contact) ## Structured data ```json { "@context": "https://schema.org", "@graph": [ { "@type": "WebPage", "@id": "https://goodforbots.com/bot#webpage", "url": "https://goodforbots.com/bot", "name": "GoodForBotsBot: crawler & robots.txt", "description": "Meet the Good for Bots crawler: its purpose, User-Agent, crawl limits, robots.txt controls and how to identify or report its traffic.", "inLanguage": "en", "isPartOf": { "@id": "https://goodforbots.com/#website" }, "breadcrumb": { "@id": "https://goodforbots.com/bot#breadcrumb" }, "about": { "@id": "https://goodforbots.com/#organization" } }, { "@type": "BreadcrumbList", "@id": "https://goodforbots.com/bot#breadcrumb", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://goodforbots.com/" }, { "@type": "ListItem", "position": 2, "name": "Crawler documentation", "item": "https://goodforbots.com/bot" } ] }, { "@type": "Organization", "@id": "https://goodforbots.com/#organization", "name": "Good for Bots", "url": "https://goodforbots.com", "logo": "https://goodforbots.com/icon-512.png", "founder": { "@id": "https://goodforbots.com/#person-mateusz-pawlica" }, "contactPoint": { "@type": "ContactPoint", "contactType": "customer support", "email": "support@goodforbots.com", "url": "https://goodforbots.com/contact" } }, { "@type": "WebSite", "@id": "https://goodforbots.com/#website", "name": "Good for Bots", "url": "https://goodforbots.com", "description": "Check if AI crawlers can access and read your website, free. Get a score out of 100 with every finding and fix prompts, then browse reports in the directory.", "inLanguage": "en", "publisher": { "@id": "https://goodforbots.com/#organization" }, "potentialAction": { "@type": "SearchAction", "target": { "@type": "EntryPoint", "urlTemplate": "https://goodforbots.com/directory?q={q}" }, "query-input": "required maxlength=120 name=q" } }, { "@type": "Person", "@id": "https://goodforbots.com/#person-mateusz-pawlica", "name": "Mateusz Pawlica", "givenName": "Mateusz", "familyName": "Pawlica", "jobTitle": "Founder of Good for Bots", "description": "Web developer who builds directories and AI-powered products, and the founder of Good for Bots.", "url": "https://goodforbots.com/about#founder", "sameAs": [ "https://pl.linkedin.com/in/mateusz-pawlica-65238816b", "https://pawlicaweb.pl/o-mnie" ], "knowsAbout": [ "llms.txt", "robots.txt", "AI crawlers", "Markdown content negotiation", "Structured data", "Technical SEO", "Web directories", "Next.js", "TypeScript", "Generative AI" ], "worksFor": { "@id": "https://goodforbots.com/#organization" } } ] } ```