State a position on AI crawlers
robots.txt names no AI crawlers, so every operator has to guess.
Read prompt
My website maevesocial.com scored 91/100 on Good for Bots (Excellent). Good for Bots measures one thing: whether a language model can read a site and cite it. I need you to: state a position on AI crawlers in robots.txt. The check it fails is "States a position on AI crawlers", worth 5 points. What the scanner saw: robots.txt names no AI crawlers, so every operator has to guess. A `robots.txt` that says nothing about AI crawlers leaves every operator to guess what this site wants, and they guess differently. The point of this task is to state a position, whichever position that is. The thing most people get wrong: the major operators run **separate crawlers for separate purposes**, and blocking one does not block the others. OpenAI runs GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for pages a user asks about. Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User for the same three jobs. Decide about each job, not about the vendor. A common and reasonable position is to refuse training while staying visible in AI answers: ``` # Training: no User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / # Search and user-requested fetches: yes User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / ``` Work out what this project wants before copying that. Allowing everything is equally valid and earns exactly as much here; what is being measured is whether there is an explicit answer, not which answer it is. Blocking a documented search or user-fetch control declares a refusal of that route. Some user-triggered fetchers may still retrieve a page; do not promise complete invisibility. Google-Extended also controls grounding, while CCBot publishes a reusable dataset. Their training opt-outs remain neutral in our score. Two things about how this is judged. The list of crawler tokens we recognise is a snapshot of a moving target: operators add crawlers faster than any scanner tracks them. A rule for a token we have not seen yet is a good rule that earns nothing here until we catch up. And a position stated in a vocabulary rather than in rules, such as `Content-Signal`, is a different check; this one is about the `User-agent` groups. Use the operator sources in the report's crawler catalogue. Unconfirmed or retired names do not count; aliases for one control count once. One mixed-use control still counts as one engagement group. The whole report, as markdown, is at https://goodforbots.com/r/maevesocial.com.md?scan=cmvezk3qr01o101qzwq8ccntc. Read it first: this prompt covers one finding, and the report has everything the scan saw. When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.