+2.5score from 98 to 100
Declare AI usage preferences
No usage declaration found in robots.txt or the final successful homepage response header.
Read prompt
My website bulkshare.cloud scored 98/100 on Good for Bots (Excellent). Good for Bots measures one thing: whether a language model can read a site and cite it. I need you to: declare the site's AI usage preferences in robots.txt or HTTP headers. The check it fails is "Declares AI usage preferences", worth 2.5 points. What the scanner saw: No usage declaration found in robots.txt or the final successful homepage response header. Blocking a crawler says who may fetch a page. This says what may be done with the content afterwards (trained on, indexed, quoted in an answer), which is a different question, and one a site can answer without blocking anybody. There are two vocabularies for it and either is fine. Cloudflare's `Content-Signal` is deployed at scale; the IETF AIPREF working group's `Content-Usage` is the standards-track version. Pick one. Cloudflare's, placed inside a user-agent group: ``` User-agent: * Content-Signal: ai-train=no, search=yes, ai-input=yes Allow: / ``` The three categories are `ai-train` (training or fine-tuning models), `search` (building a search index that links back here) and `ai-input` (feeding the content to a model at answer time, such as retrieval-augmented generation). Each takes `yes` or `no`. The IETF version, with reversed naming for the training category and single-letter values: ``` User-agent: * Content-Usage: train-ai=n, search=y, ai-use=y Allow: / ``` Alternatively, send the same IETF preference as an HTTP response header: ```http Content-Usage: train-ai=n, search=y, ai-use=y ``` Configure it on the content responses it is meant to cover, including the final homepage response after redirects. A header applies only to that response's content; it is not a site-wide declaration. Do not put a robots.txt path prefix in it. The scanner samples the final successful homepage response and awards this check once even when both robots.txt and the header declare preferences. Use lowercase category labels and the unquoted tokens `y` or `n`. Parameters are ignored. Repeated HTTP fields form one dictionary: a duplicate category keeps its last value. Keep that distinct from conflicts between separate applicable AIPREF statements, where disallow takes precedence over allow by default. Keep the carriers consistent with the intended policy; a header does not automatically override robots.txt. Decide what this project wants. A common position is to refuse training while allowing search and grounding, so the site stays citable without feeding a model. Omitting a category is allowed and means "unstated", not "allowed". One caution: nothing compels a crawler to honour either directive. This is a stated preference, not an enforcement mechanism. The standards-track vocabulary is also a draft that can still change under you. The whole report, as markdown, is at https://goodforbots.com/r/bulkshare.cloud.md?scan=cmushcswt00pr01o0wrgzu539. Read it first: this prompt covers one finding, and the report has everything the scan saw. When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.
