+6.3score from 85 to 91
Serve Markdown when asked for it
None of the 4 pages read pass. Asking the homepage for markdown still returns text/html.
Read prompt
My website saascubes.com scored 85/100 on Good for Bots (Good). Good for Bots measures one thing: whether a language model can read a site and cite it. The check it fails is "Serves markdown when asked for it", worth 6.25 points. What the scanner saw: None of the 4 pages read pass. Asking the homepage for markdown still returns text/html. The scan read 4 pages. These fell short: - https://saascubes.com/: Asking the homepage for markdown still returns text/html. - https://saascubes.com/blog/ai-seo-for-saas: Asking the page at /blog/ai-seo-for-saas for markdown still returns text/html. - https://saascubes.com/pricing: Asking the page at /pricing for markdown still returns text/html. - https://saascubes.com/directory/category/ai-ml: Asking the page at /directory/category/ai-ml for markdown still returns text/html. I need you to: serve markdown through HTTP content negotiation. Here is what that means. An agent that fetches a page gets markup written for a browser: navigation, cookie banner, script tags, and the content somewhere inside. Content negotiation lets the same address answer with the content alone when the client asks for it, which costs the agent nothing to try and costs a browser nothing at all. When a request arrives with `text/markdown` in its `Accept` header, the same URL that serves HTML to a browser should return a markdown rendering of that page's content. Requests without that header must keep getting the HTML exactly as they do now. The response must: - set `Content-Type: text/markdown; charset=utf-8`; - include `Accept` in the `Vary` header, so shared caches never serve the markdown to a browser or the HTML to an agent (RFC 9110 says a server SHOULD send `Vary` whenever the representation depends on a request header); - contain real markdown, not HTML with a different header on it; - drop `ETag` and `Last-Modified` for the converted response, since a conditional request cannot be honoured against a body generated on the fly. Work out how to do this in whatever stack this project uses. If the site is behind Cloudflare on a paid plan, the "Markdown for Agents" zone setting does all of the above and is the least invasive option. Otherwise implement it in the framework: middleware inspecting the `Accept` header, a route handler that returns the markdown source, or whatever fits the architecture here. Strip navigation, headers, footers and cookie banners from the markdown. The point is the content, without the page furniture around it. The scan asks the homepage and a few sampled pages, and the report lists which ones failed. Implement negotiation for every HTML route, including pages the report does not name: the next scan may sample different content. The whole report, as markdown, is at https://goodforbots.com/r/saascubes.com.md?scan=cmunx1nol000101nq3dc8ctm1. Read it first: this prompt covers one finding, and the report has everything the scan saw. When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.
