+10score from 73 to 83
Serve Markdown when asked for it
None of the 4 pages read pass. Asking the homepage for markdown still returns text/html.
Read prompt
My website shiftatmidnightwiki.com scored 73/100 on Good for Bots (Fair). Good for Bots measures one thing: whether a language model can read a site and cite it. I need you to: serve markdown through content negotiation on all public HTML routes. The check it fails is "Serves markdown when asked for it", worth 10 points. What the scanner saw: None of the 4 pages read pass. Asking the homepage for markdown still returns text/html. The scan checked a sample of 4 pages. It does not establish which other pages have the same issues. Use the findings below to locate the responsible templates, components or configuration, and apply the fix wherever the same issue occurs. Pages in the sample with findings: - https://shiftatmidnightwiki.com/: Asking the homepage for markdown still returns text/html. - https://shiftatmidnightwiki.com/guides/beginner-guide/: Asking the page at /guides/beginner-guide/ for markdown still returns text/html. - https://shiftatmidnightwiki.com/about/: Asking the page at /about/ for markdown still returns text/html. - https://shiftatmidnightwiki.com/doppelgangers/: Asking the page at /doppelgangers/ for markdown still returns text/html. Implement content negotiation across all public HTML routes, including page types outside the scan's sample. Put it in the shared response handling or the handlers for each page type, so new pages receive it too. Agents need the content of any page they follow, including pages the report did not name. When a request arrives with `text/markdown` in its `Accept` header, the same URL that serves HTML to a browser should return a markdown rendering of that page's content. Requests without that header must keep getting the HTML exactly as they do now. The response must: - set `Content-Type: text/markdown; charset=utf-8`; - include `Accept` in the `Vary` header, so shared caches never serve the markdown to a browser or the HTML to an agent (RFC 9110 says a server SHOULD send `Vary` whenever the representation depends on a request header); - contain real markdown, not HTML with a different header on it; - drop `ETag` and `Last-Modified` for the converted response, since a conditional request cannot be honoured against a body generated on the fly. Work out how to do this in whatever stack this project uses. If the site is behind Cloudflare on a paid plan, the "Markdown for Agents" zone setting does all of the above and is the least invasive option. Otherwise implement it in the framework: middleware inspecting the `Accept` header, a route handler that returns the markdown source, or whatever fits the architecture here. Strip navigation, headers, footers and cookie banners from the markdown. The point is the content, without the page furniture around it. Verify HTML and Markdown responses for each public page type, including URLs outside the reported sample. Check that each Markdown response contains that page's content and that ordinary browser requests still receive HTML. The whole report, as markdown, is at https://goodforbots.com/r/shiftatmidnightwiki.com.md?scan=cmuthmest002f01p3u5htf873. Read it first: this prompt covers one finding, and the report has everything the scan saw. When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.
