+5score from 95 to 100
Serve Markdown when asked for it
Neither page read passes fully; 2 partly. The homepage serves markdown, but without `Vary: Accept`, so caches may serve it to the wrong client.
Read prompt
My website proxyjam.com scored 95/100 on Good for Bots (Excellent). Good for Bots measures one thing: whether a language model can read a site and cite it. I need you to: serve markdown through content negotiation on all public HTML routes. The check it only partly passes is "Serves markdown when asked for it", with 5 points of it still on the table. What the scanner saw: Neither page read passes fully; 2 partly. The homepage serves markdown, but without `Vary: Accept`, so caches may serve it to the wrong client. The scan checked a sample of 2 pages. It does not establish which other pages have the same issues. Use the findings below to locate the responsible templates, components or configuration, and apply the fix wherever the same issue occurs. Pages in the sample with findings: - https://proxyjam.com/: The homepage serves markdown, but without `Vary: Accept`, so caches may serve it to the wrong client. - https://proxyjam.com/static-residential-proxies/: The page at /static-residential-proxies/ serves markdown, but without `Vary: Accept`, so caches may serve it to the wrong client. The responses reported as missing `Vary: Accept` already serve Markdown. Fix the header in the code or configuration that serves negotiated responses, preserving any existing `Vary` fields. A shared cache needs it to keep HTML and Markdown responses separate. Check each public page type; other findings in the sample may need the fixes described below. Implement content negotiation across all public HTML routes, including page types outside the scan's sample. Put it in the shared response handling or the handlers for each page type, so new pages receive it too. Agents need the content of any page they follow, including pages the report did not name. When a request arrives with `text/markdown` in its `Accept` header, the same URL that serves HTML to a browser should return a markdown rendering of that page's content. Requests without that header must keep getting the HTML exactly as they do now. The response must: - set `Content-Type: text/markdown; charset=utf-8`; - include `Accept` in the `Vary` header, so shared caches never serve the markdown to a browser or the HTML to an agent (RFC 9110 says a server SHOULD send `Vary` whenever the representation depends on a request header); - contain real markdown, not HTML with a different header on it; - drop `ETag` and `Last-Modified` for the converted response, since a conditional request cannot be honoured against a body generated on the fly. Work out how to do this in whatever stack this project uses. If the site is behind Cloudflare on a paid plan, the "Markdown for Agents" zone setting does all of the above and is the least invasive option. Otherwise implement it in the framework: middleware inspecting the `Accept` header, a route handler that returns the markdown source, or whatever fits the architecture here. Strip navigation, headers, footers and cookie banners from the markdown. The point is the content, without the page furniture around it. Verify HTML and Markdown responses for each public page type, including URLs outside the reported sample. Check that each Markdown response contains that page's content and that ordinary browser requests still receive HTML. The whole report, as markdown, is at https://goodforbots.com/r/proxyjam.com.md?scan=cmv6rhoz700j901qz1c9s0z44. Read it first: this prompt covers one finding, and the report has everything the scan saw. When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.
