+12.5score from 64 to 77
Serve Markdown at its own URL
Markdown is served on request, but not at any URL a crawler would find on its own.
Read prompt
My website huntbug.com scored 64/100 on Good for Bots (Fair). Good for Bots measures one thing: whether a language model can read a site and cite it. I need you to: serve the content as markdown at its own URLs. The check it fails is "Serves markdown at its own URL", worth 12.5 points. What the scanner saw: Markdown is served on request, but not at any URL a crawler would find on its own. This site already serves markdown, but only to a client that asks for it by header, which is a different check and it passes. What is missing is an address. Give the markdown a URL of its own: either `page.md` alongside each page, listed in `/llms.txt`, or a single `/llms-full.txt`. The machinery that produces the markdown already exists, so this is usually a routing change rather than a content one. A model that can only reach this site's content through HTML pays to parse markup and loses structure doing it: navigation, cookie banners and script tags arrive alongside the prose, and headings, tables and code blocks come out flattened. Markdown at its own address removes HTML from the path entirely: what is fetched is the content and nothing else. Two routes satisfy this, and it is worth being clear about which one the specification asks for. llmstxt.org asks for a markdown version of each page **at the same URL as the original**, either with `.md` appended (`page.html.md`) or with the extension replaced (`page.md`). `llms-full.txt`, the single bundle holding everything, does not appear in the specification at all. It is a community convention that grew up around it and is now widespread. Both are accepted here. The per-page form is the one the document asks for. Either of these passes: - **The bundle.** `/llms-full.txt`, one document, real markdown rather than HTML, carrying the site's actual content rather than a placeholder. - **Per-page mirrors.** A markdown twin for each page at `page.md` or `page.html.md`, **and linked from `/llms.txt`**. That second half is load-bearing: mirrors are counted from the links in the index, so twins that nothing points at are invisible to this scan and, more to the point, to anything crawling the site. For the mirror route, the index carries them as ordinary link lines: ```markdown ## Docs - [Quickstart](/docs/quickstart.md): create an account and take a first payment - [Concepts](/docs/concepts.md): customers, subscriptions, invoices and proration - [Webhooks](/docs/webhooks.md): events, signature verification, retry schedule ``` Both `/docs/quickstart` and `/docs/quickstart.md` have to answer 200: the HTML for people, the markdown for everything else. Serving markdown through content negotiation does **not** satisfy this, and that is deliberate rather than an oversight. A negotiated response only exists for a client that knows to send `Accept: text/markdown`, and most crawlers send no preference at all. A `.md` address sits in the open: it is found by ordinary crawling, it can be linked to, and it can be cited. If this site already negotiates markdown, the report says so in these words: *"Markdown is served on request, but not at any URL a crawler would find on its own"*. That is a different check passing on its own terms, not a contradiction. Four things about how this is measured. The bundle is judged by size, by being markdown rather than a page or JSON, and by not merely repeating `llms.txt`; its content is never sampled beyond that, so a couple of kilobytes of boilerplate would pass. Do not take that as permission, since the point of the file is the content in it. `.mdx` counts as markdown although the specification names only `.md`, because excluding it would punish a large part of the documentation ecosystem for its toolchain. The number of mirrors we want before calling it a convention on this site is our judgement, not the specification's. And the mirror URLs listed in the index are counted, never fetched, so a list of `.md` addresses that 404 passes this check and helps nobody at all. The whole report, as markdown, is at https://goodforbots.com/r/huntbug.com.md?scan=cmusib7bb016x01o0z3xopc38. Read it first: this prompt covers one finding, and the report has everything the scan saw. When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.
