burntop

My website burntop.dev scored 62/100 on Good for Bots (Fair). Good for Bots measures one thing: whether a language model can read a site and cite it.

I need you to: serve markdown through content negotiation on all public HTML routes.

The check it fails is "Serves markdown when asked for it", worth 10 points. What the scanner saw: Neither page read passes. Asking the homepage for markdown still returns text/html.

The scan checked a sample of 2 pages. It does not establish which other pages have the same issues. Use the findings below to locate the responsible templates, components or configuration, and apply the fix wherever the same issue occurs. Pages in the sample with findings:
- https://burntop.dev/: Asking the homepage for markdown still returns text/html.
- https://burntop.dev/featured: Asking the page at /featured for markdown still returns text/html.

Implement content negotiation across all public HTML routes, including page types
outside the scan's sample. Put it in the shared response handling or the handlers
for each page type, so new pages receive it too. Agents need the content of any
page they follow, including pages the report did not name.

When a request arrives with `text/markdown` in its `Accept` header, the same URL that
serves HTML to a browser should return a markdown rendering of that page's content.
Requests without that header must keep getting the HTML exactly as they do now.

The response must:

- set `Content-Type: text/markdown; charset=utf-8`;
- include `Accept` in the `Vary` header, so shared caches never serve the markdown to a
  browser or the HTML to an agent (RFC 9110 says a server SHOULD send `Vary` whenever
  the representation depends on a request header);
- contain real markdown, not HTML with a different header on it;
- drop `ETag` and `Last-Modified` for the converted response, since a conditional request
  cannot be honoured against a body generated on the fly.

Work out how to do this in whatever stack this project uses. If the site is behind
Cloudflare on a paid plan, the "Markdown for Agents" zone setting does all of the above
and is the least invasive option. Otherwise implement it in the framework: middleware
inspecting the `Accept` header, a route handler that returns the markdown source, or
whatever fits the architecture here.

Strip navigation, headers, footers and cookie banners from the markdown. The point is the
content, without the page furniture around it.

Verify HTML and Markdown responses for each public page type, including URLs
outside the reported sample. Check that each Markdown response contains that
page's content and that ordinary browser requests still receive HTML.

The whole report, as markdown, is at https://goodforbots.com/r/burntop.dev.md?scan=cmvjeb6hf02od01qzbhf7o4au. Read it first: this prompt covers one finding, and the report has everything the scan saw.

When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.

My website burntop.dev scored 62/100 on Good for Bots (Fair). Good for Bots measures one thing: whether a language model can read a site and cite it.

I need you to: describe the pages with schema.org JSON-LD.

The check it fails is "Describes itself to machines with schema.org JSON-LD", worth 8.33 points. What the scanner saw: 1 of 2 pages read pass. The page at /featured publishes no structured data at all: no JSON-LD, no microdata, no RDFa.

The scan checked a sample of 2 pages. It does not establish which other pages have the same issues. Use the findings below to locate the responsible templates, components or configuration, and apply the fix wherever the same issue occurs. Pages in the sample with findings:
- https://burntop.dev/featured: The page at /featured publishes no structured data at all: no JSON-LD, no microdata, no RDFa.

The pages identified as having no structured data need markup added through their
templates. Review the other public page types for the same gap. Use the homepage
example below where it fits, and describe each content page with types that match
its visible content. A passing homepage does not need its markup replaced.

Review structured data across the site's public page types, including those
outside the reported sample. Add or repair it in the templates that generate
each type, using that page's content. Verify a page from each template and keep
existing valid markup. Structured data gives a reader explicit names, types and
relationships for the entities the page describes.

Put a `<script type="application/ld+json">` block in the HTML the server returns. On the
home page describe the organisation and the site; on a content page describe the content
itself: an `Article` for a post, a `Product` for a product, a `BreadcrumbList` for the
trail above it.

Every entity must:

- declare `"@context": "https://schema.org"`;
- declare a `@type` that schema.org defines;
- carry a name: `name`, or `headline` for an article. A type with no name asserts that
  something exists without saying what, which is worth nothing to a reader;
- describe what is on the page. Structured data that disagrees with the visible content
  is worse than none.

A minimal, correct home page block:

```html
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://burntop.dev/#organization",
      "name": "Example Corp",
      "url": "https://burntop.dev",
      "logo": "https://burntop.dev/logo.png"
    },
    {
      "@type": "WebSite",
      "@id": "https://burntop.dev/#website",
      "name": "Example Corp",
      "url": "https://burntop.dev",
      "publisher": { "@id": "https://burntop.dev/#organization" }
    }
  ]
}
</script>
```

The addresses in that example are this site's. The names are not. Replace `Example Corp`
with the organisation's real name, which is the one thing here that cannot be filled in
from the outside.

Work out how to generate this in whatever stack this project uses, from the same data
that renders the page, so the two cannot drift apart.

Two mistakes account for most broken structured data in the wild. Avoid both:

1. **Do not let the template escape the JSON.** A templating engine that escapes variables
   by default will turn every `"` into `&quot;`, and the block will look perfectly correct
   in the page source while no consumer on earth can parse it, because HTML entities
   inside a `<script>` are never decoded. Use the raw output form the templating engine
   provides:
   `dangerouslySetInnerHTML` in React, `{{{ }}}` in Handlebars, `| safe` in Jinja or
   Nunjucks, `@Html.Raw` in Razor.

2. **Do not inject the block with JavaScript.** Tag managers and client-side SEO plugins
   add it after the page loads, so a crawler that does not run scripts, which is most of
   them, sees nothing at all. The block belongs in the HTML the server sends.

Several separate blocks on one page are fine; the specification requires consumers to
merge them all. A single `@graph`, as above, is just tidier.

The whole report, as markdown, is at https://goodforbots.com/r/burntop.dev.md?scan=cmvjeb6hf02od01qzbhf7o4au. Read it first: this prompt covers one finding, and the report has everything the scan saw.

When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.

My website burntop.dev scored 62/100 on Good for Bots (Fair). Good for Bots measures one thing: whether a language model can read a site and cite it.

I need you to: serve the content as markdown at its own URLs.

The check it only partly passes is "Serves markdown at its own URL", with 6.25 points of it still on the table. What the scanner saw: Only 1 markdown link in llms.txt, too few to be a convention.

`/llms.txt` lists a markdown mirror or two, so the mechanism works; there are not enough
of them to be a convention on this site rather than an accident. Generate a `.md`
twin for every page worth reading and list them all in the index. If the pages come from
markdown already, this is usually a matter of exposing the source rather than converting
anything.

A model that can only reach this site's content through HTML pays to parse markup and
loses structure doing it: navigation, cookie banners and script tags arrive alongside
the prose, and headings, tables and code blocks come out flattened. Markdown at its own
address removes HTML from the path entirely: what is fetched is the content and nothing
else.

Two routes satisfy this, and it is worth being clear about which one the specification
asks for. llmstxt.org asks for a markdown version of each page **at the same URL as the
original**, either with `.md` appended (`page.html.md`) or with the extension replaced
(`page.md`). `llms-full.txt`, the single bundle holding everything, does not appear in
the specification at all. It is a community convention that grew up around it and is
now widespread. Both are accepted here. The per-page form is the one the document
asks for.

Either of these passes:

- **The bundle.** `/llms-full.txt`, one document, real markdown rather than HTML,
  carrying the site's actual content rather than a placeholder.
- **Per-page mirrors.** A markdown twin for each page at `page.md` or `page.html.md`,
  **and linked from `/llms.txt`**. That second half is load-bearing: mirrors are counted
  from the links in the index, so twins that nothing points at are invisible to this scan
  and, more to the point, to anything crawling the site.

For the mirror route, the index carries them as ordinary link lines:

```markdown
## Docs

- [Quickstart](/docs/quickstart.md): create an account and take a first payment
- [Concepts](/docs/concepts.md): customers, subscriptions, invoices and proration
- [Webhooks](/docs/webhooks.md): events, signature verification, retry schedule
```

Both `/docs/quickstart` and `/docs/quickstart.md` have to answer 200: the HTML for
people, the markdown for everything else.

Serving markdown through content negotiation does **not** satisfy this, and that is
deliberate rather than an oversight. A negotiated response only exists for a client that
knows to send `Accept: text/markdown`, and most crawlers send no preference at all. A
`.md` address sits in the open: it is found by ordinary crawling, it can be linked to,
and it can be cited. If this site already negotiates markdown, the report says so in
these words: *"Markdown is served on request, but not at any URL a crawler would find on
its own"*. That is a different check passing on its own terms, not a contradiction.

Four things about how this is measured. The bundle is judged by size, by being markdown
rather than a page or JSON, and by not merely repeating `llms.txt`; its content is never
sampled beyond that, so a couple of kilobytes of boilerplate would pass. Do not take that
as permission, since the point of the file is the content in it. `.mdx` counts as markdown
although the specification names only `.md`, because excluding it would punish a large
part of the documentation ecosystem for its toolchain. The number of mirrors we want
before calling it a convention on this site is our judgement, not the specification's.
And the mirror URLs listed in the index are counted, never fetched, so a list of `.md`
addresses that 404 passes this check and helps nobody at all.

The whole report, as markdown, is at https://goodforbots.com/r/burntop.dev.md?scan=cmvjeb6hf02od01qzbhf7o4au. Read it first: this prompt covers one finding, and the report has everything the scan saw.

When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.