Youfiliate

My website youfiliate.com scored 66/100 on Good for Bots (Fair). Good for Bots measures one thing: whether a language model can read a site and cite it.

I need you to: serve markdown through content negotiation on all public HTML routes.

The check it fails is "Serves markdown when asked for it", worth 10 points. What the scanner saw: None of the 4 pages read pass. Asking the homepage for markdown still returns text/html.

The scan checked a sample of 4 pages. It does not establish which other pages have the same issues. Use the findings below to locate the responsible templates, components or configuration, and apply the fix wherever the same issue occurs. Pages in the sample with findings:
- https://www.youfiliate.com/: Asking the homepage for markdown still returns text/html.
- https://www.youfiliate.com/blog/broken-affiliate-links-costing-you-money: Asking the page at /blog/broken-affiliate-links-costing-you-money for markdown still returns text/html.
- https://www.youfiliate.com/blog: Asking the page at /blog for markdown still returns text/html.
- https://www.youfiliate.com/blog/what-is-deep-linking-affiliate-links: Asking the page at /blog/what-is-deep-linking-affiliate-links for markdown still returns text/html.

Implement content negotiation across all public HTML routes, including page types
outside the scan's sample. Put it in the shared response handling or the handlers
for each page type, so new pages receive it too. Agents need the content of any
page they follow, including pages the report did not name.

When a request arrives with `text/markdown` in its `Accept` header, the same URL that
serves HTML to a browser should return a markdown rendering of that page's content.
Requests without that header must keep getting the HTML exactly as they do now.

The response must:

- set `Content-Type: text/markdown; charset=utf-8`;
- include `Accept` in the `Vary` header, so shared caches never serve the markdown to a
  browser or the HTML to an agent (RFC 9110 says a server SHOULD send `Vary` whenever
  the representation depends on a request header);
- contain real markdown, not HTML with a different header on it;
- drop `ETag` and `Last-Modified` for the converted response, since a conditional request
  cannot be honoured against a body generated on the fly.

Work out how to do this in whatever stack this project uses. If the site is behind
Cloudflare on a paid plan, the "Markdown for Agents" zone setting does all of the above
and is the least invasive option. Otherwise implement it in the framework: middleware
inspecting the `Accept` header, a route handler that returns the markdown source, or
whatever fits the architecture here.

Strip navigation, headers, footers and cookie banners from the markdown. The point is the
content, without the page furniture around it.

Verify HTML and Markdown responses for each public page type, including URLs
outside the reported sample. Check that each Markdown response contains that
page's content and that ordinary browser requests still receive HTML.

The whole report, as markdown, is at https://goodforbots.com/r/youfiliate.com.md?scan=cmusfy8as007e01o0ml5tagpp. Read it first: this prompt covers one finding, and the report has everything the scan saw.

When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.

My website youfiliate.com scored 66/100 on Good for Bots (Fair). Good for Bots measures one thing: whether a language model can read a site and cite it.

I need you to: describe the links in llms.txt and open the file with a summary.

The check it fails is "llms.txt is descriptive, not a bare link list", worth 7.5 points. What the scanner saw: /llms.txt lists no links, so there is nothing to describe.

A file made of `- [Docs](/docs)` and thirty more like it tells a model that this site has
pages, which it already knew. What a model needs from an index is the thing that lets it
choose: what is behind each address, and whether fetching it is worth the tokens. That is
what a note after the link does, and it is the difference between an index and a table of
contents nobody asked for.

There is nothing normative to cite here, and saying so is the honest opening move. The
llms.txt specification makes the note after a link optional and says nothing about how
many links should carry one. This is our judgement about what makes such a file
worth retrieving, and the report says as much.

The file must:

- open with a `>` blockquote summary directly under the H1, before the first `##` heading
  and before the first link line. A summary written lower down is not read as the
  summary, by this scan or by anything skimming the file top-down;
- give each link a `:` followed by a note, on the same line as the link;
- describe what is behind the link rather than restating its name. `Pricing: pricing`
  costs a line and conveys nothing.

Before and after, on the same file:

```markdown
# Acme

## Docs

- [Quickstart](/docs/quickstart.md)
- [Concepts](/docs/concepts.md)
- [Webhooks](/docs/webhooks.md)
```

```markdown
# Acme

> Acme is a hosted billing API for subscription businesses. Everything below is written
> against the v2 API; v1 reaches end of life in March.

## Docs

- [Quickstart](/docs/quickstart.md): create an account and take a first payment in about
  ten minutes, with a runnable curl example
- [Concepts](/docs/concepts.md): customers, subscriptions, invoices and proration, and
  how the four objects relate
- [Webhooks](/docs/webhooks.md): the event list, signature verification, and the retry
  schedule
```

The second file is not longer for the sake of it. A model reading it can answer "where do
I find the retry schedule" without fetching anything, and then fetch exactly one page.

One caution. The summary has to sit above everything else. A blockquote placed after the
first heading, or after the first link, reads as a note about that section rather than
about the project, and is not counted as the file's summary.

Two things about how this is measured, both of which are limits rather than rules. The
thresholds are ours and are judgement, not measurement (a quarter of the links carrying
a note starts earning credit, three in five passes), so treat them as a floor and write a
note wherever a reader would want one. And only the presence of a note is detected, never
its quality: `: docs` counts the same as a sentence that says something. Writing `: docs`
on every link would clear the check and defeat the purpose of doing it, since the file is
for a model and not for us.

One last thing about order. A site with no `/llms.txt` at all fails this check too, and
that is deliberate: marking it inapplicable would drop it out of the denominator and
quietly reward a site for having done nothing. If both this and the llms.txt check are
outstanding, do that one first. There is nothing to describe until the file exists.

The whole report, as markdown, is at https://goodforbots.com/r/youfiliate.com.md?scan=cmusfy8as007e01o0ml5tagpp. Read it first: this prompt covers one finding, and the report has everything the scan saw.

When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.

My website youfiliate.com scored 66/100 on Good for Bots (Fair). Good for Bots measures one thing: whether a language model can read a site and cite it.

I need you to: publish an llms.txt at the site root.

The check it only partly passes is "llms.txt is present and well formed", with 5.63 points of it still on the table. What the scanner saw: /llms.txt has a title but lists no links to follow.

The file has a title and prose but no link lines, so there is nothing for a model to
fetch next. An entry is a bullet: `- [Name](/path)`, optionally followed by `:` and a
note. A bare URL on its own line, or a link inside a paragraph, is not an entry and is
not counted as one.

A model that arrives at a site with no `llms.txt` has to work out what is here from
whatever the navigation happens to expose. A sitemap does not help it. That file was
written for a search engine and lists addresses without saying what any of them are
for. `llms.txt` is the one file written for the other kind of reader: a short, curated
index that says what this project is and which pages are worth fetching.

The format is defined at llmstxt.org, currently at version 2. It is a convention with a
written specification and real adoption, not a standards-body document: there is no RFC
behind it and no working group. Worth knowing before treating any corner of it as
settled.

The file must:

- be served at `/llms.txt`, at the site root, answering 200 as text rather than HTML;
- open with an H1 naming the project (a `#`, a space, then the name). The specification
  makes this the one required element, so a file without it is not llms.txt at all;
- carry a blockquote directly under that title saying what the project is, in the terms
  someone who has never seen it would need;
- list the pages worth fetching as bullet links, `- [Name](url)`, each followed by `:`
  and a note saying what is behind it;
- group those links under `##` headings that mean something (`Docs`, `API`, `Optional`)
  rather than one undifferentiated pile.

By convention the section named `Optional` holds secondary information: links an agent
can skip when it needs a shorter context. It is the only place in the format where a
site can say "this matters less", so use it rather than leaving everything equally
weighted.

A minimal, valid example:

```markdown
# Acme

> Acme is a hosted billing API for subscription businesses. These pages cover the
> concepts, the REST endpoints, and the migration path from the v1 API.

Start with the quickstart. The API reference is generated from the OpenAPI document
linked below, so it is always current.

## Docs

- [Quickstart](/docs/quickstart.md): create an account and take a first payment
- [Concepts](/docs/concepts.md): customers, subscriptions and invoices, and how they relate
- [Migrating from v1](/docs/migrate-v1.md): what changed, and the cut-off date

## API

- [REST reference](/api/reference.md): every endpoint, with request and response shapes
- [OpenAPI document](/api/openapi.json): the machine-readable schema

## Optional

- [Changelog](/changelog.md): release notes, most recent first
```

Two things get this wrong more often than anything else.

**A catch-all route answering for the file.** A framework that serves its application
shell for every unknown path returns 200 and a page of HTML at `/llms.txt`, which is
worse than a plain 404 because nothing signals that the file is absent. Verify from
outside the application: `curl -s https://youfiliate.com/llms.txt | head -5` must print markdown,
not `<!doctype html>`. On most stacks the answer is to put the file in the directory
that is served verbatim (`public/`, `static/`, `www/`) rather than in the router.

**Generating it from the sitemap.** A dump of every URL the site has is a sitemap with
different punctuation. This file is a curated index: a few dozen entries a person chose,
each carrying a note. A generated wall of bare links satisfies the format and helps
nobody, and it fails the companion check that asks whether the index describes anything.

Two limits of this scan are worth knowing, because neither is a reading of the
specification. The specification allows the file at any subpath (`/docs/llms.txt`
covers the URLs beneath `/docs/`), but this scan requests the site root and nothing
else, so a file that exists only under a subpath is reported as absent. And the URLs
inside the file are never fetched, so an index pointing at pages that have moved still
reads as well formed here.

The whole report, as markdown, is at https://goodforbots.com/r/youfiliate.com.md?scan=cmusfy8as007e01o0ml5tagpp. Read it first: this prompt covers one finding, and the report has everything the scan saw.

When you are done, summarise what you changed and how to check it against a running instance. Do not try to run the Good for Bots scan yourself: it only sees what is already deployed, it is rate-limited, and re-running it is the site owner's call once the change ships.