---
title: "Practical guides for AI crawlers and agents"
description: "Step-by-step guides to making your website readable for AI crawlers and agents, with examples tested against our scanner."
url: "https://goodforbots.com/guides"
---

# Practical guides for AI crawlers and agents

Step-by-step guides to making your website readable for AI crawlers and agents, with examples tested against our scanner.

Each guide covers one topic our scanner checks: a complete example that passes those checks, common mistakes with their fixes, and the primary sources it was reviewed against. The current rules for every check are on [Standards and scoring](https://goodforbots.com/standards); dated news and measurements go to [the blog](https://goodforbots.com/blog).

Categories: All guides (this page) · [Crawler access](https://goodforbots.com/guides.md?category=crawler-access) · [Page content](https://goodforbots.com/guides.md?category=page-content) · [Discovery files](https://goodforbots.com/guides.md?category=discovery-files)

- [How to write robots.txt rules for AI crawlers](https://goodforbots.com/guides/robots-txt-ai-crawlers): Crawler access, reviewed Sep 25, 2026. Write robots.txt rules that treat AI training, search and user-triggered crawlers separately, with tested examples, the current token list and common fixes.
- [How to serve content AI crawlers can read without JavaScript](https://goodforbots.com/guides/server-rendered-html): Page content, reviewed Sep 25, 2026. Which AI crawlers run JavaScript, how to fix empty app shells in Next.js, Nuxt, SvelteKit or Vite, and how to test the raw HTML response yourself.
- [How to serve Markdown to AI agents with content negotiation](https://goodforbots.com/guides/markdown-negotiation): Page content, reviewed Oct 3, 2026. Return Markdown from the same URL when an AI agent sends Accept: text/markdown, with tested responses, Vary and CDN caching, and nginx and Next.js setups.
- [How to write a useful llms.txt](https://goodforbots.com/guides/llms-txt): Discovery files, reviewed Sep 24, 2026. Write an llms.txt that tells AI agents what your site covers: a complete tested example, the format explained, publishing checks and fixes for common mistakes.

Every guide is also available as Markdown at https://goodforbots.com/guides/{slug}.md.

[Check your site](https://goodforbots.com/#scan)

## Structured data

```json
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "CollectionPage",
      "@id": "https://goodforbots.com/guides#webpage",
      "url": "https://goodforbots.com/guides",
      "name": "Practical guides for AI crawlers and agents",
      "description": "Step-by-step guides to making your website readable for AI crawlers and agents, with examples tested against our scanner.",
      "inLanguage": "en",
      "isPartOf": {
        "@id": "https://goodforbots.com/#website"
      },
      "breadcrumb": {
        "@id": "https://goodforbots.com/guides#breadcrumb"
      },
      "mainEntity": {
        "@id": "https://goodforbots.com/guides#items"
      }
    },
    {
      "@type": "BreadcrumbList",
      "@id": "https://goodforbots.com/guides#breadcrumb",
      "itemListElement": [
        {
          "@type": "ListItem",
          "position": 1,
          "name": "Home",
          "item": "https://goodforbots.com/"
        },
        {
          "@type": "ListItem",
          "position": 2,
          "name": "Guides",
          "item": "https://goodforbots.com/guides"
        }
      ]
    },
    {
      "@type": "ItemList",
      "@id": "https://goodforbots.com/guides#items",
      "name": "Practical guides for AI crawlers and agents",
      "numberOfItems": 4,
      "itemListElement": [
        {
          "@type": "ListItem",
          "position": 1,
          "name": "How to write robots.txt rules for AI crawlers",
          "url": "https://goodforbots.com/guides/robots-txt-ai-crawlers"
        },
        {
          "@type": "ListItem",
          "position": 2,
          "name": "How to serve content AI crawlers can read without JavaScript",
          "url": "https://goodforbots.com/guides/server-rendered-html"
        },
        {
          "@type": "ListItem",
          "position": 3,
          "name": "How to serve Markdown to AI agents with content negotiation",
          "url": "https://goodforbots.com/guides/markdown-negotiation"
        },
        {
          "@type": "ListItem",
          "position": 4,
          "name": "How to write a useful llms.txt",
          "url": "https://goodforbots.com/guides/llms-txt"
        }
      ]
    },
    {
      "@type": "Organization",
      "@id": "https://goodforbots.com/#organization",
      "name": "Good for Bots",
      "url": "https://goodforbots.com",
      "logo": "https://goodforbots.com/icon-512.png",
      "founder": {
        "@id": "https://goodforbots.com/#person-mateusz-pawlica"
      },
      "contactPoint": {
        "@type": "ContactPoint",
        "contactType": "customer support",
        "email": "support@goodforbots.com",
        "url": "https://goodforbots.com/contact"
      }
    },
    {
      "@type": "WebSite",
      "@id": "https://goodforbots.com/#website",
      "name": "Good for Bots",
      "url": "https://goodforbots.com",
      "description": "Check if AI crawlers can access and read your website, free. Get a score out of 100 with every finding and fix prompts, then browse reports in the directory.",
      "inLanguage": "en",
      "publisher": {
        "@id": "https://goodforbots.com/#organization"
      },
      "potentialAction": {
        "@type": "SearchAction",
        "target": {
          "@type": "EntryPoint",
          "urlTemplate": "https://goodforbots.com/directory?q={q}"
        },
        "query-input": "required maxlength=120 name=q"
      }
    },
    {
      "@type": "Person",
      "@id": "https://goodforbots.com/#person-mateusz-pawlica",
      "name": "Mateusz Pawlica",
      "givenName": "Mateusz",
      "familyName": "Pawlica",
      "jobTitle": "Founder of Good for Bots",
      "description": "Web developer who builds directories and AI-powered products, and the founder of Good for Bots.",
      "url": "https://goodforbots.com/about#founder",
      "sameAs": [
        "https://pl.linkedin.com/in/mateusz-pawlica-65238816b",
        "https://pawlicaweb.pl/o-mnie"
      ],
      "knowsAbout": [
        "llms.txt",
        "robots.txt",
        "AI crawlers",
        "Markdown content negotiation",
        "Structured data",
        "Technical SEO",
        "Web directories",
        "Next.js",
        "TypeScript",
        "Generative AI"
      ],
      "worksFor": {
        "@id": "https://goodforbots.com/#organization"
      }
    }
  ]
}
```