---
title: "AI visibility & web standards blog"
description: "Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans."
url: "https://goodforbots.com/blog"
---

# AI visibility & web standards blog

Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.

Categories: All posts (this page) · [Standards](https://goodforbots.com/blog.md?category=standards) · [AI Crawlers](https://goodforbots.com/blog.md?category=ai-crawlers)

- [llms.txt adoption statistics: why the studies disagree](https://goodforbots.com/blog/llms-txt-adoption-statistics): Standards, Sep 29, 2026, 12 min read. Published llms.txt adoption figures run from under 0.005% to 57%. What each study counted, why they disagree, and which figure to quote for which sites.
- [Which AI crawlers to block in robots.txt, and which to keep](https://goodforbots.com/blog/which-ai-crawlers-to-block): AI Crawlers, Sep 24, 2026, 10 min read. Blocking GPTBot or ClaudeBot refuses model training and leaves AI search alone. Blocking OAI-SearchBot or Claude-User keeps you out of their answers.

Every post is also available as Markdown at https://goodforbots.com/blog/{slug}.md.

## Structured data

```json
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "CollectionPage",
      "@id": "https://goodforbots.com/blog#webpage",
      "url": "https://goodforbots.com/blog",
      "name": "AI visibility & web standards blog",
      "description": "Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.",
      "inLanguage": "en",
      "isPartOf": {
        "@id": "https://goodforbots.com/#website"
      },
      "breadcrumb": {
        "@id": "https://goodforbots.com/blog#breadcrumb"
      },
      "mainEntity": {
        "@id": "https://goodforbots.com/blog#blog"
      }
    },
    {
      "@type": "BreadcrumbList",
      "@id": "https://goodforbots.com/blog#breadcrumb",
      "itemListElement": [
        {
          "@type": "ListItem",
          "position": 1,
          "name": "Home",
          "item": "https://goodforbots.com/"
        },
        {
          "@type": "ListItem",
          "position": 2,
          "name": "Blog",
          "item": "https://goodforbots.com/blog"
        }
      ]
    },
    {
      "@type": "Blog",
      "@id": "https://goodforbots.com/blog#blog",
      "url": "https://goodforbots.com/blog",
      "name": "AI visibility & web standards blog",
      "description": "Web standards for AI crawlers and agents, explained from the specifications: llms.txt, robots.txt, Markdown negotiation and more, with data from our own scans.",
      "inLanguage": "en",
      "publisher": {
        "@id": "https://goodforbots.com/#organization"
      },
      "isPartOf": {
        "@id": "https://goodforbots.com/#website"
      },
      "blogPost": [
        {
          "@type": "BlogPosting",
          "@id": "https://goodforbots.com/blog/llms-txt-adoption-statistics#article",
          "url": "https://goodforbots.com/blog/llms-txt-adoption-statistics",
          "headline": "llms.txt adoption statistics: why the studies disagree",
          "datePublished": "2026-09-29",
          "author": {
            "@id": "https://goodforbots.com/#person-mateusz-pawlica"
          }
        },
        {
          "@type": "BlogPosting",
          "@id": "https://goodforbots.com/blog/which-ai-crawlers-to-block#article",
          "url": "https://goodforbots.com/blog/which-ai-crawlers-to-block",
          "headline": "Which AI crawlers to block in robots.txt, and which to keep",
          "datePublished": "2026-09-24",
          "author": {
            "@id": "https://goodforbots.com/#person-mateusz-pawlica"
          }
        }
      ]
    },
    {
      "@type": "Person",
      "@id": "https://goodforbots.com/#person-mateusz-pawlica",
      "name": "Mateusz Pawlica",
      "givenName": "Mateusz",
      "familyName": "Pawlica",
      "jobTitle": "Founder of Good for Bots",
      "description": "Web developer who builds directories and AI-powered products, and the founder of Good for Bots.",
      "url": "https://goodforbots.com/about#founder",
      "sameAs": [
        "https://pl.linkedin.com/in/mateusz-pawlica-65238816b",
        "https://pawlicaweb.pl/o-mnie"
      ],
      "knowsAbout": [
        "llms.txt",
        "robots.txt",
        "AI crawlers",
        "Markdown content negotiation",
        "Structured data",
        "Technical SEO",
        "Web directories",
        "Next.js",
        "TypeScript",
        "Generative AI"
      ],
      "worksFor": {
        "@id": "https://goodforbots.com/#organization"
      }
    },
    {
      "@type": "Organization",
      "@id": "https://goodforbots.com/#organization",
      "name": "Good for Bots",
      "url": "https://goodforbots.com",
      "logo": "https://goodforbots.com/icon-512.png",
      "founder": {
        "@id": "https://goodforbots.com/#person-mateusz-pawlica"
      },
      "contactPoint": {
        "@type": "ContactPoint",
        "contactType": "customer support",
        "email": "support@goodforbots.com",
        "url": "https://goodforbots.com/contact"
      }
    },
    {
      "@type": "WebSite",
      "@id": "https://goodforbots.com/#website",
      "name": "Good for Bots",
      "url": "https://goodforbots.com",
      "description": "Check if AI crawlers can access and read your website, free. Get a score out of 100 with every finding and fix prompts, then browse reports in the directory.",
      "inLanguage": "en",
      "publisher": {
        "@id": "https://goodforbots.com/#organization"
      },
      "potentialAction": {
        "@type": "SearchAction",
        "target": {
          "@type": "EntryPoint",
          "urlTemplate": "https://goodforbots.com/directory?q={q}"
        },
        "query-input": "required maxlength=120 name=q"
      }
    }
  ]
}
```