Skip to content

Reference · seo

AI crawlers, fetchers and robots.txt

seo/references/ai-crawlers.md161 linesupdated 16 Sept 2026

As of 2026-09-16, from each operator's documentation. Re-check tokens yearly, since operators add and rename bots (e.g. Google-NotebookLM → Google-GeminiNotebook in 2026-08).

1. Crawler table

Token (robots.txt) Operator Purpose Obeys robots.txt Runs JS
Googlebot Google Search, including AI Overviews and AI Mode Yes Yes (deferred)
Google-Extended Google Control token, no crawler. Gemini training + Gemini/Vertex grounding. Doesn't affect Search or AI Overviews Yes n/a
Google-Agent, Google-GeminiNotebook, Google-Pinpoint Google User-triggered fetchers/agents Generally ignore Chrome-based
Bingbot Microsoft Bing index → Copilot, Bing AI answers, third-party grounding (incl. ChatGPT's search provider) Yes Assume no
OAI-SearchBot OpenAI ChatGPT search index. Opted-out sites aren't shown in ChatGPT search answers (only as navigational links). ~24 h to apply Yes No
ChatGPT-User OpenAI User-triggered fetch in ChatGPT / GPTs "may not apply" No
GPTBot OpenAI Training Yes No
Claude-SearchBot Anthropic Search indexing for Claude Yes (+ Crawl-delay) No
Claude-User Anthropic User-triggered fetch Yes No
ClaudeBot Anthropic Training Yes (+ Crawl-delay) No
Claude-Code (UA only) Anthropic Coding-agent fetches (a top llms.txt requester) – No
PerplexityBot Perplexity Search index (not training) Yes No
Perplexity-User Perplexity User-triggered Generally ignores No
Applebot Apple Spotlight, Siri, Safari search Yes May render
Applebot-Extended Apple Control token. Training opt-out Yes n/a
meta-webindexer Meta Meta AI search Yes No
meta-externalagent Meta Training / product indexing Yes No
meta-externalfetcher Meta User-triggered May bypass No
facebookexternalhit Meta Link previews (reads OG tags) May bypass No
Amzn-SearchBot / Amzn-User Amazon Search (Alexa, Rufus) / user fetch; not training Yes / – –
Amazonbot Amazon Products and services, may train (honors noarchive) Yes (no Crawl-delay) –
DuckAssistBot DuckDuckGo Real-time AI answers; not training Yes –
MistralAI-Index / MistralAI-User / MistralAI-Training Mistral Search / user / training Training: yes –
CCBot Common Crawl Open corpus widely used for training Yes –
Bytespider ByteDance Training Reportedly ignores No

Where answers come from:

  • Google AI Overviews / AI Mode: the Google index (query fan-out). No special files or markup. Must be indexed and snippet-eligible.
  • ChatGPT search: OAI-SearchBot + a third-party search provider (Bing). Allow OAI-SearchBot and be in Bing.
  • Claude web search: Anthropic lists Brave Search as a web-search subprocessor (reported by secondary sources), so being indexed in Brave likely matters (check site: on search.brave.com).
  • Perplexity: its own index (PerplexityBot) + live fetch (Perplexity-User).
  • Copilot: Bing index. Bing uses IndexNow for freshness.

Measured behavior (Vercel network, 2024-12): AI crawlers fetch JS files but don't execute them. ChatGPT spent 34.8% and Claude 34.2% of fetches on 404s, and ChatGPT 14.4% on redirects. So fix broken internal links and redirect chains, and link straight to final URLs.

2. Policy presets

Intent site.ai in site.config.ts Search engines Retrieval + user fetchers Training
Fully open { retrieval: true, training: true } allow allow allow
Training opt-out (template default) { retrieval: true, training: false } allow allow disallow
Block all AI { retrieval: false, training: false } allow disallow disallow

Blocking retrieval removes you from ChatGPT search, Claude and Perplexity answers. Only choose it deliberately. Google AI Overviews can't be opted out of via robots without leaving Search (use nosnippet/noindex).

3. src/app/robots.ts

Same as templates/next/src/app/robots.ts. The policy comes from site.ai and site.privatePaths in site.config.ts.

ts
import type { MetadataRoute } from "next";
import { site } from "@/site.config";

/** Assistants fetching pages to answer users. Blocking these removes you from AI answers. */
const RETRIEVAL = ["OAI-SearchBot", "ChatGPT-User", "Claude-SearchBot", "Claude-User", "PerplexityBot", "Perplexity-User", "DuckAssistBot", "MistralAI-User"];
/** Model-training crawlers. Governed by site.ai.training. */
const TRAINING = ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended", "CCBot", "meta-externalagent", "Amazonbot", "Bytespider", "MistralAI-Training"];

export default function robots(): MetadataRoute.Robots {
  if (process.env.VERCEL_ENV && process.env.VERCEL_ENV !== "production") {
    return { rules: [{ userAgent: "*", disallow: "/" }] };      // previews are never indexable
  }
  const disallow = [...site.privatePaths];                        // don't list noindex pages: crawlers must see the noindex
  return {
    rules: [
      { userAgent: "*", allow: "/", disallow },
      site.ai.retrieval ? { userAgent: RETRIEVAL, allow: "/", disallow } : { userAgent: RETRIEVAL, disallow: "/" },
      site.ai.training ? { userAgent: TRAINING, allow: "/", disallow } : { userAgent: TRAINING, disallow: "/" },
    ],
    sitemap: `${site.url}/sitemap.xml`,
  };
}

Not on Vercel? Replace the VERCEL_ENV check with your platform's variable (e.g. CONTEXT !== "production" on Netlify, or a custom DEPLOY_ENV).

Output (retrieval allowed, training disallowed):

text
User-Agent: *
Allow: /
Disallow: /api/

User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
…
Allow: /
Disallow: /api/

User-Agent: GPTBot
User-Agent: ClaudeBot
…
Disallow: /

Sitemap: https://acme.example/sitemap.xml

A crawler obeys only the most specific group matching its token. A bot named in a group ignores *, so repeat the private paths in every named group.

4. Variant with comments and Content-Signal (route handler)

robots.ts can't emit comments or non-standard lines. Use a route handler when you want Cloudflare's Content-Signal (search, ai-input, ai-train) or human-readable notes. Don't have both app/robots.ts and app/robots.txt/route.ts.

ts
// app/robots.txt/route.ts
import { site } from "@/site.config";
export const dynamic = "force-static";

export function GET() {
  const preview = process.env.VERCEL_ENV && process.env.VERCEL_ENV !== "production";
  const body = preview
    ? "User-agent: *\nDisallow: /\n"
    : [
        `# ${site.name}: AI policy. Search and answer engines welcome; training ${site.ai.training ? "allowed" : "not allowed"}.`,
        "User-agent: *",
        `Content-Signal: search=yes, ai-input=yes, ai-train=${site.ai.training ? "yes" : "no"}`, // inside the group, as Cloudflare emits it
        "Allow: /",
        "Disallow: /api/",
        "",
        ...(site.ai.training ? [] : ["User-agent: GPTBot", "User-agent: ClaudeBot", "User-agent: Google-Extended",
          "User-agent: Applebot-Extended", "User-agent: CCBot", "User-agent: meta-externalagent", "User-agent: Amazonbot",
          "User-agent: Bytespider", "Disallow: /", ""]),
        `Sitemap: ${site.url}/sitemap.xml`,
        "",
      ].join("\n");
  return new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8" } });
}

Content-Signal and IETF AIPREF Content-Usage are drafts or vendor conventions. Keep them consistent with your groups, and don't rely on them alone.

5. robots.txt hard rules

  • /robots.txt returns 200 (or a deliberate 404). Never 5xx or 429: Google stops crawling for 12 h, then uses a cached copy for up to 30 days. 401/403 count as "no robots.txt".
  • ≤ 500 KiB, UTF-8.
  • Never disallow /_next/ or other CSS/JS/image paths needed to render.
  • Crawl-delay: ignored by Google and Amazon, honored by ClaudeBot/Claude-SearchBot.
  • Blocking a URL doesn't de-index it if it's linked elsewhere. Use noindex (and let it be crawled).
  • Spoofed bot UAs are often blocked by WAFs. If void geo reports edge-blocked, verify in server logs or the CDN's bot dashboard instead.

Sources