---
title: "AI crawlers, fetchers and robots.txt (seo)"
description: "As of 2026-09-16, from each operator's documentation. Re-check tokens yearly, since operators add and rename bots (e.g."
canonical: https://void-design.vercel.app/docs/seo/ai-crawlers
lastModified: 2026-09-16
---

# AI crawlers, fetchers and robots.txt

As of 2026-09-16, from each operator's documentation. Re-check tokens yearly, since operators add and rename bots (e.g. `Google-NotebookLM` → `Google-GeminiNotebook` in 2026-08).

## 1. Crawler table

| Token (robots.txt) | Operator | Purpose | Obeys robots.txt | Runs JS |
|---|---|---|---|---|
| `Googlebot` | Google | Search, **including AI Overviews and AI Mode** | Yes | **Yes** (deferred) |
| `Google-Extended` | Google | *Control token, no crawler.* Gemini training + Gemini/Vertex grounding. Doesn't affect Search or AI Overviews | Yes | n/a |
| `Google-Agent`, `Google-GeminiNotebook`, `Google-Pinpoint` | Google | User-triggered fetchers/agents | Generally ignore | Chrome-based |
| `Bingbot` | Microsoft | Bing index → Copilot, Bing AI answers, third-party grounding (incl. ChatGPT's search provider) | Yes | Assume no |
| `OAI-SearchBot` | OpenAI | **ChatGPT search index.** Opted-out sites aren't shown in ChatGPT search answers (only as navigational links). ~24 h to apply | Yes | No |
| `ChatGPT-User` | OpenAI | User-triggered fetch in ChatGPT / GPTs | "may not apply" | No |
| `GPTBot` | OpenAI | **Training** | Yes | No |
| `Claude-SearchBot` | Anthropic | **Search** indexing for Claude | Yes (+ `Crawl-delay`) | No |
| `Claude-User` | Anthropic | User-triggered fetch | Yes | No |
| `ClaudeBot` | Anthropic | **Training** | Yes (+ `Crawl-delay`) | No |
| `Claude-Code` (UA only) | Anthropic | Coding-agent fetches (a top llms.txt requester) | – | No |
| `PerplexityBot` | Perplexity | **Search** index (not training) | Yes | No |
| `Perplexity-User` | Perplexity | User-triggered | Generally ignores | No |
| `Applebot` | Apple | Spotlight, Siri, Safari search | Yes | May render |
| `Applebot-Extended` | Apple | *Control token.* Training opt-out | Yes | n/a |
| `meta-webindexer` | Meta | Meta AI search | Yes | No |
| `meta-externalagent` | Meta | Training / product indexing | Yes | No |
| `meta-externalfetcher` | Meta | User-triggered | May bypass | No |
| `facebookexternalhit` | Meta | Link previews (reads OG tags) | May bypass | No |
| `Amzn-SearchBot` / `Amzn-User` | Amazon | Search (Alexa, Rufus) / user fetch; not training | Yes / – | – |
| `Amazonbot` | Amazon | Products and services, **may train** (honors `noarchive`) | Yes (no `Crawl-delay`) | – |
| `DuckAssistBot` | DuckDuckGo | Real-time AI answers; not training | Yes | – |
| `MistralAI-Index` / `MistralAI-User` / `MistralAI-Training` | Mistral | Search / user / training | Training: yes | – |
| `CCBot` | Common Crawl | Open corpus widely used for **training** | Yes | – |
| `Bytespider` | ByteDance | Training | Reportedly ignores | No |

Where answers come from:
- **Google AI Overviews / AI Mode:** the Google index (query fan-out). No special files or markup. Must be indexed and snippet-eligible.
- **ChatGPT search:** OAI-SearchBot + a third-party search provider (Bing). Allow OAI-SearchBot and be in Bing.
- **Claude web search:** Anthropic lists **Brave Search** as a web-search subprocessor (reported by secondary sources), so being indexed in Brave likely matters (check `site:` on search.brave.com).
- **Perplexity:** its own index (PerplexityBot) + live fetch (Perplexity-User).
- **Copilot:** Bing index. Bing uses IndexNow for freshness.

Measured behavior (Vercel network, 2024-12): AI crawlers fetch JS files **but don't execute them**. ChatGPT spent 34.8% and Claude 34.2% of fetches on 404s, and ChatGPT 14.4% on redirects. So fix broken internal links and redirect chains, and link straight to final URLs.

## 2. Policy presets

| Intent | `site.ai` in `site.config.ts` | Search engines | Retrieval + user fetchers | Training |
|---|---|---|---|---|
| Fully open | `{ retrieval: true, training: true }` | allow | allow | allow |
| Training opt-out (template default) | `{ retrieval: true, training: false }` | allow | allow | **disallow** |
| Block all AI | `{ retrieval: false, training: false }` | allow | disallow | disallow |

Blocking retrieval removes you from ChatGPT search, Claude and Perplexity answers. Only choose it deliberately. Google AI Overviews can't be opted out of via robots without leaving Search (use `nosnippet`/`noindex`).

## 3. `src/app/robots.ts`

Same as `templates/next/src/app/robots.ts`. The policy comes from `site.ai` and `site.privatePaths` in `site.config.ts`.

```ts
import type { MetadataRoute } from "next";
import { site } from "@/site.config";

/** Assistants fetching pages to answer users. Blocking these removes you from AI answers. */
const RETRIEVAL = ["OAI-SearchBot", "ChatGPT-User", "Claude-SearchBot", "Claude-User", "PerplexityBot", "Perplexity-User", "DuckAssistBot", "MistralAI-User"];
/** Model-training crawlers. Governed by site.ai.training. */
const TRAINING = ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended", "CCBot", "meta-externalagent", "Amazonbot", "Bytespider", "MistralAI-Training"];

export default function robots(): MetadataRoute.Robots {
  if (process.env.VERCEL_ENV && process.env.VERCEL_ENV !== "production") {
    return { rules: [{ userAgent: "*", disallow: "/" }] };      // previews are never indexable
  }
  const disallow = [...site.privatePaths];                        // don't list noindex pages: crawlers must see the noindex
  return {
    rules: [
      { userAgent: "*", allow: "/", disallow },
      site.ai.retrieval ? { userAgent: RETRIEVAL, allow: "/", disallow } : { userAgent: RETRIEVAL, disallow: "/" },
      site.ai.training ? { userAgent: TRAINING, allow: "/", disallow } : { userAgent: TRAINING, disallow: "/" },
    ],
    sitemap: `${site.url}/sitemap.xml`,
  };
}
```

Not on Vercel? Replace the `VERCEL_ENV` check with your platform's variable (e.g. `CONTEXT !== "production"` on Netlify, or a custom `DEPLOY_ENV`).

Output (retrieval allowed, training disallowed):

```
User-Agent: *
Allow: /
Disallow: /api/

User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
…
Allow: /
Disallow: /api/

User-Agent: GPTBot
User-Agent: ClaudeBot
…
Disallow: /

Sitemap: https://acme.example/sitemap.xml
```

A crawler obeys **only the most specific group matching its token**. A bot named in a group ignores `*`, so repeat the private paths in every named group.

## 4. Variant with comments and `Content-Signal` (route handler)

`robots.ts` can't emit comments or non-standard lines. Use a route handler when you want Cloudflare's `Content-Signal` (`search`, `ai-input`, `ai-train`) or human-readable notes. Don't have both `app/robots.ts` and `app/robots.txt/route.ts`.

```ts
// app/robots.txt/route.ts
import { site } from "@/site.config";
export const dynamic = "force-static";

export function GET() {
  const preview = process.env.VERCEL_ENV && process.env.VERCEL_ENV !== "production";
  const body = preview
    ? "User-agent: *\nDisallow: /\n"
    : [
        `# ${site.name}: AI policy. Search and answer engines welcome; training ${site.ai.training ? "allowed" : "not allowed"}.`,
        "User-agent: *",
        `Content-Signal: search=yes, ai-input=yes, ai-train=${site.ai.training ? "yes" : "no"}`, // inside the group, as Cloudflare emits it
        "Allow: /",
        "Disallow: /api/",
        "",
        ...(site.ai.training ? [] : ["User-agent: GPTBot", "User-agent: ClaudeBot", "User-agent: Google-Extended",
          "User-agent: Applebot-Extended", "User-agent: CCBot", "User-agent: meta-externalagent", "User-agent: Amazonbot",
          "User-agent: Bytespider", "Disallow: /", ""]),
        `Sitemap: ${site.url}/sitemap.xml`,
        "",
      ].join("\n");
  return new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8" } });
}
```

`Content-Signal` and IETF AIPREF `Content-Usage` are drafts or vendor conventions. Keep them consistent with your groups, and don't rely on them alone.

## 5. robots.txt hard rules

- `/robots.txt` returns 200 (or a deliberate 404). **Never 5xx or 429**: Google stops crawling for 12 h, then uses a cached copy for up to 30 days. 401/403 count as "no robots.txt".
- ≤ 500 KiB, UTF-8.
- Never disallow `/_next/` or other CSS/JS/image paths needed to render.
- `Crawl-delay`: ignored by Google and Amazon, honored by ClaudeBot/Claude-SearchBot.
- Blocking a URL doesn't de-index it if it's linked elsewhere. Use `noindex` (and let it be crawled).
- Spoofed bot UAs are often blocked by WAFs. If `void geo` reports `edge-blocked`, verify in server logs or the CDN's bot dashboard instead.

## Sources

- https://developers.openai.com/api/docs/bots
- https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- https://docs.perplexity.ai/guides/bots
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
- https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
- https://support.apple.com/en-us/119829 · https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- https://developer.amazon.com/amazonbot · https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot · https://docs.mistral.ai/robots · https://commoncrawl.org/ccbot
- https://vercel.com/blog/the-rise-of-the-ai-crawler
- https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ · https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/