Skip to content

Rule · AI search

AI crawler policy (allow/deny matrix from robots.txt)

geo/robots-ai-policyinfovoid geoupdated

Why it matters

Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are separate tokens; an explicit policy controls each independently. A bot named in its own group ignores the * group.

How to fix it

State the policy explicitly in app/robots.ts. Allow retrieval bots to stay visible in AI search, and decide training separately.

Example

ts
// app/robots.ts
const RETRIEVAL = ['OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot', 'Claude-User', 'PerplexityBot', 'Perplexity-User']
const TRAINING = ['GPTBot', 'ClaudeBot', 'Google-Extended', 'Applebot-Extended', 'CCBot', 'meta-externalagent', 'Amazonbot', 'MistralAI-Training', 'Bytespider']
const PRIVATE = ['/api/', '/account/']

export default function robots(): MetadataRoute.Robots {
  const allowTraining = process.env.SEO_ALLOW_AI_TRAINING !== 'false'
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: PRIVATE },
      { userAgent: RETRIEVAL, allow: '/', disallow: PRIVATE }, // repeat disallows: named groups ignore '*'
      ...(allowTraining ? [] : [{ userAgent: TRAINING, disallow: '/' }]),
    ],
    sitemap: 'https://acme.example/sitemap.xml',
  }
}

References

More ai search rules

void geo reports 41 rules in this category. Generative-engine optimisation: whether AI crawlers that don't run JavaScript see the same content, valid and visible JSON-LD, an explicit AI robots policy, llms.txt, Markdown mirrors and answer-first writing.