Skip to content

Rule · AI search

robots.txt blocks AI training crawlers

geo/training-bots-blockedinfovoid geoupdated

Why it matters

A legitimate choice: it doesn't affect Google Search or AI Overviews (Google-Extended doesn't), or AI search visibility while retrieval bots stay allowed.

How to fix it

No action if intended. Make sure OAI-SearchBot, Claude-SearchBot and PerplexityBot are still allowed.

References

void geo reports 41 rules in this category. Generative-engine optimisation: whether AI crawlers that don't run JavaScript see the same content, valid and visible JSON-LD, an explicit AI robots policy, llms.txt, Markdown mirrors and answer-first writing.