Why it matters
A WAF or bot-management rule that blocks GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot removes the site from AI answers regardless of robots.txt. Challenge responses to spoofed UAs can be inconclusive, because verified crawlers are identified by IP.
How to fix it
Allow verified AI search crawlers in the CDN/WAF (Cloudflare AI Crawl Control, Vercel Firewall bot rules). Confirm in server logs that real crawler IPs get 200.
References
More ai search rules
void geo reports 41 rules in this category. Generative-engine optimisation: whether AI crawlers that don't run JavaScript see the same content, valid and visible JSON-LD, an explicit AI robots policy, llms.txt, Markdown mirrors and answer-first writing.
js-dependencyMain content only appears after JavaScript runshidden-streamed-contentCrawlers receive main content only inside hidden streaming containersai-content-differsAI crawlers get materially different HTML than browsersjsonld-parse-errorJSON-LD block is not valid JSONjsonld-unsafeJSON-LD contains a literal </script or <!--jsonld-missingPage has no JSON-LDjsonld-not-in-rawJSON-LD is injected by JavaScripthome-entity-missingHome page lacks Organization + a single WebSite node