---
title: "AI crawler user agents get 401/403/429/503 or a challenge page (geo/ai-bot-blocked)"
description: "geo/ai-bot-blocked: A WAF or bot-management rule that blocks GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot removes the site from AI answers regardless of…"
canonical: https://void-design.vercel.app/rules/geo/ai-bot-blocked
lastModified: 2026-09-16
---

# AI crawler user agents get 401/403/429/503 or a challenge page

`geo/ai-bot-blocked` · severity **error** · category AI search · detected by `void geo` and `void audit`

## Why it matters

A WAF or bot-management rule that blocks GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot removes the site from AI answers regardless of robots.txt. Challenge responses to spoofed UAs can be inconclusive, because verified crawlers are identified by IP.

## How to fix it

Allow verified AI search crawlers in the CDN/WAF (Cloudflare AI Crawl Control, Vercel Firewall bot rules). Confirm in server logs that real crawler IPs get 200.

## References

- https://developers.openai.com/api/docs/bots
- https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- https://docs.perplexity.ai/guides/bots

## More ai search rules

`void geo` reports 41 rules in this category. Generative-engine optimisation: whether AI crawlers that don't run JavaScript see the same content, valid and visible JSON-LD, an explicit AI robots policy, llms.txt, Markdown mirrors and answer-first writing.

- `js-dependency` Main content only appears after JavaScript runs — [geo/js-dependency](https://void-design.vercel.app/rules/geo/js-dependency)
- `hidden-streamed-content` Crawlers receive main content only inside hidden streaming containers — [geo/hidden-streamed-content](https://void-design.vercel.app/rules/geo/hidden-streamed-content)
- `ai-content-differs` AI crawlers get materially different HTML than browsers — [geo/ai-content-differs](https://void-design.vercel.app/rules/geo/ai-content-differs)
- `jsonld-parse-error` JSON-LD block is not valid JSON — [geo/jsonld-parse-error](https://void-design.vercel.app/rules/geo/jsonld-parse-error)
- `jsonld-unsafe` JSON-LD contains a literal </script or <!-- — [geo/jsonld-unsafe](https://void-design.vercel.app/rules/geo/jsonld-unsafe)
- `jsonld-missing` Page has no JSON-LD — [geo/jsonld-missing](https://void-design.vercel.app/rules/geo/jsonld-missing)
- `jsonld-not-in-raw` JSON-LD is injected by JavaScript — [geo/jsonld-not-in-raw](https://void-design.vercel.app/rules/geo/jsonld-not-in-raw)
- `home-entity-missing` Home page lacks Organization + a single WebSite node — [geo/home-entity-missing](https://void-design.vercel.app/rules/geo/home-entity-missing)

Detected by `void geo` and `void audit`. Explain it in a terminal: `void rules geo/ai-bot-blocked`