--- title: "robots.txt returns 5xx/429 or times out (seo/robots-txt-unreachable)" description: "seo/robots-txt-unreachable: When robots.txt errors, Google stops crawling the site (for up to 12 hours), then falls back to a cached copy for up to 30 days." canonical: https://void-design.vercel.app/rules/seo/robots-txt-unreachable lastModified: 2026-09-16 --- # robots.txt returns 5xx/429 or times out `seo/robots-txt-unreachable` · severity **error** · category SEO · detected by `void seo` and `void audit` ## Why it matters When robots.txt errors, Google stops crawling the site (for up to 12 hours), then falls back to a cached copy for up to 30 days. ## How to fix it Serve robots.txt as a static 200 (app/robots.ts is prerendered) and exclude it from rate limiting and auth. ## References - https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec ## More seo rules `void seo` reports 60 rules in this category. What search engines need from the raw response: status codes, titles and descriptions in , self-referencing canonicals, one h1, crawlable links, Open Graph, icons, robots.txt and a sitemap with real dates. - `favicon` No crawlable favicon in a Google-supported format — [seo/favicon](https://void-design.vercel.app/rules/seo/favicon) - `apple-touch-icon` apple-touch-icon missing or not a 180×180 PNG — [seo/apple-touch-icon](https://void-design.vercel.app/rules/seo/apple-touch-icon) - `manifest` Web app manifest missing or incomplete — [seo/manifest](https://void-design.vercel.app/rules/seo/manifest) - `theme-color` No theme-color meta — [seo/theme-color](https://void-design.vercel.app/rules/seo/theme-color) - `robots-txt-missing` No robots.txt — [seo/robots-txt-missing](https://void-design.vercel.app/rules/seo/robots-txt-missing) - `robots-txt-invalid` robots.txt serves HTML, exceeds 500 KiB, or has unparseable lines — [seo/robots-txt-invalid](https://void-design.vercel.app/rules/seo/robots-txt-invalid) - `robots-blocks-all` robots.txt disallows the site for * / Googlebot / Bingbot — [seo/robots-blocks-all](https://void-design.vercel.app/rules/seo/robots-blocks-all) - `robots-blocks-resources` robots.txt blocks JS/CSS needed to render — [seo/robots-blocks-resources](https://void-design.vercel.app/rules/seo/robots-blocks-resources) Detected by `void seo` and `void audit`. Explain it in a terminal: `void rules seo/robots-txt-unreachable`