Why it matters
When robots.txt errors, Google stops crawling the site (for up to 12 hours), then falls back to a cached copy for up to 30 days.
How to fix it
Serve robots.txt as a static 200 (app/robots.ts is prerendered) and exclude it from rate limiting and auth.
References
More seo rules
void seo reports 60 rules in this category. What search engines need from the raw response: status codes, titles and descriptions in <head>, self-referencing canonicals, one h1, crawlable links, Open Graph, icons, robots.txt and a sitemap with real dates.
faviconNo crawlable favicon in a Google-supported formatapple-touch-iconapple-touch-icon missing or not a 180×180 PNGmanifestWeb app manifest missing or incompletetheme-colorNo theme-color metarobots-txt-missingNo robots.txtrobots-txt-invalidrobots.txt serves HTML, exceeds 500 KiB, or has unparseable linesrobots-blocks-allrobots.txt disallows the site for * / Googlebot / Bingbotrobots-blocks-resourcesrobots.txt blocks JS/CSS needed to render